On tangent (straight) you can go as close as 3.5" (14 scale feet). On curves is where things get dicey because you have to deal with overhang from equipment on the inside track and under-hang from equipment on the outside track. The general rule I've seen for O scale is 5" on curves from 36" radius (O-72) up to 48" radius (O-96); 4.5" for 48" (O-96) to 60" (O-120); and 4" for 60" and above. This covers most standard equipment, but you still have to make provisions for articulated steam and longer equipment such as scale-length passenger cars, 86-foot HyCubes, and 89-foot Autoracks and trailer flats.
Use of flex track really comes in handy here for increasing/decreasing mainline spacing.
Hope this helps.