Will, give us the dimensions you have to work with? It sounds like you want 6 axel multi unit...multi car trains. Lots of main line running. You won't be satisfied with a small switching layout.
A quick and easy way to get a feel for what you can do with O scale is to plan with construction ribbon. There is orange construction ribbon available at Lowes that is the width of O scale 2R track...its cheap too. Buy a roll and lay it out in the basement to get a feel for what you can do with the space. Run the layout in your mind...visualize what you can do with scenery etc in the space you have. For me this has been better than paper.....here are some shots from a recent planning session I had.....
The other issue is cost. O scale is very expensive compared to HO....$80 for a switch, $400 locos, $10 for a section of flex. With that said there is nothing like the presence of O scale. I was an HO modeler for many years and could never go back.
Don
