Proportion is what is important to me. Cars come in a wide range of sizes. Also the placement of items on a layout can be used to "allow" one to use various sizes. Larger cars can be placed in the distance to appear smaller if not too close to something that will emphasize and reveal their larger size by comparison.
I have a few O scale cars which are not on my layout. The layout is Toy O27 layout with lots of Plasticville buildings. I mostly use HOish size cars and trucks and do not use the O scale cars to allow comparison. I also like the fact that the small cars and trucks put emphasis on the the trains making them appear larger than they are. After all this is model train layout. Larger vehicles would diminish the size of the train engines and cars.