I buy what I like. I have a lot of scale stuff, and quite a bit of semi-scale as well. I also have some whimsical pieces that don't have any "scale", and I plan on building more of those. I'm doing this to have fun, but I can appreciate the detailing and mass of a true scale Challenger or Big Boy, and the fun of a custom made track speeder. 
The only recommendation I can honestly make is when you build the layout, make the curves as wide as possible! I was initially thinking of O72 as the maximum curve on my layout, but as my plan evolved, I realized that it would be much smarter to make O72 as the minimum curve! This gives me complete flexibility in running scale, semi-scale, novelty powered units, etc. Your semi-scale will look great on O99 curves, but your scale stuff will look really bad derailing on that O42 curve. 
