Both Lionel and MTH systems are very good. While similar, they are different and cannot run each others equipment. They do, however, coexist just fine. I have both so I can run whatever I like.
After running HO and N the shock of O is how much space is needed. Essentially, O comes in two sizes, traditional or semi-scale and scale. In traditional you can make a circle as small as 27" diameter, In scale you will find many locomotives with minimum that require 48", 54" and even 72" diameter so, you are going to need some real estate.
Which ever you choose you will find both well represented in this forum along with lots of people ready to help, should you need it. Welcome.
