Welcome to O gauge! I just got into this myself a few months back and know what you're going through. Things you'll want to consider:
- DCS by MTH or Lionel's Legacy system: Go with DCS, I went Lionel because of the name recognition. It's great, but everything Lionel costs 20% to 25% more. Kinda sucks when you want to expand your locomotive roster. You can get top-notch MTH engines (slightly used) for $200 to $300 that are equal if not greater in quality.
- "Traditional" vs. Standard O: STAY AWAY FROM TRADITIONAL! It looks way too toy-ish. Coming from HO, you're probably used to nice, scaled versions of your rolling stock and locos. The traditional or O-27 stuff looks almost kiddish (no offense to those that like it). You'll especially want to stay away from Lionel's rolling stock as it's way too difficult for a newbie O scale person to tell if it's scale or traditional. Kinda sucks when you order a piece, then you receive it in the mail and lash it up and it looks way too small compared to your other stuff. O scale stuff is supposed to be big, so why buy small O gauge stuff????
- What track to use? Go with Atlas, looks very realistic and is very low noise. O gauge trains are heavy and can make quite a bit of noise (which is awesome btw). You're choice of track can help mitigate this.
- Power: I recommend the MTH Z-4000. Is about $200 bucks cheaper than Lionel's equivalent and will let you expand without having to upgrade anything later down the road. You can even find used ones for around $150 to $200.
As far as your layout is concerned, I'm too new to chip in here about different layouts but if you can, try to incorporate a nice long straightaway into your layout. These big o scale engines sound awesome rumbling down a straightaway. O scale diesel's typically weigh around 5 lbs. and sound great with all that weight sitting on metal wheels ontop of metal track. 
Additionally, you can always make a "shelf train" if space is an issue. That's what I did as my office is full of kid toys, computer desk and furniture etc....
Here's a video of my shelf train. You can see the HO train car sitting on-top of the O scale flatcar for a sense of scale and size.
https://www.youtube.com/watch?v=NpKtws1fAuw
If you have any other questions, you're in the right spot. Everyone here is very friendly and eager to help us "newbies" out.
