You may want to consider a slight modification to your design, which would have the dark green upper track against the wall, all around the room. This will allow you to cantilever the roadbed on a 1x6, supported on the wall by steel L-brackets (my supports are every 6 to 8 ft). The picture below shows how I did this. In my model "world", this upper level is not part of the lower level, therefore the scenery is separate, not tied together with anything. I even have different backdrops for the upper and lower level when they are separate.
To reduce shadows, I have some "string" LED lighting under the top level shelf. (along the back wall)
As upper level track goes down to the lower level, you can combine the scenery so they are considered to be part of the same "world".
At this point, the upper and lower tracks are place to allow them to pass each other without interference. My maximum grade is 6%, which was needed because my upper level and lower levels are 16" apart. This is way more than you need, I had a staircase railing to deal with, which forced the upper level elevation.
Once you add some 060 curves in the corners, you have room for some small industries, an example is shown below. This siding only holds 1 car with a second car sitting over the uncoupler track on occasion. The industry in the picture below is only 3" deep.
On my layout, I have one passing track on the top level, and one passing track on the bottom level. You can run trains continuously on both the top and bottom levels. Make sure the passing tracks are both the same length, as this will establish the length of your longest train.
As you mention, the Wye takes up a lot of space, and eliminating it would allow you the make the train table narrower, making it easier to reach the track against the wall. You want to be able to reach the back track with reasonable comfort. The corners are always a problem, so I plan not to have too many derailments there!
There are many ways to tweek your design, so enjoy looking at the options. To quote Star Wars, the solution will eventually present itself.
Note: the masking tape seen on my layout pictures from time to time are maintenance "work order" that I place in any place I need to later address. This includes trackwork issues, poor connections, burned out light bulbs, or the need to add some scenery to 'hide' stuff. You can see from my pictures that I seem to always have a number of maintenance issues that keep popping up.
Also not that my layout is designed with Fastrack on painted carpet underpad. All the scenery is painted foam, held together with toothpicks and acrylic calking. These buildings, trees and rocks are all removable for easy maintenance of track, etc. Any electrical in the building is unplugged, when the building is lifted up. My entire layout can be dismantled in a few hours. Then reassembled after I have completed electrical wiring work or changes to my track plan. I am always changing things around, as I get new and better ideas by following this forum.