OK, let's get real here. When a train jumps the tracks in the far back corner on the right (with no aisle access), how are you going to fix it? Since you don't connect your outer ovals to your inner oval, how are you going to change the direction of the trains on the outer oval? And, once you have reversed a train on the inner oval, how are you going to reverse that train again? Back it all the way to the reversing loop at the end?
In the space you have, you should be able to build a nice "around the walls" layout (about 24 to 30 inches wide) allowing easy access to all track for maintenance and repairs. You should even be able to include two reversing loops so trains will change direction if, and when, you desire. From your plan, it sure looks like you want to just sit back and watch them run, so I won't even get into railroad "operations."
Chuck