If I read your post right, I believe the photo shows your current table and the planned 3x10 addition. Since you mentioned O-54 curves, I assume you don't plan to continue with FasTrack, it goes from O-48 to O-60 with no O-54 curves. I see you have some Lionel O tubular track on the table, so that's what I used to show O-72 (Blue) and O-54 (Yellow), but added the FasTrack O-36 (Green) just to show the difference with what you currently have. FWIW, I believe FasTrack is measured from the outside rail while most other brands are measured from the center rail, so there's always a slight difference in the overall diameter of a circle.
The 2nd photo just shows a simple design with O-54 inside O-72 for a dual track setup. Obviously, the curves in the lower right are tighter than O-54 using O-31 and O-42 curves.
The problem you will face with the addition is reaching sections of the layout. Most people can only reach 30-36", so you'll either need to crawl on top or add access hatches to deal with problems, add landscaping, etc.
When it comes to software, there are several packages available and several threads outlining the pros and cons of each. But, since you didn't ask about software, I'll just say that they'll all get the job done if you plan to go that route.
