Take all of this with a grain of salt, because I have never built a layout bigger than 4x8' (I did cut the corners off of that).
In my experience, straight sides have their place. One place where straight sides work fine is on immense layouts such as model train clubs. When a layout gets big enough, it almost doesn't matter what you do to the edges because they become a smaller fraction of the overall effect. Another place where straight sides work is in certain public exhibits where the layout is a small portion of the room. I'm thinking in particular of a layout that Nabisco built several decades ago to display at their world headquarters in New Jersey each Christmas. It is currently on display in a large gallery in the Morris Museum in Morristown and still works as designed. Each side of the rectangle has a somewhat different effect. Another place where straight sides work is in retro-fabulous layouts - the "official" table in the Lionel catalogue is unabashedly rectangular, for example.
In any layout where space is a concern, including typical home layouts, I prefer curved sides for a number of reasons.
First on my list is the overall formality of the space - straight is formal and curves are casual. Curves soften up straight edges. A formal garden is straight and symmetrical. A casual garden isn't. A formal house is blended to its property with curved mulch beds or similar. Model trains are casual by their nature and need to be presented as such.
Second, large radius curves are a luxury to own. They suggest vast quantities of space. We all know this especially from model railroading. I learned this when my seven-year-old started running laps around the house and wore a path in the yard. That path through my front yard was the biggest, most graceful curve for a mile around and really impressed people, although they couldn't articulate why they liked it.
Third, when you design a layout you are also designing its access paths, like walkways through a park. Show me a park with straight paths and I will show you a sorry, poorly designed park. Joggers and bikers and moms with strollers move in curves. When you pay attention to the access paths in your layout, you make them attractive and easy to move in. You avoid sharp edges and corners that could jab someone in the side.
Lastly is the layout itself. Straight edges imply a layout that was slapped together out of dimensional sheet goods (which it most likely was). Some won't care about that, but the effect is still there. Curved edges imply a layout that transcends numbers like 96 (inches) or 90 (degrees). They imply a layout that is driven by the needs of the scene and not the constraints of the lumber at hand. Of course that implication is correct - when you move away from relying on geometric lines and shapes you are consciously choosing the exact nature of your layout and can build exactly what you need or want. This gives you much more creative control over how your track and scenery elements interact with the edges of your layout. Curving the outer edge of your benchwork to run parallel to your "mainline" or outermost loop looks great, saves space, and is a fine display of model railroad building proficiency. However, if you have the space, allowing places where the track moves further in from the edge of the table creates a real improvement. It suggests "I am not constrained by space" and "My mainline wanders through the scene".
One more closing thought. When you build a model railroad or really anything at all - good design, aesthetic, coloration, symmetry, proportion, focal point, etc. are all free. Materials cost money.
