The basic differences:
- 2-rail layouts power the trains from the running rails through the wheels only. Therefore, the axles on all rolling stock must be insulated to prevent shorting the rails. The exception is lighted cars which provide a resistor load and "resistor axles" which are used for current-sensing detection.
- 3-rail layouts power the trains from a center pickup roller providing the "hot" lead and "grounding" through BOTH running rails. The axles are typically NOT insulated.
- Note that I didn't mention the control system. The reason is that the control systems are NOT dependent on the number of rails. If you wire it keeping your hot and ground consistent, it will work with the locomotives equipped with the proper decoders (obviously wired for the rail configuration).
- I don't have a comment on track appearance as that's just a matter of personal taste. Generally, however, 3-rail trains are set up to handle sharper curves than 2-rail trains. To date, every 3-rail piece of equipment can run on O-72 curves (36" radius). Also remember that the "O" number on a 3-rail curve refers to the diameter and 2-rail curves are referred to by the radius. Larger equipment looks better on larger curves, so if you have the space, go for big curves.
As far as most rolling stock is concerned, there isn't much to prevent the use of 2-rail rolling stock on 3-rail layouts using "flat-top" rail such as Atlas, Gargraves, or MTH ScaleTrax subject to curve sizes. Because 2-rail rolling stock uses body-mounted couplers (Kadee, etc.), you must be conscientious of the length of the equipment as the longer equipment will require broader curves. O-72 curves are 36" radius which is considered sharp for scale-sized equipment with body-mounted couplers.
The other issue with 2-rail rolling stock is that the axles are insulated to prevent shorting out the power between the running rails. This isn't a problem on 3-rail track unless you're using insulated rails to trigger signals.
You can't use 3-rail rolling stock on a 2-rail layout. The flanges are too big and improperly spaced and the axles are NOT insulated. Thus they'll hang up on turnouts and short out the power.
2-rail locomotives are powered from the running rails, and with the exception of MTH "3-2" locomotives, can not be run on a 3-rail electrical setup. With the MTH engines (again subject to curve sizes and flat-top rail), they can be set up to run on 3-rail track. I do it all the time at the club (shameless plug: see my YouTube channel).
Hope this helps.
