I can't say that I've seen much about a connector "system" except for some very specific topics. For example, FasTrack can be wired using crimp on type connectors that are 0.110 inch wide.
Most of the discussions center around wire gauge usage because of voltage drop through the wire. It all depends on the size of the layout. For short distances, 4-6 feet, 18 AWG is workable. Intermediate distances of say 6-12 feet can use 16 AWG. Then if you go from 12-25 feet, 14 AWG is the recommendation proposed by most. Beyond 25 feet may require 12 AWG or larger to get the power to where it can be distributed to the track.
Once the power is in the general area where it will be connected to the track, a smaller gauge (say 18 AWG) can be used from a terminal block to the rails, but not over 6 feet.
For track power, use stranded copper wire, not copper coated aluminum (CCA). The aluminum will give a higher voltage drop over distance.
Switch control wires can be telephone type wire from 24 to 28 gauge, solid or stranded. Sensor wire can be in a similar range.
Terminal blocks with screws to make connections between wires are used extensively. There are other options like the suitcase type connectors and the WAGO type as well.
