What is the coil voltage rating on the relay?
What transformer are you using?
Have you created an insulated outside rail section (or sections)?
Do you know which two terminals on the relay are connected to the coil?
To learn by doing, I suggest that you get yourself a cheapo analog or digital voltmeter (about 12 bucks) and some test leads with alligator clips. Using these, you can do each step temporarily until you begin to see the circuit being developed. This method also ensures that you don't burn out anything if your wiring is incorrect. Then do the final wiring in a neat and orderly way. Digital meters are good for checking or comparing steady readings, while analog ones are good for watching the operation of a circuit. The movement of the needle is easier to see than the changing display on a digital one. I use one of each for these reasons.
Start by creating the insulated rail section. Connect one terminal of the coil (it doesn't matter which) to the insulated rail. Connect the other coil terminal to a constant voltage source on the transformer that shares a common return with your track power. (This is the transformer terminal that already goes to the outside rail of the track.)
Take a rail car and place it on the track at a point away from the insulated rail. With the transformer on, roll the car onto the insulated rail. The relay coil should operate. (You will hear it and if the relay cover is transparent, you will see the relay armature move.)
Now it's time to wire the contacts of the relay to the signal terminals. You only need one set of the relay contacts, what we call the "transfer" or "form C" contacts. The normally-closed contact, often labeled as "NC" gets connected to the green signal lamp terminal. The normally-open relay contact, often labeled "NO" gets connected to the red signal terminal. The common contact, often labeled "C" or "COM" gets connected to the transformer post that provides 12 - 14 volts AC ("hot" side). The middle terminal of the signal gets connected to the return of the 12 -14- volt AC source ("return" side).
The relay source and the track source must share a common return. The juice for the signal lamps themselves can either share a common return or not. It can even be from another transformer.
If you can let us know how much you already know about electricity and relays, and what transformer you intend to use, we can continue.
