It's very puzzling that you get nothing to light up even dimly with 9V DC. An LED should do "something" with 9V DC applied thru a 1K resistor in the proper polarity. But to the matter at hand, this suggests to me that trying to drive the signal with "only" 5V DC from the yellow wire also won't work. Yes, you could fuss with lowering the resistor value but I'm not aware of any published information on the capability of the yellow wire to drive more than what it is already tasked for in the pushbutton controller.
So, yes, a relay that applies your 12V AC to your 3-wire signal would surely work if you can get a relay that can be triggered with a low level yellow-wire current. As GRJ suggested, this might mean adding some kind of transistor buffer.
Frankly, if your 3-wire signal is something that is widely available and you can identify it as such, I think you'll get more feedback. I'm wondering if this might be viewed as a one-off.
