Since your toys are battery-powered (3 button cells or 4.5V) they are electrically isolated from each other so you can share the common wire to the stereo amp. The common is the long contact on the plug. The outermost contact (aka tip) is Left, the inner contact (aka ring) is Right.
I don't think you'll encounter this but if you hear clipping or distortion no matter the volume setting of your stereo amp, the toy may be putting out too much speaker voltage(!). Some audio amps are configured to accept so-called line-level inputs where full scale is less than what a toy sound chip running at 4.5V might put out. Obviously there was no consideration in the design of the toy to connect to an external amplifier.
