Does it only happen when a train is getting close or rolls over switch? If it does there is probably a loose wire, I would remove switch turn over and make sure the wires are in there proper position and not pulled out or loose.
Also what is the part# of the switch.