As follow-up, in case anyone else has this problem... I think I fixed it.
The head (black) of the semaphore moves up and down on, as well as pivots slightly around, the pole (silver/gray). Depending on the exact location up/down and the semaphore arm wouldn't drop into the bottom, red position.
After much fiddling of the semaphore head on the bench and getting it to work, I installed everything on the layout. A bit more gentle fiddling once on the layout, and it now seems to work but still very temperamental.
