Over here in the UK, lab type, switched mode bench top power supplies are popular, models commonly used have an output of 0 to 30 volts, 0 to 5 amps. It's what I use to run a wide variety of vintage locos, from a Bing 4 volt Midland Railway "Spinner" through to a Lionel 402E, via a wide variety of gauge O and 1. The thing to watch with a 4 volt motor, is whether the magnet is any good, most are not, and you will notice the motor is drawing an excessive amount of amps. A good one should need no more than 1.5 amps tops, usually less, a poor one will draw over 2 amps, and will rapidly cook the armature windings.
Cheers, Mark.