Hi David....I believe your post just answered my question regarding the vertical positioning of the armature/commutator vs. the motor's brush plate.
Sounds like I should add the extra Thrust washer(s) directly above the worm gear. This will pull the armature down, centering the armature in the field coil and providing more space between the commutator and the brush plate.