Most of my experience is with larger motors, but some may apply to Lionel motors. The commutator should not be polished, it should be chocolate brown. A commutator that is a raw copper color is wearing the copper. The brushes are probably too hard. A commutator that is black has carbon on it. The brushes are probably too soft and are leaving carbon on the commutator. If a commutator is chocolate brown it should be cleaned with a solvent and not sanded.
If the commutator face is not smooth it should be sanded using very fine garnet paper. The harder abrasives, like aluminum oxide, will embed into the copper and increase brush wear. The method I use is I have a very flat steel plate with an 1/8" clearance hole drilled in it. Punch a 1/8" hole in the sheet of garnet paper, line the holes up, and put the armature shaft through the holes. Rotating the armature will sand the commutator face keeping it flat and square with the shaft.
On larger dual direction motors some times there is a problem with wear on the sides of the brush box and on the sides of the brushes. This allow the brushes to cock a little wearing a slight angle on the face of the brush. When the motor runs the other way the brush cocks the other way and wears an angle in the other direction. This can be seen when the brush is removed as a line across the face of the brush, which would sort of be the ridge of the roof shaped brush face. If the motor is run mostly in one direction this ridge line will favor one direction causing the motor to run with a little less power in the other direction. Under size new brushes would cause this same problem.
The ideal location for the commutator bars is to slightly lead the armature coils. This will improve commutation and add a little power. But in a dual direction motor this is, of course, not possible. Lionel does this on some there accessory motors that run only in one direction. On a locomotive motor the commutator slots need to be centered on the armature coils. The commutator is a press fit on the armature shaft and it is possible for the commutator to move a little from its proper location, limited by the slack in the coil leads. If this happened it would result in different motor operation depending on the direction. Checking that the commutator is properly located would be worth while. I guess the commutator could been mislocated when installed at the factory. In this case it's mislocation would not be limited by the coil leads. Lionel has been known to have manufacturing errors in the past.
On most post war motors the e-unit reverses the power to the armature coils by reversing the connections to the brush holders. If there is short to the iron near one end of the coil, it will make little difference in one direction but will be significant in the other direction. Testing for resistance from the commutator segments to the motor shaft should detect this. However, most meters are using a fairly low voltage to check for resistance. When looking for insulation failures a Meger is usually used to get the voltage high enough to find insulation problems. I do not know what to suggest here.
Another potential problem is that between the under side of the commutator and the armature coils there is an 1/8" thick felt pad. I assume it there so they have something to press the commutator on against without damaging the armature coils. Over the years this felt pad can get a fair amount of oil and carbon dust on it making it conductive to some degree. It could be causing problems by allowing a conductive path, bar to bar, and bypassing some armature current. When ever I remove a commutator I put a new felt pad in.
Please let us know if you solve your mystery.