Get back to basics.
How does it run in conventional mode?
Run the engine by itself on a test track do the same symptoms persist?
Remove the variables caused by your consist, track, and transformer. Heck I'd even turn it upside down and run it at various speeds with alligator clips providing the power. This will give you good visuals as to the performance of the motors. Monitor amperage draw closely.
