I am presuming conventional control, based on your post. Rust and collected oily dirt can certainly affect operations, especially if you are not using command control, but your issue seems like simply a voltage drop on part of your layout. This will be cured by adding some feeders (wires and lockons) to the parts of your layout that has the drop in loco speed.
This is common and typical on layouts of any size, and will also eliminate much of the issue with rusty track pins or the like, as the power now will not depend completely on being fed from a single point.
Just add some feeders. Take it from there.
