Usually the flywheel. Pull that, and dead-smooth.
I have converted PRR turbine, 2-rail N&W "J", and a fleet of B6sbs. All are superb for the initial cost. I traded a "Gem" even for the last Williams B6.
Most are smooth out of the box. The "J" had a badly out of balance flywheel.