Well, if they're rubbing on the curves, that would CERTAINLY cause issues like you describe! Shims sound like the answer.
FWIW, if there's slop in the axles, I cut a shim out of plastic bread clips, it slips right over the axle and solves the problem. Just cut around the opening to make a circular "washer" and slip it onto the axle.