A photo would help re "stuff(?)".
OTOH, most of us have encountered significant build-up of 'crud' on the wheels, particularly on metal wheels. If you take a knife point, flat-bladed screwdriver, ...something of that sort..., to the crud on the wheels and it flakes off, it's time to get down and dirty.
You could go right at it with liquid cleaner....but that's going to be overly messy. First get the majority off by scraping...over a trash can, NOT over your wife's prized white carpet! Then get the remainder off with a liquid cleaner...mineral spirits, Goo-Gone, acetone (with lots of ventilation!...like outdoors!!). You might even want to use some nitrile gloves. Since the crud is bound to get on your fingers, take care in how you handle the rest of the car; you don't want to clean more than you have to!
If you have plastic wheels, I'll yield to someone else's recommendation. I don't have them on any of my cars/engines. I imagine you'd have to be a bit more judicious re scraping and using certain aggressive liquids.
It's all part of the NON-glamorous part of the hobby...maintenance.
BTW, if you don't deal with it, the cruddy wheels will just re-deposit their cruddy selves all over your nice clean track. Once done, though, it follows that regular cleaning of the track will slow the accumulation of crud on the wheels. All worthwhile for good operation and happy model railroading!
FWIW,...
KD
