Smooth is more important than removal of every speck of old paint. Some might be acting as filler in a low spot. You are looking at an extra coat or so of primer or color depending on the color contrasts. Primer is better as it is less likely to react to the old paint, and should seal it from topcoat reaction. Old paint is generally a risk in that way, but sometimes worth the gamble to try vs fighting the old stuff to the last atom.
Primer should be fine sanded or scotch padded, same as any other coat of paint, especially hard primers or those with any gloss when dry (epoxy & etching normally). Even clear gets fine sanded for a topcoat. It gives micro scratches for a bite when set. (sanding dust in clear will haze or turn white.. always damp dust with the solvent and rag then flash off wetness before you shoot)
