Loose ballast is not a big problem, as long as you sift your kitty litter so only the large uniform pieces are used. That said, a light dusting of cheap hairspray or dilute white glue will add to stability without undue permanence.
[N.B.: The "kitty litter" reference is a lighthearted joking tweak directed toward the expensive-ballast enthusiasts.]