Instead of WD40, I would use penetrating oil - it is designed for loosening rust nuts/bolts, and may help you here. You local hardware store will have it.
I am not familiar with this, but I assume you are trying to remove the inner part, not the outer part. This being the case, I would try to find a screw that would just tap itself into the hole, but can not to big to expand that inner cylinder you are trying to remove. Once threaded in, and giving the penetrating oil a chance to work, you can try to grab the screw and pull out.
Combining @700E and @David Johnston ideas from above, if you have a dremel tool, you can purchase burrs as in the attached screenshot. Using a very small burr, you can grind out the zinc, or as I would try, make a slot in the zinc vertically down the hole to allow you to then use a small screwdriver or awl the break the zinc, the slot allowing you space to deform the zinc into.
