Another way to trim rubber is with an abrasive. I would cut a wood block for a snug fit inside the diapragm. That will allow you to hold it perpendicular to a belt sander or flat rotary disk sander or even a sheet of sandpaper fastened to a flat surface. If you have a power tool the rubber will come off quickly so a light touch is needed.
Pete