Trying to drive a low-impedance loudspeaker in a potentiometric configuration isn't a good idea. The potentiometer value would need to be lower than the impedance of the loudspeaker, and that would add additional load to the amplifier.
If you make the potentiometer value high, you essentially wind up with a rheostat because not much of the current will flow through the bottom leg of the pot.
A potentiometer with a resistance of about 20 ohms connected as a rheostat would give at least a 10 dB control range with an 8 ohm speaker. The pot must have enough power handling rating to carry the speaker's current - probably a minimum of 1 watt.