Depends which way round it is. It's almost completely out of the main magnetic circuit, but its fringes will either add, or subtract, a little - maybe 10-15% - to the flux in the field where the voice coil operates.
If it subtracts from the flux the speaker will be slightly - paradoxically - louder (the same way an electric motor goes faster when you reduce the field strength - more motion is required to generate the same back EMF). Probably about 1dB louder which is quite difficult to hear. (Some preamps have digitally generated 1dB steps so you can test what 1dB sounds like, and yes it is just audible).
If it adds to the flux, the speaker will be quieter - but its bass will be better controlled.
I have seen some poor guy flown from Japan to Europe to spend a few weeks supergluing magnets onto the back of 5000 drive units, to increase the flux where the original magnets were weaker than the manufacturer had promised.