If one plays too loud of a sound through a speaker, it can blow - i.e. become visibly torn or stop working entirely. However is it possible that the loud sound will damage it partly and result in the speaker operating at a less loud level; can a speaker blow just a bit?
Sure. Speakers are essentially large coils of wire, and when they "blow", it's generally that the coil is taking too much current and isn't being cooled enough, and is damaged. If the speaker is completely destroyed, this may be that the wire in the coil is broken or shorted, so that it doesn't operate.
However, one can imagine a scenario where something else in the speaker is damaged, such as the surround (the rubber coil that allows the cone to vibrate freely) or the spider (cloth "guide" inside the speaker, behind the cone) or leads leading into the cone are damaged, but the coil still operates. Any of these issues have the ability to reduce the volume level of the speaker.
It's worth noting that it's usually not the "loud sound" that damages the speaker, it's the large inrush of current that the voice coil cannot handle, the excursion of the cone (moving farther than designed for), or natural wear and tear on components.
It is possible to damage a loudspeaker by driving it too hard. However, the symptoms are more likely to be no output at all, or excessive distortion (scratchy sound or rattling noises) rather than reduced output.
But let's understand exactly what you have damaged first. If your "loudspeaker" is something like a Hi-fi speaker, with separate drive units for low and high frequencies (woofer and tweeter) then it normally also has a circuit between them - a crossover - to split the signal into high and low frequencies.
In this case you might have destroyed just one of the drive units : an ear close to each will tell you which; or one of the components in the crossover, so that less power (or none) is fed to one of the drive units ("speakers") If the crossover is at fault, that is a much simpler and cheaper repair than replacing a drive unit.
In some speakers, each drive unit has its own amplifier (aka "active" speaker system) with the necessary crossover filter built in : they can be complex to repair.
You can damage the suspension of a speaker so that the voice-coil part of the cone will rub. Generally this makes a bad scratching sound, but it's possible it wouldn't be that obvious.
A speaker contains a number of components which are able to tolerate different levels of abuse. If a speaker is abused, the exact construction and nature of the abuse may affect which components fail first.
A single speaker element typically has a cone of some sort of material to which a coil of wire is affixed. Current in the wire will cause the coil to generate a force parallel to the axis of the cone; this should in turn cause the surface cone to move, thus displacing air.
For a speaker to produce good quality sound, it is important that the parts of the cone which are supposed to move, move smoothly as designed, without unwanted mechanical interference. This will generally require that parts of the cone be stretched, compressed, and/or flexed. Such stretching, compression, and flexing can cause concentrated stresses, which can easily cause tearing or mechanical failure of part of the cone. A speaker with a torn cone may behave electrically much as it should, but will no longer move smoothly and will generally give lousy sound.
Another part of the speaker that can fail is the joint between the coil and the cone. If the force on the coil exceeds the strength of that joint, it's possible that it may fail partially or completely. If this happens, motion of the coil will not be properly conveyed to the cone, causing the speaker to sound rather weakly. Additionally, the speaker will likely be more effective at producing sound when driven in the direction that would pull the coil against the one, than when driven in the opposite direction. Thus, the sound will not only be quieter than it should, but distorted as well.
Finally, it's possible for the coil to fail electrically if it is overheated sufficiently to melt the wire and/or insulation. While such a failure could occur without stressing the cone or joints sufficiently to cause mechanical failure, most speakers that are said to be "blown" have actually suffered mechanical failure rather than electrical failure. It's possible than a mechanical failure would cause a consequent electrical failure (among other things, speakers convert electrical energy into sound and heat; a mechanical failure which reduces the amount of power converted to sound may increase the amount converted to heat) but in many cases, speakers are more likely to fail for mechanical reasons than electrical ones.