If you don't connect point "1" to ground, then instead of an “L-pad” you have a variable series resistor. This will still work to attenuate the signal, but:
At its lowest setting, there will still be some signal. The L-pad goes down to zero/silence — notice that the output side terminals are actually connected together (0 Ω) in that condition, so no signal can get through.
The response (how the volume changes with respect to potentiometer setting) will be nonlinear (or rather, not the curve the potentiometer intends to provide, whether that's linear or not), because instead of operating as a voltage divider it's in series with the load impedance.
In the following schematic, the upper circuit (1) is the ungrounded (series resistance) you propose. The lower circuit (2) is a complete L-pad. If you run the simulation with a “DC sweep” of the "K" parameter of each of the potentiometers you will see the difference.
Note: for applications like driving a speaker, which is a common meaning of "L-pad" in audio contexts, the the result won't be linear because of the loading effect on the voltage divider. The resistance values in this circuit are representative of a speaker system and show this effect. But it's still much more of a useful range than the other.
Note: For simplicity of analysis the circuit specifies a DC voltage source. But since the circuit is entirely made of wires and resistors other than the source, the results (the amount of attenuation and the amplitudes) are the same as if an AC signal were applied.
simulate this circuit – Schematic created using CircuitLab