The analog telephone system is actually designed to experience this kind of voltage drop when a telephone goes off-hook and loads the line. That's a key part of how it works, and how the provider equipment is signaled that the subscriber has their phone off-hook.
Granted, the mere fact that a proper telephone would do this to the line does not mean that your circuit is functioning as intended. If it is not intended to act as an off-hook telephone as long as it is connected, then something is probably wrong with it. For example, a passive monitoring device would be useless if it held the line in the off-hook condition, since no calls could be placed or received as long as it was installed. And even if the device is intended to act as an off-hook telephone, the mere fact that it is loading the line somewhat similarly to how a telephone would does not mean that the circuit is receiving and transmitting information.