I can see 2 major reasons:
1. Software issues
With the CPU, RAM, PCI cards etc... they are enumerated and configured by the system upon boot-up (by the BIOS or equivalent firmware) and by the operating system (which should support hotplugging/hot-swapping).
USB, HDMI, Ethernet etc... are designed for hotplugging and thus there is appropriate support in the operating system.
2. Hardware issues
If the interface and power pins are not designed to provide hotplugging, they might see a large inrush current and some noise or distorted signals which can be harmful (e.g. cause an operating system crash) or downright destructive.
This application note explains a little more on how it's possible:
When a line card is plugged into a live backplane, the card's discharged power supply filter capacitors present a low impedance and demand a large, sudden "inrush" current. This sudden high load can cause the backplane's power supply to collapse.
If your connector only carries data (no power) then this is much less of an issue. (I thought Ethernet would be a case, but with Power over Ethernet it's still hot-swappable)