The voltage is one thing, but remember that the human body has a certain amount of capacitance to earth, and a relatively small series resistance. That means that there is energy stored (\$C \cdot V^2 \over 2\$ ) that can be discharged through the device in question.
If the human body capacitance was much less, say 0.1pF rather than 100pF, the capacitance of something like a MOSFET would divide down the voltage from, say, 1kV to maybe 20V, and most MOSFETs would survive that.
The primary mechanism for ESD damage of MOSFETs is breakdown of the gate insulation, which is very thin (as thin as 5 atoms) layer of silicon oxide. Once it is breached by a discharge, the transistor is probably ruined.
Because junctions (as opposed to insulation) can break down without damage if the energy is not too great in relation to the size of the junction and other factors, bipolar transistors and diodes tend to be more more rugged, however RF (especially microwave) transistors and similar diodes have very fine geometry (thin conductors and tiny junctions) so that the energy in an ESD event could cause damage by melting metal or damaging the junction itself.
Most MOSFETs made today either have internal protection or have such high gate capacitance that even (slightly) careless handling is less likely than in the past to cause damage, but caution dictates the use of ESD-safe techniques for most semiconductors, as it is still quite possible to cause damage. It is mandatory for high-reliability (such as space) applications. The big fear is that it could cause damage that would result in a failure at some later date, when repair becomes rather inconvenient and expensive.
In the old days some transistors were supplied with a spring clip around the pins, shorting them together, to be removed after the transistor was installed.