The transformers are not usually connected internally to ground or the casing if they offer a center tap, however, the transformer manufacturer could also build them and internally bond the casing of the transformer to one side of the coil, or, to the centertap, possibly tying the feed's neutral from the input to the same thing so the inductive coupling works across the transformer works.
When the coils are resonantly coupled, as in a neon sign transformer, this may not be the case because without labeling of it's input, it could throw the phasing off if the outer terminals were not connected correctly. An out of phase transformer will give no output, so, it is important to use a megohmeter and test to see if any of the taps are connected to the case before beginning.. This willl eliminate case electrocution if using an older transformer without gfci or short circuit protection and potentially save one's life. The danger of the center tap is that the amperage is double of the higher voltage and potentially lethal on the high side of the transformer. The transformers are basically rated at 30 ma-60 ma max. If we touch a 15kV transformer at it's maximum load before cutout, it could easily kill us. Do the math to see the potential wattage at 15kV x 60ma.. it equals 900 watts. That will definitely leave a mark!
The high voltage systems I work with for a living are much higher amperages, but, overall, not all that much higher.. Take a look at the wattage on these systems.. We work with 13,800 volts a lot. and, the line to ground voltage is 7970 volts. 7970 x 10 amps = 79,700 watts. This will blow a limb off and instantly kill somebody. 5kV is even more dangerous because the system feeds more amperage to equal the same wattage in it's output, so, with the 5kV input, which is either 4800v or 4160, to equal 80kW, we need 19.2 amps instead of 10 amps. That's double the damage because the amperage in the system is what decimated the layers of skin , other tissues and nerves, all basically doing sdamage the same way, however, the 5kV may not allow one to walk away from the electrocution, the 15kV may.. Don't go thinking this is possible because high voltage systems are all very dangerous... there isn't one labeled to be safer than the other.