You need a high voltage probe.

Here's one from Fluke with a 40KV peak: http://en-us.fluke.com/products/all-accessories/fluke-80k-40.html

And the same from a common supplier: https://www.digikey.com/products/en?x=0&y=0&lang=en&site=us&keywords=614-1267-ND

I once picked one up for free from someone cleaning out their grandfather's garage. I'm sure you could find some on ebay or something, but if you're going for safety I'd go with buying one from a good supplier.

You're probably going to need a high voltage oscilloscope probe if you want to do this safely and effectively.

Tektronix has a good selection here: https://www.tek.com/high-voltage-probe-single-ended

Sadly, these aren't cheap. The one rated for 20kV (P6015A) will run you about $2000. However, for that price you get to not worry as much about having to debug a high voltage measurement setup. In terms of time saved, it is very cost effective compared to paying an engineer or technician to design their own high voltage test setup.

You will probably also want an oscilloscope to go with it (a cheap one will do), but you can also get BNC->Banana Plug adapters to plug it into a multimeter (though I don't know how well most multimeters work at those kind of frequencies).