If you look at modern commercially available X-ray tube power supplies (50 kV for instance) you won't find a design that uses a transformer that generates the final voltage. What they do is use a transformer to generate maybe 8 kVp-p then use a cockcroft-walton multiplier to take the lowish AC kV produced by the transformer secondary up to 50 kV DC.
This means that the transformer secondary insulation (layer to layer and layer to chassis) is not wholly unfeasible for a ferrite core. The one I designed used 50 kHz as the switching frequency and a split primary driven by two MOSFETs controlled from a Linear Technology chip.
I am looking to make my own flyback transformer that can put out a DC
voltage from 0 to -40 kV DC and a power of 400 watts
A flyback transformer topology isn't powerful enough for 400 watts - you need a push-pull configuration at the primary either from two MOSFETs and a split/centre-tapped primary or, a H bridge primary coil driver.
This means that from a 24 V DC power supply, the primary p-p voltage will be 48 volts and, with a secondary voltage of (say) 7968 Vp-p, you have a turns ratio of 1:166.
The implication of this is that with 6 turns on the primary, you need about 1000 turns on the secondary. This means that it can fit inside the core geometries of large commercially available ferrite cores along with all the insulation layers that prevent voltage breakdown.
Preferably, the voltage input for such a design should be 0-24 V DC
with a transformer output that correlates with the input (basically
resulting in an adjustable output based on an adjusted input if
possible).
The only feasible way that I know of (other than buying an oil-filled old-fashioned X-ray transformer if you can get one) is to run the push-pull stage from a variable DC supply whilst the gate drivers were fed from a fixed supply voltage. The variable DC supply can be turned down to 0 volts resulting in zero output or turned up to 24 volts to give maximum HV output.
One final note about the cockcroft-walton multiplier after the transformer. It has to be designed very carefully and then resin filled to prevent flashover. The PCB should be slotted around each stage to prevent high-voltage-arcing along the surface (aka HV tracking) thus also allowing the resin to get to (and around) every component.