The essential problem with your circuit is that L1 and its entire driver circuit (rc delay, gate driver etc.) are connected across L2, and so get the full flyback voltage across them when SW1 opens. For fastest current decay in L2 this part of the circuit should be isolated from it. The obvious way to do this is to insert a diode between the 'ground' side of the L1 circuit and L2. This isolates not only L1 but also resistors R7, R4 and R1 etc., which are also shunting across L2.
The only problem with this solution is the ~0.8 V drop across the diode when L1 is turned on, which you deem unacceptable.
You can reduce this voltage drop by replacing the diode with an N-channel MOSFET wired 'backwards' so its body diode performs the same function, then tie its Gate to Q1's Gate to turn it on (bypassing the body diode) when Q1 is turned on. If Q1 is off then this isolation FET will be turned off when SW1 is turned off. If Q1 is on there will be a small delay before the Gate voltage drops enough to turn it off.
simulate this circuit – Schematic created using CircuitLab
I simulated this circuit in LTspice and it predicted current through L2 would drop to zero in ~180 μs when L1 is turned off, and 200 μs when L1 is turned on.
Here's another way that could also work. L2 is controlled by a FET which normally is turned on, but turns off when SW1 opens and removes power to the circuit. It is similar to the other circuit except the switch is connected to the Source side of the FET, which is turned on with a fixed voltage. The advantage of this configuration is that the FET doesn't have to pass current from L1, so its RDSON can be higher.
simulate this circuit
R1 and C1 help the FET to turn off faster. R4 and R5 quench possible oscillations in the coil caused by parasitic capacitance resonating with the inductance (I put 2 resistors in series to handle the high voltage).