It discharges very slowly because of the 20K resistor between it and the ground.
What you need is a ON/ON switch which connects either to V+ or to GND. When it connects to GND, the capacitor discharges faster, whereas when connected to V+ it charges gradually thanks to R1.
To discharge C1 immediately, you can add a schottky diode from from C1 to SW1 bypassing R1 (not visible on the schematic).
R2 prevents bounces and spikes to hit the input before being swallowed by the capacitor.
Values of R1, R2 and C1 are aproximative and up to your discretion.
The Zener diode D1 is better than a voltage divider with two resistors because bounce spikes can be much higher than the supplied voltage. The zenner will always give a predictable voltage while a voltage divider will only divide the voltage it's given and may be higher than expected.
Of course you can replace the zenner diode with a resistor if you want or doubling the zenner diode with a resistor in case the switch would be floating or if you can't put a ON/ON switch.
If you can't use such a switch, then we must think about reducing the impedance of the resistor to ground. Or something else. But then, it will consume a little bit more power because the supply is 24V.
The schotky diode D2 prevents reverse current which happens inevitably with switch bounces.

simulate this circuit – Schematic created using CircuitLab

simulate this circuit
The second schematic shows a solution where ON/ON switch is not an option because we want to use a momentary switch. The idea is to fill C1 with already divided voltage. The zenner limits the voltage to the base voltage. C1 is only 1uF and R2 10K. Discharge time should 40x shorter than with 20uF + 20K.