Most LDOs can't sink any current, only source.
This is a simplified schematic of an LM317.
The NPN output stage can only source current, it will become reverse biased if it is required to sink current. Other LDOs have PNP output stages but they also can't sink current, probably until the output voltage goes above the input voltage.
If you have a static load on the supply rail (more than 19 mA in your example - say a 220 ohm resistor) then it may not be a problem as that static load will sink the fault load.
The other thing I have done is put a zener across the clamp supply to provide path for the fault current, a 4.7V zener may be appropriate.
Even if you don't have external diodes most CMOS logic devices have diodes just as you have drawn but internal to the device.
Voltage references often have output stages that can source or sink, but they are too expensive to sue in this type of application.
An active clamp such as below may meet your needs, it won't inject any significant current into your supply - the current will pass through the PNP transistor to ground.
Does the input resistor have to be 1K? Can it be higher? That would reduce the current you have to divert.
simulate this circuit – Schematic created using CircuitLab