I have two-ish ideas.
The first is to make use of mechanical indicators. The same kind machinists and toolmakers use. Test indicators are tiny, very sensitive, fragile, use a lever, have low travel, and can measure deviations from zero down to tenths (1/10 of 1/1000th of an inch). The Dial indicators are larger, less sensitive, more sturdy, have high travel, use a plunger, and can measure down to 1/1000"...maybe 0.5/1000". Terminology to separate the two is fuzzy. "Test" usually specifically refers to the small, sensitive, lever type though.
You mention debris and low required accuracy (also vibration) so the dial indicator with the plunger is probably better than the test indicator if it is to measure things while actually running.
The mechanical ones use gears and are like watches. Digital LCD readout versions also exist now and I don't know if they still use gears, or if they use something like hall-sensors or strain gauges. They cannot be built without experts in a precision machine shop but you can buy one along and manufacture parts to allow it to be installed into your setup the way you need it.
I don't know your setup but off the top of my head, direct contact using dial indicators with a roller contact, or a rotating guide around the spindle or engraving bit where the indicator can make contact with to make measurements while running. Or have levers that make contact with the bit on two-axis and apply the indicator to the other end of the lever to keep the indicator safe and far away.
The second idea is to use strain gauges. Install strain gauges somewhere where it can measure X-Y deflection. Then apply various deflections to an installed engraving bit using mechanical indicators to measure the deflection, map out deflection vs strain gauge readouts. I'm not sure if this will have to be done for every bit you use or if one works for all. Certainly the length of the bit is going to matter.
It certainly is a pain to try to do anything at the bit. I'd try to see if you can measure anything at a non-rotating part on the spindle. Even if it does rotate, it'd be easier than at the bit.
I think accelerometers are the least plausible.
There's also wacky ideas like fiber optic whiskers and fiber optic strain gauges. You'll want indicators to calibrate things still and I don't know how repeatable that is but these are extremely sensitive. You probably need to indicators to calibrate things here as well.
One last idea: Laser interfometry. Shining a combination laser-photodiode and measuring the received beat frequency. One for each axis. But aim, debris, eye safety, not to mention the two lasers interferring with each other. Also indeterminant distance relationship as your deflections are many times a wavelength.