A schematic of your quadrature sensor would help troubleshooting.
Let me assume that it is an infrared LED (perhaps two) combined with a phototransistor generating Channel A, and another phototransistor generating Channel B. An oscilloscope may give more info than logic analyser.
Almost all of your noise glitches occur when your sensor is at logic low.
This gives a clue that LED light might be either insufficient, or too bright - only a schematic could tell which. Could also be caused by optical misalignment too. Could also be caused by a poorly chosen value for pull-up or pull-down resistors on your optical detector. However, I do see one glitch going high to low, so this may not be the only problem.
Glitch noise might also be caused by:
-A nearby switch-mode power supply
-Your BLDC motor switching from pole to pole
It would seem that glitches simultaneously affect all three channels (if they're all low, not when high). That suggests an external noise source. Such noise is usually attacked at source. I think your index line should remain mostly low, and usually would pulse high once per revolution. Use that line to judge success of your noise reduction efforts.
Data sheet for CUI AMT11-3S cautions that "GND" should be connected to motor shell with a short wire. That may violate the "star" grounding principle, creating a noise-generating ground loop. This module has only one ground pin, which must serve a few purposes purposes:
-Provides return path for +5v power supply.
-Provides ground reference for serial TX, RX lines.
-Provides logic ground reference for CMOS output logic levels.
-Likely provides metal shell electrostatic shielding
This ground pin must certainly go back to your microcontroller ground along with +5v DC line. Grounding your BLDC motor shell might help, but exactly where to connect ground isn't clear (ATM11-3S gnd?, microcontroller gnd?, chassis gnd?.)
Shielding noise, filtering noise is a far poorer alternative to killing noise at source, but as you say, sometimes you are limited by others saying "can't change my part". Oh, the fun of engineering!