It's a layout issue. There is a "no-go" clearance zone between the features for this manufacturing process. Either they must be closer or farther away. There used to be glass fiber cleanout brushes with short bristles for cleaning that sort of stuff from manually cut stripboards, but that was 30+ years ago. A blast of crushed peanut shells may also be enough to clean it out.
Any good planar mill CAM program will have a design rule check for minimum feature size, and should be able to remove such slivers automatically. Worth checking into it.
A quick workaround is to increase the pass overlap.