A number of points, in no particular order.
Points also drift arguably into meta topics, as FYI for interested readers who may have related questions. There doesn't seem a better venue to air these points (I would "sticky" them if I could), so I shall include them here for completeness.
This question probably isn't answerable as-is -- not conclusively, at least. An illustrative moment for incomplete information. The best answer I can give, then, is comments regarding likely areas of concern. But this is a frustrating state, both for the asker who receives poor quality answers, and the answerer who has incomplete information to merely speculate beyond. (This sort-of lecture, or lecturing tone, will appear here and there to point out such gaps.)
EMC is a holistic topic. Whether the concern is radio interference at a distance, or interactions between components within a single PCB, potentially the whole system (everything on the board, near the board, wired to it, etc.) is in question. The quality of possible answers is in direct relation to the amount of information given.
Mind that, even given complete information, this venue is a poor choice for EMC questions: the volume of information required is potentially vast (assuming you are free to publish such data in the first place: this is also an open venue), demanding comparable volunteer time from the community to answer; in turn making the average answer poor quality, or a complete answer unlikely to be written.
For more private, comprehensive, and confident / likely-to-succeed answers, your preferred business alternative is hiring or contracting an expert in the field, preferably a local one who can work with your entire system as-is, and who is well equipped to analyze the system, and modify the design to suit (or to at least propose modifications). Such a solution will be expensive, but so it is, merely the cost of doing business. (I'm somewhat making an assumption here, that information is hid for business reasons. But, again: incomplete information leads to speculation; I can only guess whether this is the case.)
Vias: why such large groups of vias? Fewer stitching vias are needed, and in more strategic locations, to stitch mostly-contiguous planes together. They have come at the price of sawing up your power planes, which are especially thin around the 1V2 supply connection, under the ADC (both rails), and to the nearby connector (JTAG..?).
Stitching vias should be used judiciously, at modest density as dictated by the maximum frequency of interest or signal harmonics / edge rates, should be prioritized around sources/sinks where the low impedance is necessary, and avoiding routing hazards on all layers where their negative space (the clearance enforced around them) would interfere.
The inner planes, as a hazard, seems to have been missed as oversight; perhaps an automatic stitching placement tool was used, which does not account for polygons. Regardless whether automated or manual, final review is the duty of the designer!
Power: you mention ferrite beads on LDOs, but have not shown either (schematic or layout detail) after comments asking for them, so I am unable to make any statement to their effect; only that this is a completely unknown region and may be in need of consideration.
Routing: most of the signals look reasonable: short, direct paths (length tuning aside), spacially separated, SDRAM off to one side, ADC the other. That leaves these:
The entire analog front end is not shown, and it is a mystery where these signals go, if they carry common mode interference into the front-end, or what. Several of them cross over an apparent void in the ground plane, potentially radiating noise to nearby circuitry, and compromising their signal quality.
Note that, even signals that are set to logic-low (pin connected to internal FPGA GND at the IO pad), emit some noise due to FPGA GND not being exactly PCB GND; and for that matter, PCB GND not being ideal itself, but having some switching noise in the direction of the SDRAM. Low-speed signals should have that bandwidth enforced by filter (RC, ferrite bead or better), to avoid transmitting into nearby circuitry, or radiating into space.
Again, whether this is an issue, is impossible to say without a wider layout view, without assembly details (maybe this is inside a sealed metal enclosure so that radiation is moot), without susceptibility or emissions levels, etc.
My strongest bet is on these signals interfering with the analog front-end, or the AFE itself having design issues that haven't been accounted for (beyond the scope of the question as asked).
Interference: it appears to manifest as a DC offset, or perhaps that's aliasing due to common clock sources. The fact that it aliases to AC in this figure, https://i.sstatic.net/jhS4F.jpg suggests it is aliasing, and you're effectively sampling at different phase shifts around the wave. Whether this is due to VREF stability, AFE interference, RFI rectification, or other unanticipated means, I cannot say. You might ask a separate question regarding the AFE, including the context of, or in relation to, what (digital) signals might be routed near or into it.
Notice you have only given long-term averages as evidence; whether the individual samples are complete gibberish (perhaps a timing issue?), or indeed reading correctly such interference at the ADC input (or REF), isn't known.
Which, timing: it seems bobflux has made some progress in discussing timing constraints; whether this is a matter of reading the port correctly in the first place, or phasing the clocks to just avoid a burst of interference, isn't yet clear. But this is another example of out-of-scope topics or omitted information; it's also leading to (at time of writing) an extended comment discussion, which is generally discouraged. (Again, this is by design: brief, self-contained questions and answers are preferred on this venue; if discussion would suit you problem better, I would suggest the EEVblog forum, among others. Mind, you might not get answers at all, it's just a discussion forum; but perhaps it isn't a problem that needs a conclusive answer.)
While I may not have a resolution to the problem, I hope these points at least prove helpful; cheers, and good luck!