All microcontrollers have hundreds of milliVolts of GND and VDD bounce/ring/spikes as the busses and latches and memory fetches occur. The "gnd" rail is also the substrate upon which all the FETs are built, and the capacitors (needed by ADC) are built.
Microcontroller manufacturers often allocate separate AGND and AVDD (located right next to each other, for lowest inductance and thus best control of the onchip AGND and AVDD.
Couple things you can try:
1) place 0.1uF SurfaceMount between VREF+ and VREF-
2) place a differential lowpass filter (R+C) in the ADC's Vin- and Vin+ pins; you need 2 resistors and 3 capacitors. Resistors go into the signal paths; the caps go(a) between Vin+ and Vin- right at the ADC, after the resistors; (b) between Vin+ and AGND; (c) between Vin- and AGND.
3) shut off the ADC as you digitize the AC input, placing the samples via DMA
------------------edit------------------------------
For AC performance, the sampling jitter is a big deal. Getting less than 1nanosecond jitter, given onchip clock multiplication, is doubtful. If you are grabbing samples into a DMA, the DMA logic is spiking DVDD/DGND as the flipflops accept the lastest ADC word, and there is no quiet time.
Consider dampening the VDD bypass capacitors, so the VDD does not ring and ring. [The IC manufacturer's silicon evaluation engineers may have favorite PCBs/lossyCaps, but do not document it.] Do this for DVDD and for AVDD. With 10nF and 10nH (leadframe + PCB + cap) and Fring of 16MHz, you need Rdamp of sqrt(L/C) = 1.0 Ohms. Experiment
Some MCUs + ADCs meet the ADC spec.........if the MCU clock is slowed down, so there is 100nS or 200nS of quiet time for the VDD/VREF/substrate bouncing to die away before the binary-search analog comparator makes a decision. A Burr Brown 24-bit ADC was integrated with a MCU rated at 33MHz; the datasheet, read carefully, warned the 24-bit ADC was only 24 bits if clock was reduced to 8MHz, allowing 125 nanosecond of quiet time.
By the way, its your task to ensure VREF settles quickly.
