ADC strange behavior (ATmega)

Question

I am building a hobby oscilloscope on ATmega16. I am testing ADC by generating a square wave at 2kHz with small amplitude and applying it to the ADC pin through a capacitive input with a voltage divider to shift zero-level.

This is how the signal looks after being sampled by ADC:

It looks fine, but after I start shifting zero-level by adjusting the potentiometer I get these strange spikes:

After shifting level a bit higher it looks fine again:

And then the pattern repeats:

Overall, I get 3 ranges where the spikes appear.

Some info

• AVR is running at 16MHz
• ADC uses internal 2.56 reference voltage
• ADC is in free running mode at 125kHz (prescaler=128)
• I transfer data via CP210 to an android tablet which is used for real-time plotting. The circuit is also powered from the tablet. I get the same results when the circuit is connected to PC.
• USART baudrate is 500000b/s (UBRR=1)
• AVcc and AREF are not connected. I also tried connecting AVcc to Vcc and adding a .1uF cap between Gnd and AVcc but this had no effect.

Some more info

• this noise does not come from the generator (tested with analog oscilloscope)
• this noise does not come from the potentiometer
• LFUSE = 0xFF, HFUSE = 0x89.

ADC initialization routine

ADMUX = (1 << REFS0); // AVCC with external capacitor at AREF pin
ADCSRA = (1 << ADEN)
        | (1 << ADIE)
        | (1 << ADATE) 
        | (1 << ADPS0)
        | (1 << ADPS1)
        | (1 << ADPS2); // Division factor = 128
MCUCR |= 1 << SM0;
MCUCR |= 1 << SE; // Sleep-mode enabled
ADCSRA |= (1 << ADSC);

ADC data transfer routine

volatile uint8_t adcLow;
volatile uint8_t adcHigh;

int main(void) {
    ....
    while (1) {
        if (ADCSRA != 0x00) {
            USARTSendByte(adcLow);
            USARTSendByte(adcHigh);
        }
    }
}

ISR(ADC_vect) {
    adcLow = ADCL;
    adcHigh = ADCH;
}

Answer 1

The spikes are clearly clipping to very particular values (probably a power of two) or you are using signed integers that are too small to contain the whole value.

You didn't supply full source code so I'm guessing here. You probably try to fit the measured value into a too small integer. Make a dump of the vaules you get and find the peaks in them. Then look at the binary representation of these numbers and see if you can find something in common.

With quick estimation the distance between two peaks is about 1.28V, a little bit too coincidentially close to the size of a signed 8 bit integer.

I advice to use type definitions like int8_t [-128:127], uint8_t [0:255], int16_t [-32768:32767], uint16_t [0:65535], int32_t [-2147483648:2147483647] and uint32_t [0:4294967295], which clearly shows how large the variable is.