The ADC input floats to some DC voltage. If you disconnect your voltage divider from the ADC, you will see the voltage on the ADC float to some DC voltage.
The ADC input looks something like this:
simulate this circuit – Schematic created using CircuitLab
R1 and R2 aren't resistors as such, but rather more like leakage paths from the power rails to the input. The values aren't really representative - they may be much higher or lower.
The voltage divider you've put at the input is in parallel to the figurative voltage divider inside the ADC.
The way to fix your problem is to use an op-amp to do the summing so that it has a low output impedance. That will drive the signal to the ADC to the desired level against the relatively high impedance of its internal voltage divider.
What you want to do looks like this:
simulate this circuit
Op-amps tend to have low leakage, so you should be able to select R1 and r2 so as to get exactly 2.5V when the op-amp is connected, though you may want (or need) to use lower values.
The circuit I've given inverts the signal - I don't expect that to be a problem, but if it is just use a second inverting opamp after the first.
You will want to use an opamp designed for rail to rail operation. The TL081 is the default used in the CircuitLab, and I didn't bother to go looking for its specifications - I don't think it is rail to rail.