Since this is a jfet input op-amp be aware that input voltages slightly over the + rail supply can quickly damage the part. Your circuit seems protected as is, but if you did any substantial poking/probing you might want to verify that the part in the circuit is still ok.
While this op-amp is listed as being a rail to rail part it doesn't absolutely reach the rails. Per the spec the low end will only go to within 5mv of the - rail and 10mv from the + rail. (See the spec sheet section "Output Characteristics", page 18.) Other odd things happen when the output is very close to either power rail.
A potential source of larger errors may be due to the input error voltage when the output is within 300mv of either power rail. (See spec sheet figure 13, page 12). While the error is normally in the uV range your minimum output of about 30mv would go well off the chart on the high end. With a 10k load you would need to keep the output at about 120mv above the - rail to minimize the error, (I'm extrapolating the chart between RL=20k to 2k). This chart uses an example with +5v-5v supply rails, using only +5v-0v might be even worse.
Also be sure you don't have any significant AC noise on your inputs. If you were expecting all DC outputs maybe you debugged with a DVM on DC. Use a scope to check for AC noise. Just a few mV of noise would be very significant at your lowest input levels. If there is any significant AC coming in you could put caps across the 10k feedback and the 10k going to GND, (of the diff amp). The lower the noise frequency the larger cap values would need to be used to filter it.
You may want to decrease the 2.47v reference a small bit to keep the lowest output voltage farther away from the - rail (0v). Since you say your 2.47v reference is buffered by another op-amp you could put a multi-turn pot ahead of that input to give you an accurate way to calibrate the output voltage range.
Too large a cap on the final output (going to the A/D input) might also cause problems for this op-amp.
