This is not necessarily a gain error but an offset error. It could also be an error in measurement.
The output voltage will not be exactly the same as the input but who'll be within a few millivolts - your error seems to be larger than that.
To determine the gain you need to measure the output at at least two input voltages so you can determine the actual gains and offset. Since you are using a single supply amplifier don't use zero volts as the opamp isn't specified to work with the output lower than 20mV (page 5 of data sheet).
The relevant parameter in the data sheet is the "Input offset voltage" on page 5. For this device it should only be 9mV max over the full temperature range.
How are you measuring the voltage? What meter are you using? What are you supplying the input voltage from?
Ideally you should measure both the input and output at the same time. If for example you are feeding the input from a source that has a high output resistance the act of putting the meter on that point could cause it to drop. When you remove the meter it then increases by a few millivolts - the LM324 has a bias current of only ~20nA, less than the 100's of nA or microamps of the meter.