This is a combination of a few factors.
Input offset voltage: typ. 0.3 mV, max 3 mV1. With a closed loop gain of 5, this can explain up 15 mV of error, positive or negative.
Input bias current: typ. 100 nA, max 500 nA. The output impedance of the circuit being measured is low (because the resistor under test connects to what appears to be a power rail). In the worst case, it has a very low output impedance, in series with a zero-ohm R4, so the non-inverting input sees an impedance of almost 0 ohms. Meanwhile, the inverting input sees an impedance of 120k || 30k = 24k.
Multiplying this by the worst-case input bias current of 500 nA contributes up to another 12 mV of input offset voltage, or up to 60 mV of output offset. Because the op-amp has a PNP input stage, it will source, rather than sink up to 500 nA via its input pins, which means that the up-to-60 mV of output offset will likely be negative in this case, as the output needs to go negative in order to balance the input voltages.
One quick and dirty way to fix this is to ensure that the impedance seen by the non-inverting input is also 24k.
Ultimately, the RC4580 is an audio op-amp, and is designed with different design goals in mind than yours. It is designed for extremely low harmonic distortion, but doesn't feature things you want here, like low input bias current, low offset, or offset nulling features.
You'll want to consider selecting an op amp that offers low offsets or a means of nulling the offset, as well as a low input bias current. JFET or FET-based inputs will provide this, especially at frequencies near DC (at higher frequencies, the reactive currents flowing into the input capacitance will start to see imbalanced impedances again).
Another concern I see here is the ground mismatch between the negative terminal of the resistor and the ground used in the op-amp feedback circuit. This could also contribute to an offset voltage, especially if the layout is poor, or other large currents flow through either the GND or V_CCS- nets.
1 This figure is listed in the datasheet as a conditional value only valid when Rs < 10k. The impedance of your feedback network is large, it could be even worse, but is not documented.