The effective load on the op-amp is likely capacitive. From the LM324 datasheet:
Capacitive loads which are applied directly to the output of the amplifier reduce the loop stability margin. Values of 50 pF can be accommodated using the worst-case non-inverting unity gain connection. Large closed loop gains or resistive isolation should be used if larger load capacitance must be driven by the amplifier.
It's not too hard to pick up 50pF of stray capacitance when driving an off-board cable or interconnect.
The fact that a series resistor solves the problem is another clue that supports this.