The LM324 has a maximum offset voltage of 9mV (worst case, over temperature), according to the datasheet.
With your circuit, with 0V in, you could have a current of 9mV/500m\$\Omega\$/9mV = 18mA below which your pot would not be able to set the current. So it's not a very good design if you need to set it to less than 18mA. It's luck of the draw- the next op-amp (even in the same package) could be 9mV in the opposite polarity, so you'd just move the pot.
Maximum temperature drift of the LM324 is not specified (it's not intended for precision applications, after all), but it might easily be +/-10uV/°C, so if the board changes by (say) 70°C as the MOSFET gets hot, the current will change by 0.7mV or 1.4mA, so you'd have to readjust the pot. Of course the highest power dissipation occurs at high output currents, so the change is relatively small (1.4mA out of 2A is < 0.1%). A 20°C change in ambient temperature means a change of perhaps (no guarantees) of 0.4mA, which is several percent of a 15mA current. If you only care about 5%, and currents above 20mA, probably just okay.
Another difference between a cheap amplifier and a good one is the gain. The LM324 can be as bad as 25,000 gain (and it changes with temperature). A precision op-amp will have a gain in the millions. The difference will show up in how well it compensates for load or line changes (not a big deal in this case).
The bias current of the LM324 can be as bad as 0.5uA (typical 20nA) and it changes with temperature so if you had a high resistance pot, you could see it change with temperature.
The noise of the LM324 is a fairly miserable 35nV/sqrt(Hz), and it has nasty crossover distortion, neither of which affects you much in this case.
A couple of things (other than being extremely cheap) that the LM324 has that a typical precision op-amp may not have- wide supply range (especially on the high end), though it may not do so well at very low supply voltages, and it's single supply (input common mode range includes the minus supply) which you absolutely require for your circuit.
So there are plenty of reasons to use a decent op-amp if it's required by the specifications. Or you can get clever with the circuit- increase the sense resistor to get good accuracy for low currents, but to get wide dynamic range, a good amplifier (and other techniques such as good resistors and good layout) may be worth it. For just hacking around and if your current range not huge (minimum to maximum), an LM324 is certainly acceptable. There's no point in using a $5 op-amp if a 1-cent one will do. On the other hand, there are some requirement for which the best ones are not good enough and one has to resort to discretes and other techniques.
By the way, your circuit may not be stable against oscillation. It can be fixed with some passive components, but loading op-amps with the equivalent of a large capacitance in series with a small resistance is inviting trouble.