It seems you have multiple issues. The variation with TIP120 and IFR610: For the IFR610, I suspect that you may not be turning it on completely due to insufficient gate voltage. Is the Arduino output a "5V" output or a 3.3V out? And if so, what is the actual voltage out? It may be that IRF610 needs a higher gate V to turn on enough to avoid a significant drain-source voltage drop at the current you want to control.
The TIP120 will turn on with probably a lower control voltage (combined with base resistor), but it's a darlington transistor and the saturation (turned on) voltage collector to emitter is a little higher than a single bipolar transistor, VCE(sat) in the 1.5 to 2V range.
Basically, measure the voltage at the "bottom" of the LED chain to check these possibilities.
If your switching transistor is accounting for more of the total voltage span then the LM317 perhaps won't have a sufficient proportion of the remaining voltage span to adjust to provide the current you expect. So it will be full on and the current limited by R1, but not due to R1's role in the feedback loop. By reducing R1 you manage to get LM317 and R1 to pass enough current, apparently, but not necessarily with LM317 regulating per se.
Anyhow, that's my guess.
It's not clear from your question what the "20V" and "12V" comments pertain to. It sounds like perhaps you exchanged the 12V supply for a 20V one, and then the LM317 was getting hot, because it was absorbing a relatively large voltage drop.