Suddenly I am baffled, as a hobbyist I made a variable constant current power supply based on LM338. But I never thought what could happen if I accidentally turn it on without load, will the voltage keep rising? I cannot see how, let's say the input voltage of LM338 is 30V, then the output can never rise above 30V in case there is no load, right? Or is going to be some spectacular failure of the device?
You are correct - the output of your regulator can not exceed the input.
What the 'constant current' source will do is TRY to maintain a constant current. It does this by adjusting the output voltage such that it's driving the program current.
You won't have a 'spectacular failure of the device' if you startup with no load. If anything, the regulator will be very happy to remain in that state indefinitely as it's basically doing no work.
I have a CCS circuit using LM317 that does exactly that. It has to handle the load coming and going without issue, and it does! The output of a linear regulator can never go above the input, so no worries there.
The LM317 is a pretty darn stable chip. The only caveat is to make sure you don't violate the power capabilities, as it will happily fold back its current output to protect itself.
Not all the datasheets have this graph, but it is very useful. It's in the Onsemi datasheet. Make sure you stay below the curve: https://www.onsemi.com/pub/Collateral/LM317-D.PDF