Links to ESC and motor?
Note that you cannot limit the current and voltage if the load would draw more than the desired current at the specified voltage.
eg At 15V, 10A - load = V/I = 15V/10A = 1.5 ohms effective.
If you load it with 1 ohm and limit current to 10A then Vmax = IR = 10A x 1 Ohm = 10 Volt.
The output current and power of the converter is dependant mainly on the MOSFETs used, on their heatsinking and on the main inductor.
The MOSFETS datasheet here SHOULD be capable of far far far more than 10A continuous if properly driven and properly heatsunk providing the datasheet and parts are genuine.
Any inductor failure is most likely to be due to saturation.
Saturation can be tested by examining the switching waveform when the high FET conducts as current output is increased.
A buck converter inductor when the input FET is on should have a linearly increasing current. This can usually be see moderately well by viewing Vds with an oscilloscope - where V = I x R ~= Id x Rdson. While Rdson will vary with current and other conditions this is usually 'good enough'.
When the inductor starts to experience saturation the slope of the Vdson voltage waveform starts to increase at its upper end - until it rises extremely rapidly as full saturation occurs.
If your converter shows no saturation at say 8A but starts to 'curve upwards' at 10 - 12 - 14 - 16A ... then the inductor is a likely culprit. As you have two cores (the old dead one and the new one), you could wind an inductor with about twice the saturation current using both cores and the number of turns required for the same inductance. (Number of turns depends on a few factors).
Possibly OK is to just solder the second existing inductor in parallel with the first - solder tack it to the bottom of the PCB - run at low current, and ramp up carefully to test.
Did the FETs die?
Does the heatsink get super hot?
Adequate heatsinking is essential. Adding more is easy enough.
If FET drive is adequate then Rdson of 8.5 milliohm max (so allow say 0.015 ohm) then at 10A Pd ~= 1.5 Watt and at 20A = 6 Watt = easily handled.
What capacitors died - and why?
Others noted that some capacitors are 'bulged'.
Which capacitors failed? - and, why?
What is their voltage rating?
I think I might not have explained the question properly, or you missed out the detail. I do no want to make a constant current source. I want to limit my current at a certain value. I want to achieve this thing (youtube.com/watch?v=8uoo5pAeWZI) for my own buck converter
No & no.
My answer may be of great value to you if you read it properly and do not miss out on the detail.
I at no point mentioned a constant current source EVEN THOUGH once the current limiter goes into operation, a CCS is what you will and must get.
If you address the points that I spent significant time raising and answer the questions you might get a really good result.
Vital is the 2nd paragraph, starting "Note that ...".
You MUST appreciate this point to understand the result that you are seeking.
Once your load reaches say 10A then IF you do not want it to shut down or REDUCE current then you are ASKING for a CC cct - that limits I to 10A.
If the load resistance continues to fall (IF) then Vout MUST fall to ensure that
I = V/R = 10A.
So in that mode you DO want a CC source.
This mode of operation is often termed CVCC - constant V if load allows, constant C if load does not allow Vset to be reached.
Next: My comments about the inductor are crucial.
You (seem to?) have ignored them totally.
I explained that the inductor is probably your limit and how to deal with it and suggested some options.
Your feedback would be useful.
If you are going to ask questions, and then suggest that people have not read or understood your question [which DOES indeed happen :-( :-) ], but then not read or understand or respond to input which people (not just me) have spent considerable time on, you are liable to get a lot less benefit from the site than you would otherwise.
If the video does what you want you could build the circuit.
You'll find it drops into CC mode.
Or you could consider the options that I suggest.