The issue is that this circuit breaker is not designed to trip quickly at just 120% of it's rated current. Consider the graph below:
It is fairly clear, both graphically and with the information in the tables, that the breaker will not trip for at least an hour at 60A.
If you are trying to current-limit your LEDs continuously, a circuit breaker is not the right approach. You either need a current-limiting resistor or some sort of active control, and either way you'll have to dissipate the excess energy as heat. If all you are trying to do is add protection to the circuit against faults, consider a fast-acting fuse or just a more suited circuit breaker.
As for the odd behavior you saw with the LEDs, my best guess is that your power supply was trying to compensate for an over-voltage or over-current condition, and was scaling down either the current or voltage respectively.