If you look at the typical application in section 8.2 of the LM2576HV datasheet, where it shows a 5V output design that matches yours, the maximum input voltage for 5V out at 3A is just 15V.
If you scroll down further, to 18.104.22.168 Application Curves, you'll see the operating ranges of input voltage vs. output current and the part codes that are required. Figure 8.5 for the 5V fixed output variant of the device shows that for anything above 15V input you're going to need inductor code H1000 (a 1000uH inductor) and even then you may not exceed around 0.45A draw.
An ESP32 can draw far more than 0.45A when the WiFi transmitter is powered, so this buck converter part is wholly unsuitable for your needs. This isn't really all that surprising - this chip was designed nearly 25 years ago. A more modern buck converter IC should be able to handle this with far greater ease.
EDIT: I misread the datasheet, the above is wrong.
My suspicion is that the efficiency is what's killing your chips. Figure 6-9 shows the efficiency curves vs. input voltage. The efficiency is not very good for 5V out at high input voltages. The efficiency increases with load (this is typical for buck converters) but it doesn't outpace power dissipation. With 54V input, 5V output, and 3A current, you get about 76% efficiency. Your load power is 15W, which means you're putting 19.74W in and dissipating 4.74W in your converter. That's enough to immolate it.
It's possible that your inductor is not suited for the job either - it's not enough to just pick any old 100uH inductor. It needs to at least have appropriate DCR and saturation current ratings.