The LTC3566, a PMIC intended for battery and USB supplied applications, integrates a battery charger, a power path and a buck-boost converter. For my board, I copy-pasted the schematic found on the last page of the datasheet.
During my test, I supply VBAT_POS with 3.8 V, to simulate a battery. The dc-dc converter should deliver 1 A, 3.3 V, but if I connect to 3V3 a 5.6 Ω resistor (580 mA), the 3V3 drops to an average of 2.4 V with a 50 kHz, 880 mV ripple (see oscillogram below). In 20 seconds the IC gets hot, increasing of +40°C. A similar behavior happens with lighter loads, e.g. 20 Ω. On the other side, if I apply no load, 3V3 is 3.3 V stable.
Anyway, if I load Vout (the output of the USB/battey power path) with 500 mA, the voltage is stable, so I assume the problem is on the dc-dc converter side. I already tried the type I compensation suggested in the datasheet on first page, but I get similar results.
What could I investigate?
EDIT: Hereafter the Altium layout, with power components selected in light grey. Resistors are 0402. Inside the board and on TOP layer there are two GND planes. C1 and C3 are on top side connected with many vias.