Inductor datasheet does not spec saturation current, but it does spec max DC current:
With 0.11 ohms DC resistance, at 1.23A it will dissipate 0.16 Watts. An inductor of this size (7.3mm x 7.3mm) can dissipate a lot more heat than this without overheating, therefore I believe the max DC current is not specified with regards to maximum temperature but with regards to magnetic saturation.
So we have a saturation current value, and unfortunately it is too low if you need 1.5A on the 5V output. You should calculate peak inductor current and select an inductor with a saturation current specification a bit above this. Note peak inductor current is higher than average input current, you must calculate it. Here's a guide.
Now the layout...
The chip switches inductor current between the cyan and blue paths which are the "hot loop" and that should have a very tight and short layout. If possible blue and cyan paths should be as close as possible to each other.
Also the inductor current path (green) and the path between GND of input caps, output caps and the chip should be as tight as possible.
This layout is pretty bad, I think the ground pour made you think "GND" was connected everywhere but look at the path the current has to take through the ground pour... this will radiate a lot of EMI and it probably won't work at all, considering the high switching frequency.
For this kind of layout, you should place the chip, input caps and output caps in such a way that the GND pins of the three components are connected together very close and tight. Then place the diode and inductor.
Highest priority is the hot loop, then inductor and input cap, then the resistors, which are low priority.
If you have a ground plane on the other side it's easier because you can use vias, but if you don't, then you have to make a real good single side layout. At this switching frequency a ground plane is pretty much required...
You can look at layout advice in datasheets of other boost converters in SOT23 packages. Manufacturers like TI or AD usually provide good hints.
Edit: quick'n dirty layout fix
Pushing high di/dt current (in this case a >1MHz current square wave) into an inductive path (long winding path) radiates lots of EMI but it will also cause voltage spikes to appear between various points labeled "GND" along the way. This can make your DC-DC chip misbehave, for example if the spikes find their way in the feedback the chip might think the output voltage is wrong and do stuff you don't expect (like shutting down).
You can scratch the soldermask, drill some holes and solder wires through the board to the ground plane on the other side to create vias (green dots on the picture) on the important GND pins. This will make a much shorter and less inductive ground connection.
This should fix some but not all EMI problems, and I think it'll probably work if you use a suitable inductor.
If you redo the board you can post a "check my layout" question and ask for advice.
VIN => 3.3V (PSU) | 3.7-4.2V (Li-ion 26650)
I can post up PCB wiring if you need. 
(line on the right goes to VIN ESP32)
