I'm working on ESP32, so I just put a 1R resistor in series with the supply to check power supply current at boot. Yellow trace, 1 ohm, so 1V = 1mA.

Labels on plot:
RST - reset button released.
1 - it enters arduino setup() and the first thing it does is pulse a GPIO (blue trace) so that shows up on the scope.
4 - frequency is lowered to 80MHz, which does reduce current draw a bit.
3 - WiFi is enabled, so that's not relevant to your application. Current spikes much higher than what the graph shows, but I cropped it for better viewing of the interesting parts.
Between RST and 1, you have no control, it's executing the bootloader and startup code, and it draws about 55mA for 250ms. That's quite a lot for CR2032, it is surprising that it even manages to initialize the serial port and print something. The battery must be new!
Would a capacitor help ? i=C dv/dt, so C=i dt/dv. For the input voltage to drop from 6V to 3.5V (dv=2.5V) during dt=0.25s with i=55mA, this gives C=5500µF, to place in parallel with the battery, before the power switch. Not entirely practical, and not very "thin and fashionable" either, but doable.
To reduce this current, you'd have to hack the SDK to insert some code that would execute right at boot. Perhaps something to lower clock speed, or insert some periodic sleeps to reduce average current draw. Of course, you can't modify the ROM bootloader, so if it wants to read the whole flash to checksum it before running the code, then you're stuck...
I'm not going to do it, but if you really want to run an ESP at 5mA, I guess you'll have to recompile the SDK and bootloader and insert a bit of code at the beginning to pulse a GPIO like I did... If that happens reasonably quickly after boot, then perhaps you can reconfigure it for low power at this point, and get away with a much smaller capacitor.
If you're not using the arduino "framework" but instead the espressif SDK, it could behave differently (it would be nice if it booted faster).
Another potential culprit would be the LDO. It is a micropower LDO, and these tend to be very slow and take forever to react to pulsed loads:

Note, they decided to do this measurement with a 4.7µF output capacitor, whereas the first page of the datasheet has a 2.2µF. Presumably the smaller capacitor value resulted in graphs that were not conducive to more sales. I would recommend 10µF X7R MLCC, that's cheap enough.
Another suspect is the NeoPixel. I have some of a different brand, and when these power up, they flash, and therefore they draw a pretty chunky current spike too. I'd recommend checking for this. They also draw an unreasonable amount of current even when set to dark, so a PMOS on the VCC to turn the thing off when not in use could make the battery last longer. Of course, then, when you want a slow and smooth fade from black, then you have to power it up first, and it flashes, so bummer.
But well, if you're not using the WiFi and the bluetooth, why use an ESP32? OK, the PWM is great, but with a pair of CR2032 you're not running a bunch of high power LEDs either... and the PWM doesn't work in light sleep mode because the peripheral clock is disabled.