The GDP2846A module has an in-built 2W amplifier which is fine for a small speaker but not for a pair of headphones as it is indeed way too much.
I've thought of directly taking the DAC outputs (before the amp) to the headphones and call it a day as I've seen people doing with their ESP32s. This would allow me to have both headphone output and speaker which is the ideal result.
But I am not sure it is a good idea. Resistors to turn down the power when using headphones will waste a ton of precious power which due to the small battery is unacceptable. Is my only option to use an external op-amp (which would complicate things a lot and I want to keep things as simple as possible) or is there an easier way out?