What you have is a "bridged" configuration where two complementary amplifiers are used on each channel, so that for each speaker each end of the coil can be driven deferentially against the other. This is desirable, because it yields twice the output voltage for a given supply voltage, or four times the power - particularly in a battery powered design where the available voltage is low. More importantly, it also means that a single-supply amplifier doesn't problematically and wastefully put a DC level on the speaker coil.
For your purposes, the problem is that there is no output ground. For each channel, both wires see signal, only in opposite directions. This scheme is designed to directly drive speakers with a pair of wires each, not to drive a TRS jack with a common ground.
The "correct" solution would be to use an audio transformer per channel to convert the bridged, or differential output, to a single ended output relative to ground which you could wire up the way you hoped to. Realistically, nobody does that. The whole point of these amplifiers is that they don't need transformers the way early generation designs did.
The "cheat" solution would be to only hook up the positive outputs from each channel, through a DC blocking capacitor, and hook the TRS ground to actual ground. You'd probably leave the negative outputs unconnected.
If you look at the data sheets/application notes for various modern gadget amplifiers, you'll see that some (most?) are actually specified to be used this way when driving headphones. If you have one of those, great. But others probably aren't designed for this, and could behave in odd ways. In some, you might be able to put the series combination of another capacitor and a 32-ohm resistor to ground to crate a bit of a dummy load and fool them into working - probably unnecessary, but hard to know without the specific of protection circuits, etc. You could also try putting something like a 16 or 32 ohm resistor of a few watts rating across each pair of outputs, and then tapping off the positive side through a capacitor to the corresponding pin of the TRS - in effect, your single-ended headphones would then be in parallel with a dummy pair of speakers. Some switch-mode Class-D amplifiers might be happier directly seeing that dummy resistor without capacitance.
Ultimately of course, the module you have just isn't designed for this usage, or at least didn't come with any guidance for using it this way. If there's a an identifiable amplifier chip on it, pulling the data sheet for that could yield some useful recommendations.