From the comments:
One LED's plus pin is connected to the left channel and the other LED's plus pin is connected to the right channel. The minus pins are connected to the other respective channels.
That means that they'll only light when there is a difference between the two channels. If you have a mono recording or a stereo recording with the same signal on both channels then you will have no voltage difference between L and R. I don't understand how you got this to work at all unless you have zero volume on one channel or you inverted the signal on one channel to maximise the difference in L and R.
simulate this circuit – Schematic created using CircuitLab
Figure 1. (a) What you've got. (b) What you need.
If you've got the same signal on L and R then you would have to wire as shown in Figure 1b but would need a higher output level than if you were running (a) with anti-phase signals.
The signal on the right channel is inverted to the signal on the left.
That is critical information and should be mentioned in your question.
So i do get the voltage difference necessary to send the signal.
Yes, you would.
I forgot to mention: If the mono signal solution + different circuitry is more reliable, I'll absolutely switch over.
No, your complimentary (inverted) signal gives twice the voltage to the LEDs in (a) than you would get in (b).
Standard IR transmission uses 38 kHz modulation on the IR light. Your phone's audio won't go above 20 kHz so I don't know how this is working at all.