IS it possible for the source device to detect what's connected to the audio output jack? Perhaps by exploiting the presence of the headphone's speaker coils and its response to oscillations?
What you want to detect is the impedance of the "load". In one case the load is the headphones. In another case the load is the aux input of the device.
With the headphones, the impedance will typically be less than 1K Ohm (and as low as 4 ohms). If it is feeding an aux input then the impedance will be greater than 1K, and possibly as high as 1 mega-ohm.
Measuring the impedance is difficult, since the impedance changes depending on frequency. And frequency depends on the music or audio program that is being "played". The normal way is to measure both the voltage and current to the load and then do some crazy complex math to calculate the impedance. Most of the time this is done by feeding the voltage and current into an ADC, and then to a DSP.
There might be a way to do a cheap and dirty version that doesn't require an ADC or DSP, if all you want to know is if it's greater or less than 1K Ohm. I haven't given this much thought, however, since everything I've done has focused on the accurate version and not a quick and dirty version.
If you are free to choose the receptacle in the device yourself, you could use a plug with a split ground connection or mechanical detector pin.
- With a split ground connection: You can then measure whether a small DC current can flow between the two parts of the ground connection. If there is no plug in the receptacle, the two parts of the ground connection will be isolated and no current can flow and vice versa. Note: this will give the attached speaker a slight DC offset.
- With a mechanical detector pin: with these receptacles, insertion of a plug will break a link. With a pullup/pulldown resistor you can feed this to a logic circuit.
For electronic solutions have a look at this article.