I built myself a headphone amplifier using the LM49600 and a PGA2310 for volume control. This part of the circuit is fairly standard and I don't think it's the problem. I wanted it to be powered by a simple 5V wall wart or an USB charger. I used a TPS65131 followed by some linear regulators and a lot of filtering. There is a single ground plane.
There is a faint but unacceptable audible noise when the 5V input is connected to a wall wart and the audio input is connected to a desktop PC. The amount and frequency of the noise changes with different chargers or power supplies. The noise becomes much worse when powering from one of the USB ports on the same PC.
It goes away completely when powered from a USB power bank, or when using a phone as the audio source.
What's causing this? My best guess is common mode noise coming through the transformer capacitance or the y-cap in the wall wart.
How do I fix this? I'd like to do something on my end without touching the external supply to keep it universal.