For simplicity, I am assuming that the noise is all conducted and not electric or magnetic field coupled. This could not be 100% correct, but from your description of the symptoms, I think this is likely ok.
The way to think about this to figure it out is to visualize where the currents are flowing. To help with this, it is very helpful to draw out the circuit and not use the notion of ground but rather think of all wires to "Ground" as return wires. As you surely know, all current must return to their source. therefore the positive wire from the PSU or signal source is the signal, and the negative is the return. Once you have this view, it becomes easier to see what is going on with the circuit.
From this we can see that there is a noise source that is sending noise current through the amplifier. The noise source could be either the PI itself or the USB audio adapter.
If we assume the noise source is indeed the PI, we can see that the it draws current from the PSU and must return all of it back to the PSU via some return path. Ideally all the current will flow back to the PSU via the return wire which runs directly back to the PSU, but unfortunately this never happens. In reality you will always get some current flowing in all return paths back to the source. The amount of current in each branch mainly on impedance. At DC this is just resistance, but as frequency increases this translates into inductance.
The issue here is that a small noise current flowing through the amplifier will get amplified by the gain of the amplifer making the problem worse. In order improve this situation, you must bypass this return path as much as possible.
As you likely already know inductance is proportional to loop area, so the first thing to do is minimize loop areas in the paths where you want the current to flow (ie. between the PI and the PSU). To do this you should shorten the wires up as much as possible and keep the loop area small. In your setup the easiest way to do that might be to shorten and twist the power wires to the PI. This is not likely to solve the issue, but it may help alleviate some of the noise.
Next several other things could be attempted to increase the impedance of the return path of the Pi through the amplifier. Here are some ideas.
The PI itself may have poor decoupling, so adding some caps between power and ground near the IC generating the noise could be helpful
Try adding a bypass cap as shown in the diagram. Note that the value is not hugely important here but a low ESR cap larger than say 100nF would do fine. Ideally this would be a small surface mount ceramic cap.
You could try adding a low frequency ferrite on the USB or audio cable.
If could add galvanic isolation (transformer or optocouplers, but from you earlier post this would not be an attractive solution.
