Audio signals only go down to 20 Hz. That means DC is irrelevant. A simple way to deal with audio signals with different DC offsets is therefore to capacitively couple the signals. That effectively high-pass filters them, which eliminates and DC offset.
For example, if the input of your summing amp has 1 kΩ impedance, then you would theoretically need 7.96 µF in series for a 20 Hz high-pass rolloff frequency. That would cause 3 dB attenuation at 20 kHz.
3 dB attenuation at the ends of the frequency range is acceptable in audio. However, there may be multiple DC blocking filters in the whole chain, so I usually set my rolloff frequencies lower. In this case, a 15 µF capacitor would be better. That allows for a few other DC blocking filters with the same rolloff in the whole eventual signal path, and still only attenuate by 3 dB at 20 Hz.
Be careful with the capacitor selection. Normally you'd use a multi-layer ceramic at this voltage and capacitance. However, those have two undesirable effects for audio.
First, the capacitance is somewhat dependent on voltage. At higher voltage, you get lower capacitance. This is a non-linearity that introduces distortion. One way to minimize this is to use capacitors rated to a much higher voltage. A 30 V 15 µF ceramic capacitor used in the ±5 V range will be pretty linear, for example.
Second, some ceramic materials exhibit significant piezo-electric effect. Unfortunately the ones that allow for higher capacitance do this more so. That means the capacitor acts a bit like a microphone. Shock and vibrations the board is subject to can end up in the audio signal.
In this case, electrolytic may be a reasonable option, especially if the polarity of the DC offset is known to always be one way. Otherwise, it gets more tricky.
Using a higher input impedance in the summing amp allows for lower capacitance. For example, with 10 kΩ instead of 1 kΩ, you can use 1.5 or even 2 µF instead of 15 µF. That opens up more possibilities in capacitor selection.