The main trouble with phase shifts is where the same signal arrives at the listener via two signal paths, and there is a different phase shift in each. In that case, if there is a 180 degree phase shift between the two signals, (and they have the same amplitude) they cancel out, and the signal (at that frequency) disappears.
That certainly can be audible, and it can be worth delaying the earlier signal with a phase shifter to keep them in phase.
It's often more useful to work out the phase shifts in terms of group delays : for example a 360 degree phase shift on a 200Hz signal is 5 ms (1 full cycle) and a 1080 degree phase shift would be 15ms. Now consider the speed of sound : about 1000 feet per second, so one way to delay the other signal is to move its speaker back ... 15 feet! Probably not practical, but often done for high frequency units where a group delay of (say) 0.1ms (3cm) is required.
Another aspect of viewing phase shift as group delay is that it is independent of frequency : consider that 100 degrees at 20Hz and 1000 degrees at 200Hz are an identical group delay.
Also notice the caveat about the same amplitude above : if two 200Hz signals arrive 180 degrees apart, but one is 10dB lower in amplitude (carries 10% of the power) cancellation occurs, but the sum has 90% of the overall power thus causing a 1dB change in amplitude : few listeners will notice. So beyond 10dB attenuation, it becomes less important to worry about phase shift.
EDIT: The fact is, you have massive delays in your passband, not just beyond your -10dB point, amounting to about 2 full cycles at 60Hz, and as Horta comments, this could be quite clearly audible in some environments. I'd want to see that plotted as group delay vs frequency. If it's relatively constant, I'd consider delaying the main channel by the same amount. (easy if the signal source is digital, not so easy if it's an analog AM or FM car radio).
Alternatively I'd start looking at linear phase filters - there's been a lot of work done on matching phase between LPF and HPF filters (such as Linkwitz-Riley for loudspeaker crossovers) which would seem to be applicable for your case. If you can use these up to say -15dB or -20dB attenuation you can safely add a filter that does anything you want (Cauer, etc) above that point.
EDIT2 : splitting the filters looks like a good idea. If you apply the 20Hz HPF filter to the entire audio band, and its group delay is constant at 15ms across the entire spectrum, that amounts to pressing "Play" 15ms later ... inaudible, and a very different situation from delaying part of the spectrum.
I recommend building it with the flexibility to try both configurations, and testing to see if you can hear the difference. You can do that with normal amplification if the Class D amp isn't ready yet.