The issue is that your amplifier has a non-zero output impedance (as real amplifiers are wont to do), and the loading of the two filters (LP and HP) are causing the output voltage to droop.
To fix this, adjust your filter values so that the frequencies stay the same but the impedance to ground is higher. For example, try increasing the resistance in the filters by a factor of 10 or 100 and decreasing the capacitance by the same amount.
This modification will load the buffer amplifier less, and give you you a much flatter pre-filter response.
Because I like LTSpice so very much, here are the relevant simulations (I ignored the MOSFET, it shouldn't change things too much. It will, however, change things a little bit and you should check that):
The original circuit:
The frequency response. Not so good.
Increasing the filter impedances (increasing resistors, decreasing capacitors), but keeping the original filter frequencies (which now need to be adjusted, by the way, now that the loading on the buffer stage has changed. I'd suggest changing C3 to ~47nF to pull the filter corner back down).
As I said above, you'll need to re-adjust the filter corners. Due to the impedance changes (the filter impedances were interacting with the output cap and other things), the filter corners have moved.
More edits:
After inspecting the transient analysis, it's apparent that your transistor amplifier is clipping (you don't want this, you'll get icky harmonics in the filters, and it will lead to a not-so-great color organ).
You'll want to adjust that first stage so you don't get so much distortion. (Or go the easy route and use an op-amp).
