I'm currently designing a digital audio mixer. One common thing found on audio mixers is a optional 20dB pad on the input. I've seen some attenuator circuits but all of them require you to know your source impedance. However, with mixers, it's impossible to know the impedance of the connected device. How could I create a 20dB pad then? If I try to design a circuit that has large enough resistance values that the source resistance doesn't matter, it gets REALLY noisy.
If you take a look at existing mixer designs, you'll find that they typically use a passive resistor attenuator, and that the values aren't necessarily huge. Because it's just a pad, the 20dB needn't be exact (the trim control will sort that out).
So, determine the typical source impedances that your mixer is going to be handling, and pick resistor values that give you approximately 20dB of attenuation. If it varies depending on the source, that's not a huge issue.
Most of the sources you are dealing with are going to be 600Ω +/- 300Ω (ish). Don't bother with trying to design for instrument pickups and other Hi-Z sources: that's what DIs/preamps/active pickups are for. Yes, there are microphones in the 5kΩ and up range which you may want to consider. However, any microphone past 15kΩ or so ought to be considered Hi-Z, and I would argue that microphones past 5kΩ are atypical enough to warrant a preamp of their own.
I'm a big fan of Allen & Heath because lots of their mixer schematics/service manuals are easily available (also, the consoles sound great). If you take a peek at some of these, you'll get an idea for design and component values of a typical mic pre.
As long as you want a passive attenuator you cannot avoid the tradeoff you're experiencing now. Some options:
Use the large resistance values, and work on getting rid of noise sources. (Check your digital logic; do you have the right bypass caps? Are the ground return currents passing through the analog input section?)
Don't be passive; use an input buffer (disadvantage: must be able to handle the full unattenuated voltage range without clipping).