As Bob says, the circuit you want is called Automatic Gain Control, or (more generally) a "compressor" (because it compresses the dynamic range, which is the difference between the loudest and quietest parts of the signal). The term "compressor" is more general, as this type of circuit is also often used for other purposes, for example, controlling the peaks of a signal, or "fattening" a specific sound such as a vocal or a snare drum in a mix.
Before diving in, you should take some time to understand the basic topology(s) of such a circuit:
- main signal path with gain control element
- detector circuit (derives a DC control voltage from the signal)
- typically, some processing of the DC control voltage before applying it to the gain element
and the important parameters that result:
- threshold
- ratio
- attack time
- release time
- "makeup" gain
There is a ton of info on the web about this, much is more aimed at people learning audio engineering (mixing), but here is one paper particularly suited to the electronics engineer. It's by a company that make a number of parts specifically aimed at making compressors and limiters (a limiter is more or less a compressor with a very high ratio). If you decide to continue with this project, I would recommend their app notes and devices highly.
These days such a device is often implemented in the digital domain, but I assume that you wish to make an analogue processor.
For the application that you want you would typically want a very slow attack and release time, however the ideal settings will probably vary a bit from song to song (and on personal taste). But here are some starting values I would try if I was making this:
- attack/release time : 500ms to 2s
- threshold : assuming the signal is peaking to about -2/3dB of maximum, about 10 to 15dB below the maximum.
- ratio : bit hard to say, but enough to give about 6dB of attenuation, probably around 3:1
- makeup gain - about 6dB
(By the way, the terms "boost" and "buck" are usually used to refer to switch mode power supplies. With audio, you would be better talking about "amplification and attenuation".)