Please see the Bode magnitude plot as we steadily increase the input frequency to the filter. Why does the magnitude oscillate? What physical component(s) in the filter circuit is producing that oscillation effect in the magnitude plot?
For example: capacitor reactance decreases (and does not oscillate) as we increase the frequency. But here in the case of a filter the magnitude oscillates.
Please note that this is a quite "normal" expected/desired behaviour. We have several different methods to approximate the ideal lowpass response (this would be "brickwall" response which, however, cannot be realized.)
Higher order filters are "made" from building blocks of 2nd order filters and, to get a steeper response in the roll-off area (above \$f_c\$), quite often a 2nd order filter with a low Q factor followed by a 2nd order filter with a much higher Q is used.
The above are facts but, the clincher is that a 2nd order filter can have a significant pass-band peak such as this one with a Q factor of 10: -
Image from my basic website.
So, if you cascade the above filter with another filter having a very much smaller Q factor (and different \$f_c\$), you get a significantly steeper roll-off but also a little bit of pass-band ripple.
If you then extend this concept to a 6th order or higher (in order to get the roll-off area even steeper), you will introduce more ripples in the pass band.
Ripples in the stop band (Elliptic/Cauer filters) are caused by a similar effect of cascading multiple notching filters.
Filters lie in the domain of rational polynomials, that is, the transfer function is a ratio of the form: $$ H = \frac{P}{Q} = \frac{p_n \omega^n + p_{n-1} \omega^{n-1} + \ldots + p_1 \omega + p_0}{q_n \omega^n + q_{n-1} \omega^{n-1} + \ldots + q_1 \omega + q_0} $$ When \$P = 1\$, we say the transfer function \$H\$ is "all pole" (a pole is a frequency \$\omega\$ where the denominator polynomial \$Q\$ has roots (zeroes)), which is typical of for example low-pass filters with plain L-C ladder topology. Which is typical for all types shown except Elliptic.
When we're designing a filter with analytical design methods, we're doing an approximation process. Approximation means, we could reduce the error between our desired response, and what we get, if we spent infinite effort doing it; but we're engineers, and we have to cut things short for time, cost and space reasons, so we truncate these approximations, leaving some tolerable error behind. That error manifests as some margin where passband is within some ~dB, or the interval between passband and stopband edges (e.g., say we need some 40dB attenuation by 1.5 fc), or how much stopband attenuation is required (minimum 40dB, 60, etc.?).
The most common situation is a flat passband spec, a flat minimum stopband attenuation spec, and some interval between the two (transition band). This is a roughly square or trapezoidal constraint, which curvy polynomials fit very badly into. How they fit, then, is a matter of what else we want to tune.
The standard filter types are optimized for certain specific goals. Butterworth targets maximal flatness (i.e., \$\frac{d^n H}{d\omega^n} = 0\$ at \$\omega \rightarrow 0\$, for a filter of order \$n\$). Bessel targets maximally-flat group delay (a much more difficult thing to express [I don't remember offhand, and explaining how to derive it would be even harder(!)], but suffice it to say, it gives the rounded profile seen in the plot). Chebyshev optimizes for sharpness of cutoff (note the second-steepest passband-stopband transition), at expense of passband flatness. Elliptic (Cauer) further sacrifices stopband attenuation (it doesn't have asymptotic attenuation i.e. attenuation increasing as 20\$n\$ dB/decade), which requires zeroes in the transfer function (non-unity \$P\$).
You can define your own filter types, not so easily in analytical terms (solving these polynominals is far from trivial, even with the pure constraints the standard types use!), but as a matter of hand-waving and tweaking parameters until enough curves look close enough for the purpose -- it's doable.
As for a less abstract explanation -- it suffices to know that, while one capacitor has response \$Z \sim 1/\omega\$, a row of them, alternating with inductors (\$ Z \sim \omega \$), as in the usual ladder network (shunt C, series L, etc.), can have peaks and valleys most anywhere you want. If the peaks and valleys line up on top of each other, you get a rounded response (like Butterworth). If staggered, you can get a passband ripple repsonse like Chebyshev or Elliptic. (If zeroes are present, they can likewise be staggered, giving a lumpy stopband as well.
...If you're not much of a mathematician as well as EE, maybe this won't land very well either. It may be technically adequate, but didactically, who knows.
The ripples in the frequency response of Chebyshev and elliptic filters are due to the use of resonant circuits in their design.
In these filters, multiple stages of LC (inductor and capacitor) circuits are used. Each of these stages can be thought of as a resonant circuit, which has a specific resonant frequency where it allows signals to pass through most easily.
When these resonant circuits are combined in the filter design, their resonant frequencies are arranged to be close together but not exactly the same. This means that as the input frequency increases, it will alternately hit the resonant frequency of one stage, then fall in between the resonant frequencies of two stages, then hit the next stage's resonant frequency, and so on.
When the input frequency is at a stage's resonant frequency, the output signal is at a maximum (creating a peak in the frequency response). When the input frequency falls between two stages' resonant frequencies, the output signal is less (creating a dip in the frequency response).
This alternation between peaks and dips as the frequency increases is what creates the ripples in the frequency response. It's a direct result of using multiple resonant circuits with slightly different resonant frequencies in the filter design.
