Butterworth and Chebyshev filter problem

Find the minimum order Butterworth lowpass filter whose attenuation isn't bigger than 2 dB at 5 rad/s and has 10 dB at 8 rad/s, 20 dB at 10 rad/s, and 40dB at 30 rad/s, then do it for Chebyshev filter.

$$\omega_p=5 \frac{\mathrm{rad}}{\mathrm{s}}\,,\, \omega_1=8 \frac{\mathrm{rad}}{\mathrm{s}} \,,\, \omega_2=10 \frac{\mathrm{rad}}{\mathrm{s}} \,,\, \omega_3=30 \frac{\mathrm{rad}}{\mathrm{s}} \\ A_p=2 \mathrm{dB} \,,\, A_{s_1}=10 \mathrm{dB} \,,\, A_{s_2}=20 \mathrm{dB} \,,\, A_{s_3}=40 \mathrm{dB}$$

\begin{align} A_p&=10\log(1+\varepsilon^2-1) \\ \varepsilon^2&=10^{\frac{A_p}{10}}-1=10^{\frac{2}{10}}-1=0.5849 \\ A_s&=10\log(1+\varepsilon^2 V_s^{2n} ) \\ n&=\frac{\log\sqrt{10^{\frac{As}{10}}-\frac{1}{\varepsilon^2 }}}{\log V_s} \\ n_1&=\frac{\log\sqrt{10^{0.1*12}-\frac{1}{\varepsilon^2}}}{\log\frac{8}{5}}=2.8\rightarrow 3 \\ n_2&=\frac{\log\sqrt{10^{0.1*22}-\frac{1}{\varepsilon^2}}}{\log\frac{10}{5}}=3.6\rightarrow 4 \\ n_3&=\frac{\log\sqrt{10^{0.1*42}-\frac{1}{\varepsilon^2}}}{\log\frac{30}{5}}=2.6\rightarrow 3 \end{align}

I think I got Butterworth results OK, but how do I do Chebyshev?

$$|H(j\omega)|=\frac{1}{\sqrt{1+\epsilon_p^2T_n^2(\frac{\omega}{\omega_0})}}$$
where $$\T_n(\omega)\$$ are the Chebyshev polynomials Since you're outside the passband, you can consider $$\T_n(x)=\cosh(n\,\mathrm{acosh}\,x)\$$, so the characteristic attenuation is given by $$\\epsilon_p^2\,\cosh^2(n\,\mathrm{acosh}\frac{\omega}{\omega_0})\$$. In the same manner the Butterworth formula for calculating the order is derived, you can derive the order for Chebyshev from this.
$$n=\lceil\frac{\log\left(\frac{\epsilon_s}{\epsilon_p}\right)}{\frac{\omega_s}{\omega_p}}\rceil$$
where $$\\epsilon_p, \epsilon_s\$$ are defined as you have it shown. Still, your results come close enough and, when rounded, they turn out fine (but that doesn't mean you should rely on them as they are).