Time constant in coils L/R is relative constant for a given design sizing, where R is the coil DC resistance ( DCR in spec ).
But Inductor impedance reduces with lower f. So when you choose a lower L with lower loss ,DCR you have to raise fsw to obtain the same impedance ratio.
Then knowing the minimum R of the Peltier device and Xc cap impedance and XL choke impedance you choose the parts for low ripple and low Q which is a tradeoff.
Losses in Peltier devices increase with ripple voltage, so find good specs for optimal tradeoff on efficiency and ripple voltage would be a good start.
You have the wrong filter impedance causing massive circulating currents in the LC part.
25% of the power at 500Hz is dissipated in DCR loss.
If driven at the resonant frequency near 500 Hz the current increases x ~30 times or 28 dB around LC and not into load.
Since I guessed at your Peltier device with no specs, this is all I can say for now until your revise your question with all values of V, PWM f , duty cycle, L,DCR, C, ESR, and Peltier specs
using 8V 2.4A
~ 3 Ohms while LC at 500 Hz is 0.3 Ohms
When you add DCR to L and ESR to C all the energy is dissipated in these parts, so a precise filter spec needs all parameters to choose best Buck converter for a 2 to 3 ohm load
Assuming L 
Solution (Change L to much lower DCR)

- based on new info for Peltier at 26 , 50 'C on hot side.
Lessons learned: Always be conservative on L current limit and choose one greater than max. I chose C because I know that low ESR caps are ~ 10us so from R=T/C, 220 uF ,you can find low DCR caps = 10 us/220 uF = < 50 mohm and reducing L by 5 in same form factor also reduces winding loss by 5.
Imax (Amps) 6.4 6.4
Vmax (Volts) 14.4 16.4
Resistance (Ohms) 1.98 2.30
Power (Watts) 92.2 105.0
Root Cause design fault: Inductor DCR 73 mOhms @ 6.4A exceeds spec of 2.4A ,max as Pd= 3W will get too hot you need Pd < 0.5W so DCR must be 0.5/3 * 73m= 12mohm which results in lower L and thus higher f needed or more ripple . Cap must also have ESR < 5 mOhm as they are less heat conductive.
This however has very high Q at 1kHz so startup will amplify losses 10x in 50 ms to 5 watts or so.
Thus C must also be reduced then ESR increases so losses on here are critical.
Consider required f for ripple at 220 uF 50 mOhm low ESR. I suggest 20 kHz min. Let's see the Q next based on step response from 70 to 90% at 20kHz.

Conclusion
- Looks good and Pd in L is < 0.5W ripple < 3mVpp
But you decide on your specs if it looks good.