You are on the right track headed in the right direction.
To do this correctly you will need more than a potentiometer.
In the following image I circled in red where the potentiometer (33KΩ) would go.
The MOSFET is highlighted in yellow.
The easy way is to replace your MOSFET with an On-Semi CAT4201.
I am driving the mosfet with a PWM ic
The PWM controller in the block diagram has a switching frequency of up to 1 Mhz.
The switching frequency is an operating parameter which affects nearly characteristic of the LED driver design including efficiency and cost. Inductor value is inversely proportional to switching frequency for equal peak-to-peak ripple current.
See Effects of High Switching Frequency on Buck Regulators
LED drivers should run in Continuous Conduction Mode. This is the purpose of the inductor. With slow PWM (under 10,000 khz) the inductor is of little use.
See also Section 2.1 Buck Steady-State Continuous Conduction Mode Analysis in this app note Understanding Buck Power Stages in Switchmode Power Supplies