Theoretically, yes you could use a pot to control the brightness of an LED. In practice, not so much.
To start with, let's assume that the LED has a \$V_F\$ of 2.0v, an \$I_F\$ of 20 mA, and our power supply has 5v. If we wanted a standard current limiting resistor it would have to be 150 ohms to limit the current to 20 mA.
With a pot, we also want a 150 ohm fixed resistor in series. The reason for this is that the pot will go down to 0 ohms, and we don't want to blow anything up in that case. So by putting the 150 ohm resistor in there there will be a maximum current of 20 mA through the LED.
Let's also say that we want the LED current to go down to 1 mA. Unless the pot has a super high resistance, it won't go down to 0 mA, and 1 mA seems like a reasonable lower limit. To make that work, our pot needs to be about 2K Ohms.
Going through the math, the maximum power dissipation on the pot is when it is at about 8%, and the resistance is 160 ohms. In this case the dissipation in the pot is about 0.016 watts -- which is fine for almost every pot. Even so, it is an important step to make sure you won't be burning up your pot.
But here is the important thing: The human eye has a logarithmic response to brightness. Let's say that we have 100% power going through the LED and we want to turn it down. It needs to go down to about 50% before we sense that as being reasonable. The next step down would be at 25%, etc.
Put a different way, if our knob was marked 1 to 10, then 10 would be 100%, 9 would be 50%, 8=25%, 7=12%, 6=6%, 5=3%, etc.
The problem is that a standard pot doesn't quite do that. It will work, and the LED will be dimmed. But a large part of the pots range (maybe 50%) will essentially be useless, producing very little change in brightness.
You might be able to use an audio pot, which has a logarithmic taper, but I'm guessing that the log part is in the wrong direction. (Sorry, even though I work in audio I don't use log taper pots.)
So yes, you can use a pot. It just might not give you the effect you seek.