This is because of Voltage Droop. As you pull more current, the voltage on the ATMega's pin drops. Since LEDs have a correlation between voltage and current, this causes a sort of self-balancing act. (The First Picture is an LED v vs i graph, the second is the Pin driver v vs i graph from the ATMega328 datasheet)
The led and the pin driver eventually come to an equilibrium. As the source voltages drops, the led wants to pull less current. The problem is that this might be at a current that can damage the pin driver, AND damage the led. It might not be immediate, but it can drastically shorten the life of the Driver, and of the LED, and harm the accuracy of the driver as well.
Individual Arduino/ATMEGA Pins should only source 40mA, and that's with a 0.6V droop, The Port should only source 100mA, and the entire Microcontroller should only source up to 200mA. These are the limits of safe/recommended/tested use as stated by the manufacturer. Any more than that stresses the microcontroller, might be okay, might not, but it's a gamble.
There is NO current limitation in the pin driver. If you short a logic high pin to ground, you just drew a few Amps and killed the driver. If you want to pull 100mA from a 40mA recommended max pin, you can! It is up to you, the designer, to restrict current draw. The 40mA is a recommendation, like a speed limit. You can drive faster than the limit, you shouldn't for obvious reasons.
Leds can be driven above 20mA. That is the Typical rating for x amount of hours. You can give it more, it will be brighter, but will die sooner. ANd it's really because of heat concerns. The more current, the hotter the led junction gets, the more current it wants to let through and eventually it falls into thermal runaway.
In short, you can directly connect the led, and through a unintentional combination of factors, it will not blow up (immediately). If you hook up a red led (1.8~2.0v) directly to a 5v pin without a resistor, it will probably die, but a blue or green or white one might not.
Last thing, you can use the internal pullups in the ATMega to safely light an LED. The internal pull up of 20k will allow enough current to light an led, dimly.
pinMode(pin, INPUT); // set pin to input
digitalWrite(pin, HIGH); // turn on pullup resistors