A simple light bulb can work straight on say 3V, 5V, 12V (depends on what one you get). A LED is different, it requires a certain amount of current to pass through before it lights up. Because a LED is a kinda of diode (the symbolic shows that), the voltage remains almost constant when it's conducting. It will rise a bit, but that's almost negligible.
A typical LED requires a minimum of 1 or 2mA to light up. Most have a maximum of about 20mA. The voltage depends on the colour and sometimes type of LED you have. Let's say you have a simple red LED.
Typically it would say '2V drop at 20mA'. That means that if you run 20mA through it, there will be a 2V voltage drop (NOT the other way round - this may be kinda hard to understand at first).
But, we have a 5V supply right? So if we put 5V on it, the LED will conduct far more than 20mA and will blow up. What we want is to make a circuit that the resistor takes 3V up, and 20mA will flow through both the resistor and LED (because they are in series).
We can do that with Ohm's law. It describes the relation between current and voltage of a resistor:
In this case we want U 3V (the voltage across the resistor) and I of 20mA. So we fill it in:
R=3V/20mA = 3V / 0.02A = 150 ohms.
Now, because the LED is running of an Arduino, the microcontroller may not be able to deliver 20mA. Furthermore, I don't know the exact specifications of the LED, which may be different. So I assume they have calculated their 560 ohms on good basis.
Why it's obviously not 20k or 2 ohms.. well. If you put the whole 5V on a 20k resistor, you only get 0.25mA of current. Assuming the LED will take a little bit, there nearly won't be any light at all.
If you take 2 ohms you're going to blow the LED up. There will be so much current flowing that the LED is toast.
As for the arduino;
A arduino contains a microcontroller chip. These are intelligent devices which can change the output of a pin. It can make a pin high (make it 5V), or low (make it 0V). We can program that by software.
If you make the pin high, it will just put 5V on your LED and resistor. Current will flow, LED will light up etc.
If you make the low, it will put 0V on the LED and resistor. That will not do a whole lot, and the LED will be off.
Circuits always require a loop indeed, but the microcontroller has hardware inside it to fix that. Think of it as switches inside that will connect the either the 5V supply to pin 13, or GND (if its state is low).
The LED and resistor is connected to ground so it completes that circuit.
We could also have done it the other way around, but then the LED will be ON if you make the pin low (0V) and off if you make the pin high (5V).