USB is a communication protocol. When I say its a protocol, it means that there are certain rules that you have to follow in order for your computer to communicate with the targeted device.Bellow is the wire structure required for USB communication.

Now, by default, your USB power supply is always on. But there are few computer motherboards that provides you to turn on or off depending on your need. If you just want to control a single led (turn on or off) then you would want to check weather this facility is available with your motherboard.
If you don't have, then the solution becomes a bit difficult. This is because you can not control the voltage levels of DATA- and DATA+. Thus you might have to create a device that would understand and decode a USB packet. So when you send the turn on or off signal encapsulated in a USB packet, the device would understand your command and control the led for you.
for the second option, check out Arduino. There are very good examples for you to work on for arduino. These boards make use of FTDI chips that @MarkU suggested.