Basically you need something that will know how to talk USB at the other end of the line. Same thing goes for any other port, but USB itself is pretty complicated, so I'll focus a bit more on it.
Unlike simple serial or parallel ports, doesn't just send data over to the other side. Instead, there's a large (well compared to RS-232) amount of communication between the device and the computer and the device needs to identify itself to the computer and so on and so on.
So if you aren't particularly interested in how exactly USB works and just want to use it to control something, you have two basic approaches: Get some sort of converter, such as a USB to serial port chip (FTDI makes may types of such chips and is very popular) or cable and then use USB just to send serial commands from the computer. The second approach is to get a microcontroller which can talk USB. There are many such microcontrollers today and there are libraries available that will allow you to simply program a microcontroller to work as a USB device. There are also numerous PC side examples which you could use to work with your micro. The bad side of this is that if you want to go a bit beyond what pre-made libraries offer you'll either have to go the serial to USB converter way (and it's not difficult to implement a virtual serial port inside of a microcontroller) or to learn how to work with USB, which is difficult.
With traditional serial port, what you have is basically just a few wires which are toggles between various states by the computer. It's much easier to program it on both the PC side and on microcontroller side. Also, since you can directly control pin states, you don't have to use microcontrollers at all. You can simply build a circuit which will react when a certain pin state goes high or low and that's it.
Do note that traditional PC serial ports use RS-232 signaling levels, so zero is positive voltage and one is negative. Also the voltages are pretty high at around 12 V.
Another type of port that was extremely popular with hobbyists in the past, but is getting rare today, is the parallel port. It's main advantage is that you basically have an 8 bit bus which you can control, so you have much more pins you can directly toggle from computer. Main problem with it today is that you'd basically have to get a parallel port card for new computers since many do not have it anymore on their motherboard and the USB to parallel port converters often only work for printer use. Another problem are drivers, since in the post Windows XP era, drivers which allow you to directly experiment with the port are a bit rare.