I think you have a couple of options.
Parallel ports are still the simplest solution for controlling simple circuits. There are still kits out there that allow you to use the parallel port to control a set of relays.
A quick web search for "parallel port relay kit" yields a bunch of results. I've included the PDFs from two of those kits, because they contain part lists and block diagrams:
The software for controlling those kits can get a bit hairy, but I found a nifty Python library that worked really well on both Windows and Linux: http://pyserial.sourceforge.net/pyparallel.html
Most modern motherboards don't come with a parallel port any more, but there are many inexpensive expansion cards that provide one or two parallel ports. Alternatively you could buy an inexpensive all-in-one PC that still has a parallel port, like the Intel Atom DH2500.
Note that USB to parallel port cables will generally not work for this kind of thing. There are hacks out there that can make them work, but the hacks only work for certain cables.
Finally, if you want to use the parallel port for a permanent installation, be aware that during start-up, all of the relays will switch on and off briefly. There's not much you can do about this, unless you want to rewrite your PC's BIOS.
I've been playing with Arduino boards for a couple of months now. It's very easy to get it to send text to the PC on the USB serial port, and I imagine it won't be too difficult to make it listen for serial commands from the PC either.
Alternatively, programming the board to do all the control itself is also a very good solution. There are bunches of expansion boards, including sets of relays. Arduino boards are also really simple to feed - I've seen them run from about 5v all the way up to 20v, although I think they recommend between 9v and 12v for long-term use.
You shouldn't have too much trouble finding help with Arduino projects. The community for them is huge, and they're pretty much designed for tinkerers. I highly recommend their starter kit for people just starting with electronics.
Serial Ports (Including USB)
Learning to design and build USB port hardware is still on my TODO list. While I haven't used them myself, I've heard good things about two books in particular:
They're only about $25 each, and if you have more specific questions after reading them, I'm sure that you can find more help by asking specific questions here.