Alright, so I just got an ATTINY2313-2oPI .. But I have no idea how to get started with this thing. Anyone have any tips, or ideas? I just want to get it hooked up, like with a 5V voltage reg. and a USB cable for programming.
To get started with a bare ATTiny2313 you'll need an AVR ICSP, such as this USB AVR Programmer. You can breadboard the connection from the ICSP cable to the chip (see ATTiny2313 datasheet and ICSP pinout). Depending on the programmer used, it may be capable of providing power to the circuit (depending on your needs you wouldn't need to start with a 5V regulator).
Also mentioned in the product page for the programmer is 'avrdude'; basically a free gcc based toolchain for developing and loading code onto Atmel AVR parts (including the attiny family).
In theory an Arduino can also be used for ICSP if you have one laying about, but I'm less familiar with that process.
I learned it from this page: http://letsmakerobots.com/node/31379 Although it mentions ATtiny13, my first Tiny was a 2313 too.
I love to use an Arduino as ISP, because it gives me the opportunity to experiment with a larger device (ATmega168) which is really easy to program. Then I port the s/w to the Tiny, which isn't too hard either. Don't go for the Arduino Uno, the ISP s/w doesn't work on that one.
Here's a reasonable single-page tutorial. (It shows using AVR Studio though which you may (or may not) want to bother with if just starting out as opposed to WinAVR / AVR-GCC.)
Here's a page that covers basic I/O operations which I found useful.
Another page I regularly refer to is the Engbedded Atmel AVR® Fuse Calculator.