How would I create a circuit which turns on an LED when the voltage goes below 2.5V? I am using 2x AA batteries
A simple comparator circuit is one way of doing it, something like this:
You will have to adapt it as necessary for the lower voltage. Use a low voltage comparator (if not open drain output then switch the LED/resistor between output and ground) and change the zener for a low current voltage reference (TL431 would do okay).
You can use the TL431 mentioned directly as in this circuit (does high and low levels):
The TL431 conveniently has a reference voltage of 2.5V so you don't need the R1A/R2A divider, you can just tie the ref input to Vbatt. If you want an even lower reference, there is also a 1.24V reference version, the TLV431.