I am building a little circuit and I am adding a little 6 volt LED to tell me when I have power going through. But my power source is a little 9 volt battery, will this LED kill this battery quickly? I want it to last, will I be okay?
The 6-Volt Minature Lamp you've linked from Radio Shack isn't a LED, it's a small incandescent light bulb. It lists the current draw as 25mA and a 9V PP3 alkaline battery normally has a capacity of around 500mAh so I'd expect it to last around 500 / 25 = 20 hours. It will be less for cheaper zinc carbon batteries.
The other issue is that it's a 6V bulb so driving it from 9V isn't recommend and ways of dropping voltage (called a voltage regulator) tend to either be inefficient or a bit on the complex side for something like this. You'd probably be better finding a LED and suitable current limit resistor. A LED will probably be bright enough at 5mA so last five times longer.
I don't think the part linked is an LED, but it does specify 6V @ 25mA nominal.
Looking at some rough 9V battery capacities, let's say the battery has about 565 mAh.
Suppose we designed some smart circuitry which magically converted the 9V to 6V and was 100% efficient.
The power draw is then 6V*0.025A = 0.15W. Assuming the battery is at 9V until it dies, it has a total energy capacity of 565mAh * 9V = 5.355 W*hr. This is most likely a gross over-estimate, but we're just getting a rough estimate on the upper bound so this is ok.
Dividing the battery energy density by the power draw gives us a rough upper-bound on how long the battery will last: 35.7 hrs of continuous use. Depending on your application this may be good enough or way too short.
Now suppose we go the opposite direction and use a simple resistor current limiter. That means we only get to divide the current capacity of the battery by the current draw of the lamp: 565 mAh/25mA = 22.6 hrs of continuous use. Again, this may be good enough, or way too short depending on your application.
Also, just because the bulb is rated at 25mA nominal doesn't mean you can't run it with a lower current. This will give you increased battery life and a dimmer light.
If you directly connect the two, it may quickly age (possibly destroy) your led/lamp, and draw slightly more current from your battery than was intended by the light source's designer. The specs you gave are so unusual, that it must either be a lamp or a led with an integrated current-limiting resistor for 6V. To use it, the easiest option is to add another 120 Ohm resistor to create a 3V drop at the nominal 25 mA.
EDIT: Depending on your choice of 9V battery, expect that this light source alone will deplete it on a timescale from several hours (old NiCd rechargeables with maybe 80 to 120 mAh) to possibly nearly ten times that much for the most powerful primary batteries.