FINAL
A single 9V alkaline battery will easily power both LEDs at full (9V) brightness for 10 hours.
This is more complex than the typical LED circuit where you select the LED, resistor, and battery. This is a little more complex. Here we have three unknown red LEDs with a fixed resistor.
With time the battery voltage will continuously drop and the current will decrease as the voltage drops. so we cannot just say we have a 20 mA load and a 1000 mAh battery.
The amp hour rating of a battery is from full power discharging down to a cutoff voltage. 1.5V alkaline batteries have a cutoff of 0.8V. A 9V battery has a cutoff of 4.8 (six times 0.8V).
What we do not know with this LED is the acceptable operating voltages before the LED gets too dim as the battery discharges. We do not know how much the current will be at the various operating voltages. The human eye (and mind) do not perceive LED brightness the same as a light meter. We do not know how much capacity is required due to the above variables.
The Dynamic Variables
- Voltage
- Current
- Perceived LED brightness
- Battery capacity
UPDATE TWO
I wanted to see for myself how the advice given actually worked out.
I bought ten of these LED lamps from the link provided.
They came with 40 AAA/LR03 Phillips "Power Alkaline batteries.
I used six of them in serial for testing.
In the Lab
At 12V I measured 45 mA.
At 9V I measured 22 mA.
At 8V I measured 11 mA.
At 7V I measured 6 mA.
At 6V I measured less than 0.1 mA.
The 6-9V measurements were lower than I would expect with respect to 12V so retested and results were confirmed. I then also used another LED and at 12, 9, and 7V and got very similar results: 45, 22, 5 mA.
Current measurements were made measuring the voltage across a 0.01 Ω 1% resistor in series with the LED.
Perceived Brightness
At 8-9V the brightness is consistent.
At 7-8V the brightness is slightly dimmer.
At 7V is where the brightness begins to significantly dim, yet at 7V still very acceptable nearly as bright as at 9V.
At 6V the brightness is borderline acceptable.
At 5.75 it is very dim and begins flickering.
Testing
I tested a single LED lamp powered by 6 Phillips AAA alkaline batteries.
12 hrs, 8.5V
15 hrs, 8.3V
18 hrs, 8.23V
20 hrs 8.2V
24 hrs 8.13V
29 hrs 8.03V
30 hrs 7.97V
40 hrs 7.93V
I estimate the LED could have easily lasted more than another 40 hours.
An AAA battery has about 1100 mAh discharged down to 0.8V @ 25 mA.
A 9V 550 mAh @ 25 mA.
It is fair to say if six AAA lasted more than 40 hours, a 9V will last 20 hours. So both LEDs powered by a single 9V alkaline would conservatively last more than 10 hours.
After 40 hours of viewing the LED brightness in my office, I took it back to the lab and compared the brightness to another LED on my bench supply at 9V. The difference was nearly imperceptible.
I would go with a 9V alkaline battery for each LED.
One alkaline could power both, I prefer a margin of error.
If I used one 9V alkaline for both, I'd bring and extra battery.
An AAA has a capacity of about 1000-1200 mAh.
A 9V has a capacity of almost 600 mAh.
mAh capacity is not so useful. The cutoff for the 9V capacity is 4.8V.
We need a cutoff of 7V with a load that ranges from 22 mA @ 9V to 6 mA @ 7V.
The dynamic resistance of the LED ranges from 400 Ω at 9V to about 700 Ω at 7V.
In the discharge curves below, we need to look at the number of service hours at 7V.
The service hours according to the discharge curves, is 15-35 hours.
Our 400 Ω load is better than the 35 hour point.
Our 700 Ω load is better than the 15 hour point.
so saying a 9V alkaline will last 20-40 hours for one LED lamp is very reasonable.
And very safe to say a single 9V alkaline could power both LEDs for 5+ hours.
Energizer AAA Discharge Curves

A 9V battery actually has six AAAA cells in them.

Six Fresh AAA
After 24 hours, very slight perceptible difference

Datasheets:
9V alkaline
9V lithium
Red with cover sawed off

These are peculiar LEDs. There is a metallic coating with a mirror like finish and a very small hole for the light to exit. In the above photo you can see the mirror in the center LED and the reflection of the red plastic on the right side of the right LED. Below the red regions are the mirror coating.
The LED on the left, all the red light (washed out in photo) comes from the little hole, The one on the right I scratched off some of the coating and it appears brighter. Still emits all red (not white as it appears in photo).

End of Update
ORIGINAL ANSWER
To run these from battery you would likely need a 12V battery.
My guess is that each lamp contains 3 LEDs and a resistor to set the brightness.
If you can, you should rewire the 3 LEDs in each lamp to be powered individually rather than 3 in series.
If the LEDs are red (not white) you will need a little over 2V to power them. If white you will need a bit over 3V.
If these are 2V red LEDs a lot (50%) of power is being wasted. That's not good for battery powered projects. If 3V white then only 25% is being wasted.
We can estimate the battery life. If each lamp draws 30 mA, for 6 hours of run time you will need a 180 mAH battery for each LED.
First Update
I can definitely verify that I can get these light units to light up
with less than 12V input. Is that important/relevant? For example,
they'll light when I attach the wires to both sides of a 9v battery,
or to 4 AAs in series.
Four AAs, that's good, only 6V.
Are they bright enough for you with the four AAs?
The problem I see with AA alkaline is the voltage drops fairly fast.

But the Ultimate Lithium start at a higher voltage and do not drop much over the battery lifespan.
This is a AA, but the AAA will work just as well for you.
you mention that each (LED I think, not 3-LED light) draws 20mA - how
do you know that?
I don't know, I said "let's say..."
20 mA is a good guess.
If you could measure the current that would be a big help in determining how long the various battery choices would last.
Otherwise you can only find pout how long by experimenting as I suggested.
Or when you take one apart we may be able to find the resistor value to find the current.
Not what I was expecting. This appears too well made for the price.

The bad news is, if the above is true then with four AA at 6V then there would only be about 1 mA flowing.
The good news is any voltage over 6V will work. The higher the voltage, up to 14V, the brighter they will be.
If you have the capability to add a resistor you could make them bright with as little as 7V, maybe 6V. At 9V I estimate they will be half as bright at 9V as at 12V.
The big question is how bright are the LEDs with four AA compared to the when they are powered with 9 or 12V?