There are two questions here, really:
- how to get 8 Watts out of a mains connection
- how to provide a battery backup
The absolutely easiest way to get 8 Watts out of a mains connection is to use a cheap, ubiquitous, 5V/2A or 5V/2.5A USB wall wart. You can also buy these things as components from suppliers like Digi-Key or Mouser, although you'll likely pay more in singles than you'd pay for the USB charger from Amazon.
To get a battery backup, you need to choose a battery chemistry, and then have the appropriate circuitry to support safely charging the batteries, without overcharging them. Lithium batteries are extra annoying here, because they don't like being "floated" at their top voltage; doing so leads to premature wear-out.
Also, if your LEDs need 4V to develop full current (2 A if it's 8 Watts) then a bare Lithium battery isn't enough; it will quickly drop to about ~3.8V, and it will slowly discharge through to 3.2V or so at "fully discharged." Thus, you will need some kind of boost/buck regulator when the battery is powering the LEDs. Ideally with an undervoltage cut-off to prevent the battery from being over-discharged and damaged/destroyed.
Finally, there's the "provide 500 mA regulated current to the LEDs" part -- LEDs for illumination are typically specified on drive current, not specific voltage!
Your final solution block diagram would look something like:
Power Supply <-> Battery Manager <-> Current Regulator <-> LED
Each of these typically use IC-based circuits, and many of these ICs aren't even available in through-hole packages, so you might need to buy break-out boards for each block if you want to wire it up on a breadboard.