I Want to drive one or two small ( 5mm or 10mm ) LEDs in parallel
is this as simple as putting a resister and a switch and i'm done?
It is even simpler. Put a resistor and LEDs in series and you are done.
What about when the battery is drained and can only output 2.x volts?
If Li-Ion battery is drained to 2V it is as good as dead. You should never let it discharge below 3V, ideally recharging at 3.3-3.4V
Which means it will most likely be higher than double of most LED forward voltage, allowing you to connect diodes in series.
If your LED forward voltage is above 1.7V then you can use them in parallel, but you need individual resistors connected in series with each LED.
So, that takes care of LEDs. Now, if there is no very strong reason to use two batteries in parallel I'd recommend staying with one 18650 cell. There are many potential problems with using Li-Ion cells in parallel. You can search this site and find literally hundreds of questions and answers on this one, if you are curious. Otherwise just stick to single cell.