Is there a way I can test the capacity of a 3.7v rechargable lion battery like this at home? All I really have is a multimeter.
To really test a battery capacity requires a timed test with a constant current load on the battery. Since the battery voltage changes as the battery gets depleted trying to use a fixed resistor type load leads to the load current getting smaller as the battery voltage goes down. The constant current load varies its effective resistance dynamically to so as to keep the current the same over time.
For low current levels it is relatively easy to build your own constant current load device using an opamp, power MosFET, a few resistors and a capacitor or two. Power the constant current circuit from an external power supply. There are plenty of circuits on the web to go by.
A lithium ion cell is considered to be discharged when it reaches 3.0 volts. The amp-hour capacity is over a one hour period. Your picture shows 9900 mAh cells which means they should supply 9.9 amps for one hour at which time the voltage of the cell should be 3.0 volts or greater.
You can approximate a capacity test by connecting a resistor of 0.34 ohms across a cell. You can make this value by paralleling 20 6.8 ohm resistors for example. The resistor will need a power rating of at least 40 watts. It will get quite hot, so take care. Monitor the voltage of the cell during the one hour test. If it drops below 3.0 volts, stop the test. If it makes it to one hour, you can be reasonably sure of its capacity.
This is only an approximate test since as the test progresses, the voltage of the cell is dropping. This means the resistor is not a true common current load. But for some simple home testing, this should be close enough.
When testing batteries, always put everything in a fireproof enclosure. Anticipate that battery will explode so wear appropriate personal protection gear and have appropriate fire fighting equipment at the ready.
JUWEI 2in1 Digital Type-C USB tester. Has a built in countdown timer and reset switch. Tells watt hours as wells as mAh. It's under $20. Use a simple usb fan(one on a stand) and let it run until it goes off. This will give you an accurate reading of usable the current needed to power a device from the battery!