I see recommendations of 2-4 mils for solder mask expansion. But why is it necessary?
If the solder mask expansion were 0, in theory -- assuming everything aligned perfectly -- the board would work fine.
In practice, things never align perfectly. The actual hole punched in the solder mask may be slightly less than what you specified ("shrinkage"), and that hole is always be placed in a slightly different location than what you specified ("movement"). If your solder mask expansion is too small, then these misalignments cause the solder mask to partially or completely overlap SMT pads and through-hole pads.
If the solder mask completely covers most or all of the pad, the SMT part will be completely disconnected from that pad. Then the board will immediately fail the end-of-line go-nogo test.
Many people specifically design the pads of a footprint to comply with IPC's fillet recommendations. If the solder mask even partially covers some of that pad, then the fillet of solder will be smaller than a person looking only at the copper might expect. If the fillet of solder is too small, then the (SMT or through-hole) part will not be mechanically attached as well. After a few thousand cycles of vibration, the solder may eventually crack, and the part will be completely disconnected from that pad or hole. Then your customer will notice the problem. (This is much worse than a board failing the end-of-line go-nogo test).
Daniel Grillo gives an excellent explanation of what happens if the solder mask is too big.
There are various sources of errors in the plotting and placement of different layers (and holes!) that go into the board. The board house knows their capabilities/yields, and will tell you the allowance that you have to make for that.
That is why they have things like minimum annulus (when the drilled holes end up so off center that it might end up not plating to the pads). And the solder mask clearance ensures that the worst-case mask placement offset does not end up covering the pad.
For simple prototypes, it's usually not a big deal, and you can just use "default numbers" and then tell the board house to accept the clearance violations. However, for very expensive boards, large runs, et cetra, you should get the exact numbers for all the various clearances to meet the board house's capabilities.
Note that board houses can also have different classes of ever increasing tolerances, and charge different amounts of money. Unless your product really requires super tight registrations, use as "sloppy" a clearance number as you can afford.
With some designs, you want to make sure that the solder mask is never on top of SMD pads. This is especially critical with QFN or LGA packages where the contacts of the parts don't stick out over the plastic molding or with parts that have a very fine pitch: Even small registration issues would cause the solderable areas of the tiny pads to become even more tiny.