This depends entirely on the board fab and the assembler. 0.5mm pitch components are common (MSOP-10 et al). Depending on the manufacturer they may recommend as big as 0.4mm pads, meaning you have only 0.1mm gap between the pads! I would personally ignore such advice as 100µm enters the realm of "special" board fabbing technologies. If you make the pads 0.3 or 0.35mm, you will have comfortable 150-200µm gap between the pads. Never make pads smaller than the physical pins unless absolutely necessary and never make paste mask opening bigger than the pad size.
If you have something actually fine pitch such as the 0.4mm component mentioned here it gets more and more painful. It's very important to talk with the assembler to establish if they can work with the specification and what recommendations they would have re: solder mask and paste mask.
On fine-pitch BGAs (0.5mm is very very fine pitch here) it's even more important to establish a rapport with the assembler. Often the chip manufacturer is unhelpful on this as they'd have to take responsibility if they give out design parameters. BGA pads are NOT that straightforward as the paste dot size and solder mask opening size versus the pad size really does matter.
Some things to consider here are copper thicknesses, stencil manufacturing process and thicknesses, paste mask shrink if any etc etc. Theoretically the PCB fab guys should "know better" and adjust your paste mask/solder mask gerbers but that's a gamble. On the other hand, if you tell them not to adjust the gerbers you better be damn sure you have the parameters correct.