There are a couple of answers to this, but pertaining to optical smoke detectors: it doesn't. It responds to anything that reflects enough light of that particular wavelength.
Now, there is some method to the madness of course. You can make a list of things you expect to be in the air in a home and tune your detector to ignore these things, increasing the sensitivity to smoke.
A great example for optical smoke detectors is the wavelength that is used for the light source. Water vapor, even thick vapor, absorbs light of a certain wavelength. Most smoke detectors use infrared light sources because 1) those can be operated very efficiently, requiring the smallest (cheapest) components available and 2) because water vapor absorbs this light instead of reflecting it.
Of course, in the same vein you know roughly what size and spectral response smoke particles have. In any kind of fire, you will emit soot (small lumps of carbon) in a well-defined size range. This brings with it certain optical characteristics that the sensors are tuned to.
However, as it stands optical smoke detectors actually have fairly unfavourable discrimination characteristics. A big problem with them is that they require a LED to be actively turned on a significant amount of the time (duty cycles of about 1/1000 are not uncommon) thus running down the battery fairly quickly. Also, smoke from gas fires is usually very rich in water vapor and not at all sooty, so optical detectors don't detect this kind of 'smoke' until it is too late. That is why there are better types of detectors, most notably radioactive ones. These use the property of smoke that it is very ionizable compared to just normal air or even high air humidity, and can detect flames much earlier than optical alarms. Also, they use almost entirely passive circuits (charge plates and a passive source of ionizing radiation), requiring very low operating current and having very long battery life, often longer than the life of the battery itself.
Because of the relatively poor discrimination capacity of optical smoke alarms, but the obvious benefit of not needing radioactive materials, sometimes these alarms are outfitted as a combination alarm, reacting both to the smoke and another characteristic of flames: heat.