The truth is, the capacitor doesn't have to be polarized. However, if you are using a polarized capacitor, it has to put in the circuit the way it is shown as to not charge the capacitor backwards (reverse bias). Electrolytics will fail, sometimes catastrophically, when charged with the wrong polarity.
In your defense, this is a very poor circuit for learning and demonstrates bad design practices that I don't want to get into right now. But to give you the gist of what's irking me:
You don't want to slam a switch directly across a charged capacitor to discharge it. It should have some series resistance. At low voltage and low capacitance, and for learning purposes only, this could be acceptable (still poor design).
The LED is reverse-biased when the push switch is depressed and the circuit is operating off the transformer. This is not good for the LED. But at 9VAC, and for learning purposes only, this could be acceptable (again, still poor design).
This demonstration could have been done with a SPST switch instead of a DPDT switch. This is just me being picky. I assume the DPDT switch depicts greater "isolation" of the AC vs DC mode.