Usually if you want to use for example an NMOS as a switch to ground you would simply connect the body (bulk or backgate) to ground. Then source = body and things are simple, Vgs is the same as Vgb (gate-bulk voltage).
Note that when we want to operate this NMOS as a switch we want a large Vgs. In the (grounded source) example I mention above Vgs = Vgb. Now think about it, do we actually need a large Vgs or do we really just want a large Vgb?
The conductive channel in an NMOS forms when Vgb is large enough. So when the Gate to Body (bulk, substrate whatever you call it) is large enough. Would the channel still be there if I disconnect the source: yep.
So the source voltage/connection doesn't matter that much. The large Vgb is what we really want and need. So that means we should connect the Body to the lowest voltage so that Vgb becomes as large as it can be giving us the lowest on-resistance.
If you're designing this circuit to be on an IC then for the NMOS you might not even have a choice. If the NMOS are made in the "global substrate" of the silicon die than you have no choice: all NMOS Body contacts will be this "global substrate" and will be connected to ground.
Some CMOS processes do allow you to make a separate "P-well" to place your NMOS in. But even then, for reasons mentioned above it is still the most beneficial to connect the Body contacts to the lowest voltage, i.e. Ground (or VSS).
For PMOS you generally do have a choice as these are inside an "N-well". Generally this N-well needs to be connected to the highest voltage which is the supply (VDD). The Vbg and channel story also applies to the PMOS so generally you can just connect the Body to VDD.
I have also heard that transistors in common design processes are symmetric devices.
If you look at the most simple sideview of an NMOS transistor, you can see that the transistor is indeed symmetrical:
And in all CMOS processes the "basic" transistors are symmetrical, source and drain are the same.
Only for "special" (for example RF or High Voltage) transistors this might not be the case.
For PMOS the same is true, the "basic" PMOS transistors are symmetrical, source and drain are the same.
Does this mean that it does not matter if the drain and source terminal are swapped?
Does the body potential influence this polarity?
No it does not, as the drain and source are physically identical structures.