My question is that if free electrons are not attached to any particular atom, then how come they leave a hole behind when they go to the P side ?
They do not.
A pure (undoped) semiconductor has an equal amount of mobile charge carriers (electrons and holes). A very useful property of semiconductors is that you can control the electron and hole concentrations by "doping" the semiconductor. You do this by introducing a small amount of atoms that either add (donor dopants) or remove (acceptor dopants) an electron from the semiconductor. When you do this, the number of mobile charge carriers of each type change according to the law of mass action:
Where \$n\$ and \$p\$ are the electron and hole concentrations and \$n_i\$ is the intrinsic carrier concentration, which is the number of electrons and holes in the undoped semiconductor. By adding donor atoms to the semiconductor you can increase the electron concentration by an amount equal to the donor concentration. Doing so also reduces the hole concentration in that area. However, the total charge of the semiconductor stays zero. The total net charge from the mobile charge carriers is cancelled out by fixed charge: ionized dopants.
To address the question directly:
how come they leave a hole behind when they go to the P side ?
They do not, they leave behind a positively charged ionized donor.
aren't holes formed when the covalent bonds can accept an electron to make covalent bond with two electrons?
No. Holes are created when an acceptor ionizes and creates a hole. Holes exist due to a bunch of quantum mechanics that I can't write down off the top of my head. But its best to think of holes as just like electrons but with a positive charge. People might try to tell you to think of a hole as just the absence of an electron, but don't fall for it. Holes are very much not just the absence of an electron and it will confuse you to think of them as such.
Since these electrons were already free, they shouldn't leave a hole behind.
The electrons were free, which is precisely why they leave behind an ionized donor. The ionized donor is the reason the free electron was there in the first place.