It is a combination of several effects. To make the discussion easier to follow, let's just talk about NMOS transistors.
As you observe, when transistors are in series then only one transistor has its source connected to ground. The sources of the other transistors will be at a higher voltage until the transistors below them all start conducting and bringing the source voltage lower, which means that \$V_{GS}\$ will also be lower for the series transistors until the transistors closer to ground have conducted current.
It's also the case that the bodies of all the transistors are connected to ground, so if the source of a transistor is at a voltage above ground then there is effectively a negative voltage from the source to the body. The "body effect" causes the effective threshold voltage of the transistor to increase, making matters even worse.
Finally, even when all of the transistors in series are conducting you will have the series combination of their \$R_{DS}\$ values. Generally the transistors in series are made a little wider so that their series resistance is not so large.
Having said all of that, you will never achieve equal propagation delays for all input transitions. The reason is that when you have parallel transistors there will be some cases where all of those transistors are conducting simultaneously and other cases where only one of the parallel transistors is conducting, so the transition time of the output will vary greatly. Consider, for example, a 2-input NAND gate that has parallel NMOS transistors. Changing from a 00 input to a 11 input will cause two NMOS transistors to pull the output low quickly, but changing from a 00 input to a 01 input will cause a single NMOS transistor to pull the output low.