Your understanding is correct.
For circuit design the source is usually the reference terminal. Consequently so called source-referenced transistor models were introduced to formulate the drain current as a function of voltages relative to the source terminal (\$V_{GS}, V_{DS} V_{BS}\$). In cases where the bulk source voltage \$V_{BS}\$ is zero the gate-source voltage \$V_{GS}\$ is equal to the gate-bulk voltage, so we actually see the effect of the gate-bulk voltage \$V_{GB}\$. For non-zero \$V_{BS}\$ the body-effect is used to model the behavior of the transistor. The body-effect describes a change of the threshold voltage \$V_T\$ and so the behavior of the transistor depending on \$V_{GB}\$ is obtained.
The threshold voltage \$V_T\$ with a backgate-bias voltage \$V_{SB}\$ is given by the following expression (see Wikipedia).
$$
V_T = V_{T0} + \gamma\left(\sqrt{|-2\phi_F + V_{SB}|}-\sqrt{|2\phi_F|} \right)
$$
where \$V_{T0}\$ is the threshold voltage for \$V_{BS} = 0\$, \$\gamma\$ is the body-effect parameter and \$2\phi_F\$ is the surface-potential.
Integrated MOS transistors are often symmetric. Therefore the source and drain terminals are not defined by the layout of the transistor but only by the applied voltages. In order to get equations that reflect that symmetry body-referenced models are used, where indeed voltages are referred to the substrate and not the source of the transistor.
