As an example, a typical op-amp based non-inverting amplifier, the way we are usually taught about how it works generally uses assumptions about extremely high open loop gain, and negligible so-and-so, zero this, or infinite that.
At some point in the derivation of gain there may appear an idea of some "reference potential" such as "ground", but strictly speaking that's merely one of many potentials present in the system:
simulate this circuit – Schematic created using CircuitLab
If we ignore what any of these potentials actually mean, and concentrate only on how we expect an ideal op-amp to respond (at Y) to potentials at X and W, we get a conveniently simple set of equations that we can combine to derive the overall relationship between potentials at X and Y.
For example, the potential \$V_W\$ is somewhere between \$V_Y\$ and \$V_G\$:
$$
V_W = V_G + (V_Y-V_G)\frac{R_2}{R_1+R_2}
$$
We also have the ideal op-amp behavior, where A is its open-loop gain:
$$
V_Y = A(V_X-V_W)
$$
When you combine these equations, and make the assumption that \$A\$ is very very large and positive, \$A\$ disappears from the equations and you get something like this:
$$
V_Y - V_G \approx (V_X-V_G)\left(1+\frac{R_1}{R_2}\right)
$$
Note the "approximately equals" symbol; this is not exact, because A is not infinite in reality. Already we've lost precision.
Note also that we haven't yet declared what \$V_G\$ means, but you can see where I'm going here, because I chose the letter G, short for ground. This circuit behaves as it does, not because there's any "reference potential" anywhere, but purely because that's what the maths says it should do. All the voltage terms in this equation are potential differences, \$V_{SOMETHING}-V_G\$, and if we declare (arbitrarily) that \$V_G=0\$, the algebra simplifies to the familiar relationship for a non-inverting amplifier:
$$
V_Y \approx V_X \left(1+\frac{R_1}{R_2}\right)
$$
So, by declaring that \$V_G\$ is our reference "zero" potential, we've got ourselves a super-simple equation, but we mustn't forget that the previous equation containing \$V_G\$ is still valid, and approximately true, and is (technically speaking) more general.
In other words, however we build our op-amp, it needs no knowledge of where zero is, it will always do \$V_Y - V_G \approx (V_X-V_G)\left(1+\frac{R_1}{R_2}\right)\$, and it's only because we arbitrarily choose \$V_G\$ to be zero that it appears as if the op-amp knows where zero is. It doesn't. In fact, it doesn't even know what \$V_G\$ is in this example, since there's no direct connection to the op-amp from \$V_G\$.
--edit 1--
Remembering that the op-amp doesn't have any idea of a "reference", no clue what "zero" actually means, even if it decides to add some ridiculous offset to its output (say 10V), so that the relationship:
$$
V_Y = A(V_X-V_W)
$$
becomes in reality:
$$
V_Y = A(V_X-V_W) + 10
$$
The term \$A(V_X-V_W)\$ is still so large compared to \$10\$ that we may still make this approximation:
$$
A(V_X-V_W) + 10 \approx A(V_X-V_W)
$$
Thus we may ignore that errant offset, further illustrating why, algebraically, the op-amp's ignorance of zero (or any absolute reference potential) makes little difference to its behaviour when negative feedback is at work.
--/edit 1--
We're not done yet, though, because I still haven't answered why variations in \$V_S\$ or \$V_T\$ (with respect to \$V_G\$) would cause \$V_Y\$ to vary at all.
To understand that, first remember that we made an assumption about \$A\$ being huge, which enabled us to make \$A\$ disappear from the equations. It's not the only assumption we made about \$A\$. We also assumed \$A\$ is constant, and it most definitely is not. We removed \$A\$ from the equations for convenience, but in reality it is still in there playing a role. The hope is that it's always large enough that even significant changes in \$A\$ (say, by a factor of 2) will produce no appreciable change in output, but there will still be some change, never-the-less.
It's impossible to make a differential amplifier, using real-world transistors and other components, that guarantees a constant open-loop gain regardless of how near or far inputs \$V_X\$ and \$V_W\$ are from supplies \$V_S\$ and \$V_T\$. Op-amp designers go to great lengths to get as close as possible to this ideal behaviour, using constant current sources and current mirrors in their long-tailed pairs, but in the end, the truth is that A is not constant, it's a function of all other potentials, and really should be written \$A(V_S, V_T, V_X, V_W)\$. It is always huge, but it varies quite wildly as potentials at the op-amp's various pins change with respect to each other.
An extreme example of \$A\$ changing is when the difference \$V_X-V_W >> 0\$, causing op-amp output Y to saturate; in that case open loop gain \$A\$ drops to zero, and the approximation \$V_Y - V_G \approx (V_X-V_G)\left(1+\frac{R_1}{R_2}\right)\$ fails completely.
The approximation remains true, however, as long as A is still huge, but it's still only an approximation. Variations in \$V_S\$ and \$V_T\$ with respect to \$V_G\$ will cause changes in \$A\$ (and other parameters, I'll mention below), which will result in \$V_Y\$ changing somewhat, as you have seen in your experiments, but the hope is that this effect be negligible.
There are other things that are not constant. Input bias current changes with variations in \$V_W\$, \$V_X\$, \$V_S\$ and \$V_T\$. So does input offset voltage, to a small degree. Any change anywhere causes changes everywhere else, to some degree, but the approximation \$V_Y - V_G \approx (V_X-V_G)\left(1+\frac{R_1}{R_2}\right)\$ remains true.
