The common mode voltage is the average of the two values so that it has these useful properties:
If the input values are at zero voltage, then the common mode voltage zero. (Horse sense: in what useful sense can two zero voltages possibly have a nonzero voltage in common?)
If the same voltage \$\Delta V\$ is added to both inputs, then the common mode voltage changes by \$\Delta V\$, and not by some inconvenient \$f(\Delta V)\$ (or worse, \$f(\Delta V, V_+, V_-)\$). Not even something like \$\frac{2}{3}\Delta V\$. Just \$\Delta V\$. That is what common means! We make a common, equal change to both inputs, and the common mode voltage changes by exactly that amount.
If the voltage \$\Delta V\$ is added to one input, and subtracted from the other, then the common mode voltage does not change. This is rational. We have moved the inputs in opposite directions by an equal amount: there is no common movement.
Let us formalize things slightly and regard the common mode voltage \$V_c\$ as a two dimensional function of the two input voltages. Rule 1 means:
$$V_c(0, 0) = 0$$
and so on. Rule 2 means:
$$V_c(a + c, b + c) = V_c(a, b) + c$$
Note that together with Rule 1, if we substitute \$a = b = 0\$ we also get this:
$$V_c(0 + a, 0 + a) = V_c(0, 0) + a$$
$$V_c(a, a) = V_c(0, 0) + a$$
$$V_c(a, a) = a$$
Rule 3 means:
$$V_c(a + c, b - c) = V_c(a, b)$$
Suppose we accept these requirements as reasonable. Now, can we find a function \$V_c(x, y)\$ which satisfies them, yet which is not the arithmetic mean \$(x + y)/2\$. We can prove that no, the function must be the arithmetic mean.
Let's start with:
$$V_c(a + c, b - c) = V_c(a, b)$$
Next we can take the rule \$V_c(a + c, b + c) = V_c(a, b) + c\$, and apply it by adding \$c\$ to both arguments of \$V_c(a + c, b - c)\$:
$$V_c(a + 2c, b) = V_c(a + c, b - c) + c$$
Then substitute, to obtain this very useful derived rule:
$$V_c(a + 2c, b) = V_c(a, b) + c$$
By symmetry of \$a\$ and \$b\$ we also have:
$$V_c(a, b + 2c) = V_c(a, b) + c$$
The second equation above also gives us this, if we use \$b\$ in the place of \$2c\$:
$$V_c(a, b + b) = V_c(a, b) + \frac{1}{2}b$$
(If you double either of the inputs, the common mode voltage rises by half that input! We are getting there!)
Now let us combine these derived rules with \$V_c(0, 0) = 0\$, by adding \$2c\$ to either parameter:
$$V_c(0, 2c) = V_c(0, 0) + c = c$$
$$V_c(2c, 0) = V_c(0, 0) + c = c$$
In other words:
$$V_c(a, 0) = \frac{1}{2}a$$
$$V_c(0, b) = \frac{1}{2}b$$
Now, we can apply \$V_c(a, b + b) = V_c(a, b) + \frac{1}{2}b\$ to \$V_c(a, 0) = \frac{1}{2}a\$:
$$V_c(a, b) = \frac{1}{2}a + \frac{1}{2}b = \frac{a + b}{2}$$
Thus we show that requirements 1, 2, or 3 make it necessary that the function for the common mode voltage can be no function of two arguments other than their arithmetic mean. And since each of those three properties of the common mode voltage is an incredibly sound and useful idea, disagreeing with them insane; hence the arithmetic mean of the two differential voltages of the signal is the Right WayTM to define its common mode voltage; Q.E.D.