I don't understand the physical significance of Common Mode Voltage.

It is arbitrarily defined as the average of two voltages and is expected to be rejected by a differential amplifier. Why? What is its physical significance?

EDIT my question was specific to differential amplifiers. Both inputs to the differential amplifier need not be equal and opposite. What would be the meaning of common mode voltage in such case?