If you don't match the impedance, parts of the energy flowing through the transmission line is reflected at the point where the impedance changes.
The reflection factor for the wave is:
\$ \Gamma = \frac{Z_{new}-Z_0}{Z_{new}+Z_0} \$
The power transmission has its maximum when no wave is reflected. This is when
\$ Z_{new} = Z_0 \$ (matched impedance).
If you leave the end of the line open. You have a reflection of "1". The wave is coming back as it was sent. If you short the end, you get a factor of "-1". The wave is reflected inverted. For any resistance other than these three cases, some energy is reflected and some not.
More detailed explaination:
In an RF system you have three impedances: The source impedance \$ Z_i\$, load impedance \$Z_L\$ and line impedance \$Z_0\$.
Maximum Power transfer occurs if the input impedance \$Z_{in}\$, seen by the source, equals the complex conjugate of \$Z_{i}\$. this is basic knowledge in AC analysis for every frequency range.
$$ Z_{in} = \overline{Z_{i}} $$
The line impedance does not have to be equal to the source impedance. Only the resulting input impedance (which depends on the line impedance) has to.
The input impedance \$Z_{in}\$ can be calculated or evaluated graphically using a smith chart.
$$Z_{in} = Z_0 \cdot \frac{Z_L+ Z_0 \tanh \gamma l}{Z_0+Z_L \tanh \gamma l}$$
For a detailed explaination of this formula you can look into various RF books or Wikipedia.
Example 1: If you have a source and load impedance of both 50 Ohms, a valid solution is a 50 Ohm line with any length, as long as it is considered lossless. The input impedance seen into the line equals 50 Ohms and therefore the above condition for maximum power transfer is fullfilled.
Example 2: If you take a 100 Ohm load and want to connect it to a 50 Ohm source, a valid way would be:
Take a line with an characteristic impedance of \$Z_0=70.7\Omega (=\sqrt{50 \Omega \cdot 100 \Omega})\$ and a length of a quarter wavelength. The input impedance \$Z_{in}\$ is equal to \$50 \Omega\$ in this case. Therefore, maximum power transfer occurs.
You see, that the line impedance does not necessarily have to be the same as the source/load impedance. However, an easy way to achieve a valid solution is to set all impedances equal, as shown in the first example.
I hope this explains it a little better.