I'm currently testing a design, and using two small wires to connect an oscillator with a frequency counter (both are small circuits I have build).
So here is a question, when designing RF and connecting different circuits, when is something to be considered a transmission line?
I mean, when do I have to think about transmission line impedance matching (and to start worrying about reflection and ect.).
An interconnection should be considered a transmission line if reflections are detrimental to your application. As Leon Heller says, 1/10th wavelength is a good rule-of-thumb.
However, all this depends on the nature of the signal (1/10th of what wavelength?). With sine waves, the wavelength in the cable will be about $\frac{2 X 10^8}{f}$ where f is the frequency, but will vary according to the cable type. The effect of reflections on sine waves is to cause standing waves, so the amplitude will vary along the length of the cable but will always be sinusoidal. With square waves the situation is complicated by harmonics. Although the square-wave may have a low fundamental frequency, the harmonics introduced by the edges will be significant. The effect of reflections in this case is undershoot, overshoot or ringing depending on where the mismatches are (if both $R_S$ & $R_L$ are lower than $Z_0$ for example, you will get ringing).
So do you need to match the source? No. As long as the load is perfectly matched, the wavefront will travel down the transmission line and be fully absorbed by the load. Consider a digital system with 3.3V logic for example. If you terminated both source and load you would only get 1.65V swing at the load. Terminating only at the load is termed parallel termination.
There is also a technique known as series termination in which the source is terminated but the load is left open circuit. In this case, a half-amplitude edge propagates down the line and is reflected back to the source, lifting the voltage to full amplitude as it returns until it is absorbed by the source termination. Along the line, you would see a 'stepped' signal except right at the load where you would see a clean waveform.
As a rule, a transmission line isn't required if the length of the connection is under 1/10 of a wavelength.
A low-impedance source can generally drive a high-impedance load without any problems. Loss of signal results if they are the other way round, and a buffer stage is usually the answer.
Impedances should be matched when using transmission lines.
Every line is a transmission line, however different situations require different level of rigour.
At low frequencies - from DC to few hundred MHz, short transmission lines have only very small inductance and capacitance and can be usually ignored because they don't contribute much to input of output impedance of what they are connecting. The situation with them is similar to the case when the stages are connected directly. Once you start moving into GHz range, impedance of the line becomes significant and you have to treat it as separate stage of your device and match it. When the line is long, like in the case of cable, impedance also goes up and becomes significant, and you have to match it.
At high power levels reflected wave can destroy previous stage and you have to care about matching even at low frequencies. For example if you try to transmit 100W with disconnected antenna, you are likely going to fry your power amplifier (unless it has protection against such situation).
Example: short PCB trace is going to have say 10 nH inductance, which at 10 MHz would give it reactance 0.63 ohm (open circuit), shunt capacitance 1 pF, or capacitive reactance 15783 ohm (high impedance), and resistance under 1 ohm - all values insignificant if the input and output impedance of the stages is 50 ohm. At 2 GHz it would be reactance 125 ohm and shunt capacitance 79.6 ohm - significant values that would require redesign into transmission line.
