What is noise?
In most general terms, noise is any undesired signal.
Power supplies are commonly modelled as ideal voltage sources. For logic or small-signal analog circuits, it's a DC voltage source. It has no variations from the desired voltage. Note that some circuits require two sources, and some even more. So
For my understanding noise defines high frequency signals.
is close. Usually, it's technically any AC signal which is undesired. In logic circuits, high-frequency transients are the biggest source of problems, but in part that's because getting rid of low-frequency components is so easy. Except when it's not. For instance, when making ultra-high-impedance amplifiers (at pA levels) it's easy to get leakage currents from DC components which affect an amplifier, and which require measures such as guard rings. Extreme cleanliness can come into play in such circuits, and a fingerprint on a pc board can cause problems. Even logic circuits can occasionally have problems when using CMOS (high impedance inputs) and failing to clean solder flux from a board after working on it.
But, for instance, in audio systems the biggest source of noise is usually power line frequencies - 50 or 60 Hz and harmonics.
To prevent these signals in my circuit I would need low pass filters, so the impedance of the circuit is relevant. But is it really as easy as measuring the impedance?
Well, sort of. But do you have any idea how hard it is to accurately measure impedance at high frequencies? You need to become familiar with the concept of parasitic components.
Additionally, there are such things as external sources which can be picked up by antenna effects, and there are issues such as ground loops which can cause real problems. If two conductors run side-by-side and one contains an AC signal, the other conductor will to some degree intercept the radiation from the first conductor, and this will produce what is called cross-talk, and which can be very difficult to track down. Trust me on this.