Detection of signals buried in noise

Question

This is more of a general question to see which methods are the most common to improve the detection of a signal which is buried in noise. Currently, we are building an optical system for medical imaging and the signal that is being detected is 1000x lower than the noise floor.

Currently, we are looking at methods such as lock-in amplification/detection and other types of filtering but it was an open question to see which are common methods to improve the detection of a signal in a noisy background.

Answer 1

The only way that a signal that is "buried in the noise" can be detected is if you can run the signal + noise through some filter that attenuates the noise more than it attenuates the signal. At which point the signal is no longer buried in the noise, so "buried in the noise" was just a hasty assumption.

In radio carrying an audio (or Morse code) signal in SSB or AM, you take the signal + noise and you filter it by the approximate bandwidth of the signal, then you run it through a detector.

In radio carrying digital data, you run it through a matched filter and then a detector.

In spread-spectrum radio, you correlate the signal + noise with a pseudo-random sequence, then bandpass filter, then detect.

In visual systems, you correlate the noisy image with a 2-D prototype of the anticipated signal, or you run the noisy image through a spatial low-pass filter, then you detect.

In all cases, the signal has to be distinct in some way from the noise -- if it is not, then you cannot filter out the noise without filtering out the signal, too.

I'll add to this:

At the top level, a filter for signals is like a coffee filter or a colander: you have the stuff you want (coffee or fresh-cooked pasta) and the stuff you don't want (coffee grounds, or starchy hot water), but it's all mixed together. So you run the mess through a filter. In the case of coffee, you keep the stuff that gets through the filter. In the case of the colander, you keep the stuff that gets left behind. In either case, you're using the fact that one thing (coffee grounds or pieces of pasta) is bigger than the other (water molecules and all the other stuff you want in coffee, and don't want in pasta).

A signal filter does the same thing -- you get rid of what you don't want because it is different from what you do want. If you can't figure out how it's different, and how to build an algorithm to separate it -- you can't filter your signal from your noise.

Answer 2

The general concept of detecting a signal in noise is knowledge of something about the signal, and ideally something about the noise.

The easiest thing to use is spectral knowledge. If you know the signal occupies some part of the spectrum, then you can safely filter out noise in other parts of the spectrum without losing signal. This is taken to its extreme in the 'lock-in amplifier', which is basically just a method of creating a very narrow bandpass filter at precisely the frequency of the signal.

A more general property of the signal is its waveform. We can correlate the signal plus noise with a copy of this waveform, and then average. The noise does not line up with the correlating waveform so adds as power. The signal does correlate, so adds as voltage, leading to a 3dB improvement in SNR each time the number of averages is doubled.

Answer 3

Another technique that might be useful is autocorrelation

Autocorrelation, also known as serial correlation, is the correlation of a signal with a delayed copy of itself as a function of delay. Informally, it is the similarity between observations as a function of the time lag between them. The analysis of autocorrelation is a mathematical tool for finding repeating patterns, such as the presence of a periodic signal obscured by noise, or identifying the missing fundamental frequency in a signal implied by its harmonic frequencies. It is often used in signal processing for analyzing functions or series of values, such as time domain signals.

Different fields of study define autocorrelation differently, and not all of these definitions are equivalent. In some fields, the term is used interchangeably with autocovariance.

Answer 4

And then there is the idea of averaging multiple independent observations. In overly simplified terms, the aim here is to increase the signal strength while letting the noise cancel itself out, i.e., the desired signal increases faster than the noise, and the more samples you average the better.

Answer 5

If you know exactly what the signal looks like, in the time domain, you can implement Matched Filters to discard noise energy in frequency regions that are not needed to construct the waveform.