8
\$\begingroup\$

I'm using a sensor to measure water potential. We record the output value every minute. Up until 4/16, the signal is pretty clean but I need some sort of way to detect when the signal starts to get noisy so that I can go check on the sensor.

Link to Data https://drive.google.com/file/d/1DO531cF0Om5WmfFArvERDBPEx08uHv-y/view?usp=sharing

enter image description here

\$\endgroup\$
6
  • \$\begingroup\$ If you don't mind, please provide access to your dataset used for that chart. Can't be much more than 10,000 data points. Or, just create a new dataset constructed by taking the difference between adjacent points (finite difference dataset) and plot that for us along the same time axis, please. \$\endgroup\$ Apr 16, 2023 at 20:18
  • \$\begingroup\$ Link to data should be accessible to anyone with a link. \$\endgroup\$ Apr 16, 2023 at 20:37
  • \$\begingroup\$ So I get something like this using your spreadsheet. (I do also take note of the digitization in the data.) \$\endgroup\$ Apr 16, 2023 at 22:19
  • \$\begingroup\$ @periblepsis what application did you do that in. Also, visually that shows the noise more profoundly but how would I be able to detect the when the noise reaches an unacceptable level? Would that range be something I set? \$\endgroup\$ Apr 16, 2023 at 23:29
  • 1
    \$\begingroup\$ @ChrisH I was just trying to keep things very simple to try out for the questioner. Of course finite differences is functionally similar to using a capacitor DC-blocker. I didn't mean to imply anything by the suggestion except that it is fairly easy to do as a quick test and to show something interesting quickly and easily. I wasn't intending on interfering with anyone's answer here. Nor did I care about adding one. There's not enough information to make me care much. \$\endgroup\$ Apr 17, 2023 at 17:19

3 Answers 3

21
\$\begingroup\$

One man's noise is another man's signal. In this case the noise is the signal of interest, and you need a way to filter out the "noise" - the thing you don't want to see.

In this case it seems that your signal band and noise band are far apart - the noise has a high frequency and the signal has a low frequency. Running the signal through a high-pass filter and then calculate the variance of the result and compare with a threshold should give you a good result.

Not being a signal processing expert I whipped something up in octave using arbitrary constants until I found something that seemed to select for the noise:

[b,a]=butter(2,1/6,"high")
filtered = filter(b,a,data);
noise = movingrms(filtered,96,1);

OPs data through filter and RMS detection

I used a moving RMS instead of variance because it's more or less the same thing when you have removed the DC component through the high-pass filter, and it was the first I found in octave. I used the butterworth filter because I like it! (And it doesn't have any parameters to tweak). There are many ways to filter and find the resulting energy, even a peak detection could work. Details about selecting the best methods are better asked over at DSP.

The high-pass filter is necessary to remove what you call "seasonal" variations (looks more like a daily cycle) and the downward trend. Experimentally I found that a second order filter is probably necessary.

Then you need to figure out a way to tell the computer how much the signal is fluctuating, and while a simple peak detection on the resulting noise may work, I find that calculating the RMS over a short range is far more robust. In the example above I made that range as long as one of your daily cycles, and that seemed to work well. It is easy to experiment with different lengths to see what works best.

\$\endgroup\$
3
  • \$\begingroup\$ Would the fact that the data is seasonal and trending have an effect on that? Without a filter, I calculated the variance of the 24 hours time periods and was expecting the last 24hrs to have the highest variance but it was comparable to the other two and I'm pretty sure it was due to the seasonality of it. \$\endgroup\$ Apr 16, 2023 at 19:44
  • 1
    \$\begingroup\$ @BiosphereEngineering Was this after high-pass-filtering the signal? I'd expect the variance to increase significantly on April 16 when properly filtered, based on the graph you showed. \$\endgroup\$
    – nanofarad
    Apr 17, 2023 at 3:16
  • 4
    \$\begingroup\$ @BiosphereEngineering The filter is necessary because there is a large seasonal variety and trending that you want to remove. A high-pass filter gets rid of all that but leaves most of the noise. I added a few graphs to illustrate the concept and to show that it works with your example data. You will of course have to experiment getting the parameters right for your particular setup. \$\endgroup\$
    – pipe
    Apr 17, 2023 at 10:55
6
\$\begingroup\$

First, you need to define what is your "noise", or what is your "clean" signal.

How do you know that your signal is "clean" when it has no small ripples? Is it the expected behavior of the system?

This is where the frequency analysis becomes invaluable. By passing the signal through an FFT, you can see the frequency response of your signal and then you can determine which one you want to keep or want to remove. Then you apply a frequency filter, in this case, a lowpass filter, to remove the higher frequencies, but first you need to determine how frequent or what is the maximum periodicity of your signal to determine what is the acceptable "clean" signal then filter out the higher frequencies (a.k.a your ripple noise).

\$\endgroup\$
3
\$\begingroup\$

If the ease of implementation on small devices is a primary concern, then a simple comparison to a rolling mean might work:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('Water_potential_(MWP)_767730093.csv', skip_blank_lines=True, header=2, parse_dates=[0])
df = df.rename(columns={'DateTime': 'dt', "LEO-E_2_5_2_MPS-2": 'x'}).set_index('dt')

window=10
df['ma'] = df['x'].rolling(window=window).mean()
df['lagged'] = df['x'].shift(window//2) 
df['diff'] = abs(df['lagged'] - df['ma'])
df['ema'] = df['diff'].ewm(alpha=0.1).mean()
print(df)

df.plot(subplots=[['x'], ['ma', 'lagged'], ['diff','ema']], sharex=True, xlabel=None, rot=0)
plt.show()

The 10-sample rolling mean introduces lag, compensate by lagging the sensor reading by half the window. If you take the absolute value of the difference, it is roughly (a) your high frequency noise plus, (b) quantisation resolution error of 0.05 (half your smallest change), plus (c) some error at the points of inflection. You can optionally filter the result before comparing to 0.2 (a threshold which fits your example data), an ema is a quick way to do this if you have floating point.

plot of results

To do this live with the sensor values you need only keep a small array of past values and track if the window is 'primed' yet. It can be done quite easily on a microprocessor.

\$\endgroup\$
1
  • \$\begingroup\$ What language is your code written in? It seems Python, but I'm not sure because my limited knowledge of it. You should expressly state that in your answer because you cannot expect anyone here knowing any computer language. The answer should give enough information for people to be able to verify your statements, if they are willing to. \$\endgroup\$ May 11, 2023 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.