I'm using a sensor to measure water potential. We record the output value every minute. Up until 4/16, the signal is pretty clean but I need some sort of way to detect when the signal starts to get noisy so that I can go check on the sensor.
Link to Data https://drive.google.com/file/d/1DO531cF0Om5WmfFArvERDBPEx08uHv-y/view?usp=sharing
One man's noise is another man's signal. In this case the noise is the signal of interest, and you need a way to filter out the "noise" - the thing you don't want to see.
In this case it seems that your signal band and noise band are far apart - the noise has a high frequency and the signal has a low frequency. Running the signal through a high-pass filter and then calculate the variance of the result and compare with a threshold should give you a good result.
Not being a signal processing expert I whipped something up in octave using arbitrary constants until I found something that seemed to select for the noise:
[b,a]=butter(2,1/6,"high")
filtered = filter(b,a,data);
noise = movingrms(filtered,96,1);
I used a moving RMS instead of variance because it's more or less the same thing when you have removed the DC component through the high-pass filter, and it was the first I found in octave. I used the butterworth filter because I like it! (And it doesn't have any parameters to tweak). There are many ways to filter and find the resulting energy, even a peak detection could work. Details about selecting the best methods are better asked over at DSP.
The high-pass filter is necessary to remove what you call "seasonal" variations (looks more like a daily cycle) and the downward trend. Experimentally I found that a second order filter is probably necessary.
Then you need to figure out a way to tell the computer how much the signal is fluctuating, and while a simple peak detection on the resulting noise may work, I find that calculating the RMS over a short range is far more robust. In the example above I made that range as long as one of your daily cycles, and that seemed to work well. It is easy to experiment with different lengths to see what works best.
First, you need to define what is your "noise", or what is your "clean" signal.
How do you know that your signal is "clean" when it has no small ripples? Is it the expected behavior of the system?
This is where the frequency analysis becomes invaluable. By passing the signal through an FFT, you can see the frequency response of your signal and then you can determine which one you want to keep or want to remove. Then you apply a frequency filter, in this case, a lowpass filter, to remove the higher frequencies, but first you need to determine how frequent or what is the maximum periodicity of your signal to determine what is the acceptable "clean" signal then filter out the higher frequencies (a.k.a your ripple noise).
If the ease of implementation on small devices is a primary concern, then a simple comparison to a rolling mean might work:
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('Water_potential_(MWP)_767730093.csv', skip_blank_lines=True, header=2, parse_dates=[0])
df = df.rename(columns={'DateTime': 'dt', "LEO-E_2_5_2_MPS-2": 'x'}).set_index('dt')
window=10
df['ma'] = df['x'].rolling(window=window).mean()
df['lagged'] = df['x'].shift(window//2)
df['diff'] = abs(df['lagged'] - df['ma'])
df['ema'] = df['diff'].ewm(alpha=0.1).mean()
print(df)
df.plot(subplots=[['x'], ['ma', 'lagged'], ['diff','ema']], sharex=True, xlabel=None, rot=0)
plt.show()
The 10-sample rolling mean introduces lag, compensate by lagging the sensor reading by half the window. If you take the absolute value of the difference, it is roughly (a) your high frequency noise plus, (b) quantisation resolution error of 0.05 (half your smallest change), plus (c) some error at the points of inflection. You can optionally filter the result before comparing to 0.2 (a threshold which fits your example data), an ema is a quick way to do this if you have floating point.
To do this live with the sensor values you need only keep a small array of past values and track if the window is 'primed' yet. It can be done quite easily on a microprocessor.
