# Discarding divergent sensor data from running average

I'm building a inductive vehicle loop detector. By measuring the frequency of a LC-oscillator where the cars will pass over the inductor of the LC-Oscillator.

If the cars pass over the inductor the frequency will change over time according to one of the following profiles. The frequency will also slightly change / drift based on environmental variables: like temperature, humidity or other variances in the environment of the loop. To filter these last variations out, I used running average and std_variation based on: This post/comment on stack exchange. And was planning to detect vehicles when the frequency would go too many standard deviations away from the average.

The measurements made when there is a vehicle present shouldn't be added to the average calculation. This is easy to do when a certain threshold is passed.

But the problem is when a car is approaching the frequency will start to change but hasn't yet passed the threshold so these samples are added to the statistics. When a car approaches very slowly the standard deviation will be changed so much that the car will never be detected.

What solutions / alternatives / suggestions do you have to detect approaching cars and discard the values from the statistics for the reference empty loop.

Thanks in advance

Update

## - Link to the post in the DSP section

• This question might be more appropriate for the signal processing stack. – Edgar Brown Mar 28 at 13:12
• I posted this question there first, but I got no response. I think this problem / situation applies to many of us who would like to detect a variation in a sensor value compared to an average. But don't want the average to be influenced by the detect value. – Bruce Mar 28 at 13:35
• Please add a link to the question there, I’ll try to put together an answer in the next few days. – Edgar Brown Mar 28 at 13:40
• Are vehicle-caused frequency variations unipolar? You might try weighting samples going into your average differently. Averaging to establish a threshold seems fraught with problems: dense traffic, stalled or tied-up traffic for example (related to your rate difficulty). – glen_geek Mar 28 at 14:36
• And if you simply integrate this time-dependent signal, can the "area below the curve" have two different thresholds ("in" and "out")? You could add two examples of plots like "coming fast" and "coming slowly." – Dirceu Rodrigues Jr Mar 28 at 17:35

## 2 Answers

You have built a system that is by design rate sensitive, to allow it to reject large but slow environmental changes. So it is now insensitive to slow target changes, as when a car approaches very slowly.

There is no magic that you can apply to discriminate between these. You have got the classic binary classificiation problem. If you could get a clear measurement difference between your detect and no detect measurements, then you can get 100% success rate with 0% false positives or false negatives. If you have overlapping measurements (which you allow for by having a car approach very slowly) then any rate of change threshhold you set will allow at least one or the other error to occur. A very low threshhold would get falsely triggered by sudden temperature changes, a high threshold will miss slow cars.

You either need to specify the present behaviour in terms of rate threshhold versus false negative and false positive rate, or take some steps to incorporate your knowledge of the system to reduce the allowance you need to make for environmental effects. Perhaps measure temperature, and if it correlates with frequency drift, use this to reduce the allowance you make for it. The ideal system, as used in food metal detection which has to be extremely sensitive, is to use two loops, and detect differentially, however you might not have the freedom to modify the road in this way.

• Thank you for your answer, it's not really the solution for my problem I'm looking for, but it's good food for thought. – Bruce Mar 29 at 9:40

I already posted an answer in the DSP stack exchange. But just for completeness here is the main point:

A very similar problem to this is in the biological field with the detection of spikes in neural action potentials. The way this is commonly solved is by using the standard deviation of a section of the input that is "known" to be free of signal and is therefore noise and using that standard deviation to set the threshold (in these applications the voltage threshold is set between 4 to 6 standard deviations of the noise). Of course, this has not completely solved the problem, as you still have to figure out how to get the signal out of your noise calculations.

A clever way to do this, is to rely on the statistics of the noise being Gaussian, and only use the tail of the total input distribution where you are almost assured that all you will see is noise. You can then infer the noise standard deviation only from that tail. A family of algorithms can be derived from this, in neural spike detection these are known as the ADAband family of algorithms.

The idea is:

1. calculate the power (squared magnitude) of the input
2. calculate a short-term estimate of the standard deviation using a small number of samples.
3. store a number of estimates, let's say 100 (a longer time segment) in an array.
4. sort the stored values (this is in a sense similar to a median filter).
5. use only the bottom bins (e.g., the first 15 represent the bottom 15% of the corrected standard deviation estimates.
6. weigh each bin appropriately according to the chi-squared distribution it comes from and combine them to produce a better standard deviation estimate.
7. appropriately filter/combine successive estimates to reduce its variability.

The advantage with this algorithm is that the threshold adapts to the conditions of the input and, given that it is basically using an intrinsic threshold that lies deep within the noise region (e.g., the bottom 15% of the standard deviation estimates) it is a very reliable estimate of the existing noise.

The problem with the algorithm is that the relatively long records used determine adaptation speed (it is intrinsically an FIR combined with an odd variant of a median filter), this is an issue with spike detection in the presence of extracellular potentials. But in your case, that you are simply trying to get rid of slow circuit variations, adaptation speeds in the multi-minute ranges should be more than adequate.