Bounty Ended with 50 reputation awarded by zacharoni16
3 added 1141 characters in body
source | link

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since thatthat's how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

The lowest of the 5 peaks is over 3 times higher than the highest background noise. This is of course assuming these 5 bumps represent events you want to detect and the rest doesn't.

Added in response to comments:

I did the filters in the time domain, so I don't know the frequency response directly. For the low pass filter I convolved the input signal with a COS^2 filter kernel. If I remember right, the radius (distance from center to edge) of the kernel as a few 100 ms. I experimented with the value until the plot looked good. To low pass filter the RMS, I used the same filter kernel but this time with a radius of about a second. I don't remember exactly. Experiment until you get good results.

The non-linear filter did not detect doublets. As I said, I found the difference between the current point and the lowest of all the points within 1 second before, and also the difference between the current point and the lowest of all the points within 1 second after. Then I took the min of those two.

The software I used was a program I hacked up for this purpose. I already had various routines to read and write CSV files, so all I had to write was the filtering code, which is very simple. The rest was done with pre-existing programs I have for manipulating and plotting CSV files.

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

The lowest of the 5 peaks is over 3 times higher than the highest background noise. This is of course assuming these 5 bumps represent events you want to detect and the rest doesn't.

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that's how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

The lowest of the 5 peaks is over 3 times higher than the highest background noise. This is of course assuming these 5 bumps represent events you want to detect and the rest doesn't.

Added in response to comments:

I did the filters in the time domain, so I don't know the frequency response directly. For the low pass filter I convolved the input signal with a COS^2 filter kernel. If I remember right, the radius (distance from center to edge) of the kernel as a few 100 ms. I experimented with the value until the plot looked good. To low pass filter the RMS, I used the same filter kernel but this time with a radius of about a second. I don't remember exactly. Experiment until you get good results.

The non-linear filter did not detect doublets. As I said, I found the difference between the current point and the lowest of all the points within 1 second before, and also the difference between the current point and the lowest of all the points within 1 second after. Then I took the min of those two.

The software I used was a program I hacked up for this purpose. I already had various routines to read and write CSV files, so all I had to write was the filtering code, which is very simple. The rest was done with pre-existing programs I have for manipulating and plotting CSV files.

    Mod Removes Wiki by clabacchio
2 added 600 characters in body
source | link

This looks like it can be solved by fairly straight forward filtering. Here is your original data:

That's too much to see what goes on in a individual event at the level of detail appropriate for here. Here is just the data from second 26 to 28:

I had originally thought to low pass filter this, but that doesn't work because there isn't a low frequency signal in there. The amplitude of the high frequency signal goes up instead. Here is a low pass superimposed onto the original:

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

This now points out a obvious approach for detecting the event. There is a lot more signal amplitude during the event than otherwise. We can detect this by computing the RMS and applying some low pass filtering:

Zooming back at the whole data, we see:

This clearly identifies five events in the data, although I don't know if that's what this data is supposed to show. Looking at the events more closely, you notice that each of them has low dips about 1 second before and after the peaks. This means more can be done if simply thresholding the RMS signal as it is now is not good enough. For example, a simple algorithm that looked for the height of a point relative to the lowest within 1 second either way should further reduce the background noise. Another way to say about the same thing is to differentiate this signal looking for the rise over a 1 second period. A pothole event would then be detected by a doublet, meaning a high peak follwed by a low peak.

Another way of looking at this is to band pass the RMS signal. It is already low pass filtered, but since you are looking for sudden events with strong slopes, lopping off some of the low frequencies should work to reduce the background noise too.

There are lots of ways to refine the signal from here, but hopefully I've shown how to get to at least a first pass useful result.

Added:

I was curious how well looking for dips either side of a peak would work, so I tried it. I used a non-linear filter starting with the RMS from the previous plot. The value of each point is the minimum of how much it is above the lowest point in the previous second and the lowest point in the next second. The result looks quite good:

The lowest of the 5 peaks is over 3 times higher than the highest background noise. This is of course assuming these 5 bumps represent events you want to detect and the rest doesn't.

This looks like it can be solved by fairly straight forward filtering. Here is your original data:

That's too much to see what goes on in a individual event at the level of detail appropriate for here. Here is just the data from second 26 to 28:

I had originally thought to low pass filter this, but that doesn't work because there isn't a low frequency signal in there. The amplitude of the high frequency signal goes up instead. Here is a low pass superimposed onto the original:

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

This now points out a obvious approach for detecting the event. There is a lot more signal amplitude during the event than otherwise. We can detect this by computing the RMS and applying some low pass filtering:

Zooming back at the whole data, we see:

This clearly identifies five events in the data, although I don't know if that's what this data is supposed to show. Looking at the events more closely, you notice that each of them has low dips about 1 second before and after the peaks. This means more can be done if simply thresholding the RMS signal as it is now is not good enough. For example, a simple algorithm that looked for the height of a point relative to the lowest within 1 second either way should further reduce the background noise. Another way to say about the same thing is to differentiate this signal looking for the rise over a 1 second period. A pothole event would then be detected by a doublet, meaning a high peak follwed by a low peak.

Another way of looking at this is to band pass the RMS signal. It is already low pass filtered, but since you are looking for sudden events with strong slopes, lopping off some of the low frequencies should work to reduce the background noise too.

There are lots of ways to refine the signal from here, but hopefully I've shown how to get to at least a first pass useful result.

This looks like it can be solved by fairly straight forward filtering. Here is your original data:

That's too much to see what goes on in a individual event at the level of detail appropriate for here. Here is just the data from second 26 to 28:

I had originally thought to low pass filter this, but that doesn't work because there isn't a low frequency signal in there. The amplitude of the high frequency signal goes up instead. Here is a low pass superimposed onto the original:

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

This now points out a obvious approach for detecting the event. There is a lot more signal amplitude during the event than otherwise. We can detect this by computing the RMS and applying some low pass filtering:

Zooming back at the whole data, we see:

This clearly identifies five events in the data, although I don't know if that's what this data is supposed to show. Looking at the events more closely, you notice that each of them has low dips about 1 second before and after the peaks. This means more can be done if simply thresholding the RMS signal as it is now is not good enough. For example, a simple algorithm that looked for the height of a point relative to the lowest within 1 second either way should further reduce the background noise. Another way to say about the same thing is to differentiate this signal looking for the rise over a 1 second period. A pothole event would then be detected by a doublet, meaning a high peak follwed by a low peak.

Another way of looking at this is to band pass the RMS signal. It is already low pass filtered, but since you are looking for sudden events with strong slopes, lopping off some of the low frequencies should work to reduce the background noise too.

There are lots of ways to refine the signal from here, but hopefully I've shown how to get to at least a first pass useful result.

Added:

I was curious how well looking for dips either side of a peak would work, so I tried it. I used a non-linear filter starting with the RMS from the previous plot. The value of each point is the minimum of how much it is above the lowest point in the previous second and the lowest point in the next second. The result looks quite good:

The lowest of the 5 peaks is over 3 times higher than the highest background noise. This is of course assuming these 5 bumps represent events you want to detect and the rest doesn't.

1
source | link

This looks like it can be solved by fairly straight forward filtering. Here is your original data:

That's too much to see what goes on in a individual event at the level of detail appropriate for here. Here is just the data from second 26 to 28:

I had originally thought to low pass filter this, but that doesn't work because there isn't a low frequency signal in there. The amplitude of the high frequency signal goes up instead. Here is a low pass superimposed onto the original:

Notice this follows the "average" of the signal pretty well not during the pothole event. If we subtract this average from the original signal, we are left with much higher excursions from this average during the event than otherwise. Put another way, what we really want is a high pass filter. We'll do that by subtracting the low pass from the original since that how we got here, but in a production system you'd do this by explicitly high pass filtering. Anyway, here is the high pass filtered original:

This now points out a obvious approach for detecting the event. There is a lot more signal amplitude during the event than otherwise. We can detect this by computing the RMS and applying some low pass filtering:

Zooming back at the whole data, we see:

This clearly identifies five events in the data, although I don't know if that's what this data is supposed to show. Looking at the events more closely, you notice that each of them has low dips about 1 second before and after the peaks. This means more can be done if simply thresholding the RMS signal as it is now is not good enough. For example, a simple algorithm that looked for the height of a point relative to the lowest within 1 second either way should further reduce the background noise. Another way to say about the same thing is to differentiate this signal looking for the rise over a 1 second period. A pothole event would then be detected by a doublet, meaning a high peak follwed by a low peak.

Another way of looking at this is to band pass the RMS signal. It is already low pass filtered, but since you are looking for sudden events with strong slopes, lopping off some of the low frequencies should work to reduce the background noise too.

There are lots of ways to refine the signal from here, but hopefully I've shown how to get to at least a first pass useful result.