I have been tasked with creating a calibration equation for a power sensor. I have taken data with a calibrated signal generator and have plotted the various frequencies. Now I need to understand the noise that is the data. I have the R2 plotted in each set. The R2 look pretty good like 0.997-0.982 depending on the frequency, but when I back-calculate for the points my numbers can be off sometimes by 3% (not acceptable). I plotted the residuals for the frequency, and its shape is parabolic.

How do I create find the E for the for the Y = a + bX + e. Pics are included.

Residuals vs. Voltage

enter image description here:

enter image description here

  • \$\begingroup\$ I may have overwritten your most recent edit. We were almost simultaneous. \$\endgroup\$ – Transistor Mar 4 '20 at 19:37

We usually use polynoms for correcting nonlinearities in some already linearized analyzers, which fail the linearity check. You have to determine the polynom grade by experience. (Polynom fitting might be unstable/unreliable if the coefficients are small.)

Other way is to measure the error in multiple points and store it as a look-up table. In my experience it is the easier way, both computationally and for the understanding/debugging. And it scales well.

According to your graphs a 2nd degree polynom or a 3-point look-up table might be enough to get the error under 1.5-2%. It might be different for an other sensor.

A small advice for LUT: if i had to do it, i would not store the measured values, but i would fit 2 straight lines on them to minimize the LMS error, then store the 2 endpoints and the crossing point of the lines.

But first of all: you have to make sure that the error is systematic and not random. Meaning, that it is "calibrated" into the power sensor. If it is random, you can not do anything against it (perhaps average multiple measurements). If it is systematic, but dependent on some other physical quantity that is not controlled (e.g. temperature), you have to measure the quantity, and use polynoms or look-up table to correct for it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.