Skip to main content
4 of 4
added 14 characters in body
Curd
  • 16.5k
  • 36
  • 46

I guess it's just a matter of precision.

Suppose the factor 2.36 is represented by a fixed point number with only 4 bits to the right of decimal point (fractional bits), you get:

56 * 2.36 in fixed point representation with only 4 fractional bits:
(56 * round(2.36 * 2^4)) / 2^4 = 
(56 * round(37.76)) / 16 =
(56 * 38) / 16 = 133

The solution would be just to use more fractional bits (maybe 8).

56 * 2.36 in fixed point representation with 8 fractional bits:
(56 * round(2.36 * 2^8)) / 2^8 = 
(56 * round(604.16)) / 2^8 =
(56 * 604) / 256 = 132

But still then it is possible for some factors to get the same effect, only less likely/frequent.

It only works always perfectly if the fixed point representation of the factor doesn't loose any bits by rounding.

This is the case if (and only if):

  • the fractional part of the factor is a multiple of 1/2^n and
  • the fixed point representation of the factor uses at least n fractional bits.
Curd
  • 16.5k
  • 36
  • 46