Emerson Exchange 365

• Not Answered

Using decimal points on a CALC block

Hi experts! We have a calculation and we must use all digits (i.e. 0.066666663) to get the right value for process control, but if we use a CALC block or standard math blocks the result doesn't give us all digits. Is tehre any way to do it? Does anybody have an idea?

Today the CALC block has the following script:

OUT1 := ((0.066666663 * (IN1 * IN1 * IN1)) - (19.799999 * (IN1 * IN1)) + (1960.6827 * IN1) - 64635.16

And if we compare with a standard calculator, assuming IN1 = 97.75:

CALC = 98.2227

Calculator = 98.49984704

Thanks for your help.

Samuel Bandeira

• 2 Jun 2015 10:11 AM
• Cancel

5 Replies

• Maarten van der Waal

  

  • 2 Jun 2015 10:58 AM

  It all has to do with numerical accuracy for floats.
  A float has an accuracy of about 6 decimal digits, so the number 0.066666663 wil be stored as .0666667, 19.799999 will be 19.8 etc.
  So you will have to think of another way to do your calculation.
  
  Second, i would be better to use a formula like O := I * ( I * ( I * .066 ) - 19.79 ) + 1960.68 ) - 64635.16
  This uses much less processorpower, and may result in better accuracy!

  • Cancel
• Youssef.El-Bahtimy

  

  • 8 Jun 2015 9:19 PM

  Do you actually have instrumentation that can be trusted to the precision you have set forth? As Maarten stated, when making use of floating points, you trade precision for processing/overhead.
  
  Even if you were to convert the numbers to integers first (multiplying by # of required decimals), then perform the arithmetic, then convert the final value as a string to ensure exact representation, I don't believe a 32 bit unsigned integer will allow for a number as large as you require (4294967295 max I think).
  
  What does your calculation actually do? Is there a physical and measurable basis for the precision of the calculation?

  • Cancel
• jgrunschel

  

  • 24 Jun 2020 5:29 PM

  In reply to Maarten van der Waal:

  I know this is old, but I think your formula has an extra closing parenthesis.  The formula that worked for me is:

  OUT1 := IN1 * (IN1 * (0.066666663 * IN1 - 19.799999) + 1960.6827) - 64635.16

  • Cancel
• Jason.Brumfield

  

  • 6 Jul 2020 10:56 PM

  In reply to jgrunschel:

  This is a crude solution but I have had to do it  occasionally to transfer serial data on occasion. Multiply the decimals to make them whole numbers, do the calculation, then divide it back down to return to floating decimal. Not ideal so don't flame me, but it will work in a pinch.

  • Cancel
• Andre Dicaire

  

  • 7 Jul 2020 5:07 AM

  In reply to Jason.Brumfield:

  the inaccuracy is a resolution issue that occurs when one adds a small number to a large number. the 32 bit float uses a 22 bit mantissa, which sets the resolution. When a number passes 4194304, any number added is rounded to the nearest while number.
  
  with 3rd order polynomial we are adding small and large values where the log value of both is pushed to the larger number and results in rounding error in the terms.
  
  one way to improve the accuracy is to break the curve into two or more splines, were each section can be represented by a simpler expression, such as two second order curves.
  
  Given that most process measurements rarely have accuracy better than 0.1 %, the significant digits of the result cannot be better than the source data.
  
  none the less, when accuracy is lacking, we can improve the result from 32 bit floats by simplifying the math and avoid the problematic rounding error that creeps in when adding small to big values.

  Andre Dicaire

  • Cancel