1. Martin Kaltenböck
  2. PowerBuilder
  3. Monday, 14 March 2022 11:09 AM UTC

Hello,

using PB 2019 R3 2703, Windows 10 64

in my application I must make use of the LongLong datatype.

Unfortunately I noticed calculation errors on doing a integer division with this datatype.

Here is the code:

longLong x, y

x = 7129750713734852035      // LongLong-Value with 19 digits

y = x / 16                  //  y gets the value 445609419608428224, which is wrong

The same division done in Windows Calc.exe and other calulators calcuate a value of 445609419608428252

The error seems to happen with large values only

Here are examples

PB:
1) 7.129.750.713.734.852.035 / 16 = 445.609.419.608.428.224      // wrong
2) 712.975.071.373.485.203 / 16   = 44.560.941.960.842.824        // wrong
3) 71.297.507.137.348.520 / 16      = 4.456.094.196.084.282           // OK

Windows Calc.exe:
1) 7.129.750.713.734.852.035 / 16 = 445.609.419.608.428.252
29 712.975.071.373.485.203 / 16   = 44.560.941.960.842.825
3) 71.297.507.137.348.520 / 16      = 4.456.094.196.084.282

Any idea, whats going wrong?

Thank you, Martin

Accepted Answer
John Fauss Accepted Answer Pending Moderation
  1. Monday, 14 March 2022 14:05 PM UTC
  2. PowerBuilder
  3. # Permalink

I'm able to replicate this issue in PB 2019 R3 Build 2703 and in PB 2021 R0 Build 1506.

As a work-around, you can perform the calculation correctly using the Decimal data type.

I suggest you open a bug report/ticket to report this problem:

   https://www.appeon.com/standardsupport/newbug

Comment
There are no comments made yet.
Martin Kaltenböck Accepted Answer Pending Moderation
  1. Tuesday, 15 March 2022 08:00 AM UTC
  2. PowerBuilder
  3. # 1

Thank you John, I  already worked around this by using the decimal data type.

I will report this as a bug.

 

Comment
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.
We use cookies which are necessary for the proper functioning of our websites. We also use cookies to analyze our traffic, improve your experience and provide social media features. If you continue to use this site, you consent to our use of cookies.