1. Martin Kaltenböck
  2. PowerBuilder
  3. Monday, 14 March 2022 11:09 AM UTC

Hello,

using PB 2019 R3 2703, Windows 10 64

in my application I must make use of the LongLong datatype.

Unfortunately I noticed calculation errors on doing a integer division with this datatype.

Here is the code:

longLong x, y

x = 7129750713734852035      // LongLong-Value with 19 digits

y = x / 16                  //  y gets the value 445609419608428224, which is wrong

The same division done in Windows Calc.exe and other calulators calcuate a value of 445609419608428252

The error seems to happen with large values only

Here are examples

PB:
1) 7.129.750.713.734.852.035 / 16 = 445.609.419.608.428.224      // wrong
2) 712.975.071.373.485.203 / 16   = 44.560.941.960.842.824        // wrong
3) 71.297.507.137.348.520 / 16      = 4.456.094.196.084.282           // OK

Windows Calc.exe:
1) 7.129.750.713.734.852.035 / 16 = 445.609.419.608.428.252
29 712.975.071.373.485.203 / 16   = 44.560.941.960.842.825
3) 71.297.507.137.348.520 / 16      = 4.456.094.196.084.282

Any idea, whats going wrong?

Thank you, Martin

Accepted Answer
John Fauss Accepted Answer Pending Moderation
  1. Monday, 14 March 2022 14:05 PM UTC
  2. PowerBuilder
  3. # Permalink

I'm able to replicate this issue in PB 2019 R3 Build 2703 and in PB 2021 R0 Build 1506.

As a work-around, you can perform the calculation correctly using the Decimal data type.

I suggest you open a bug report/ticket to report this problem:

   https://www.appeon.com/standardsupport/newbug

Comment
There are no comments made yet.
Martin Kaltenböck Accepted Answer Pending Moderation
  1. Tuesday, 15 March 2022 08:00 AM UTC
  2. PowerBuilder
  3. # 1

Thank you John, I  already worked around this by using the decimal data type.

I will report this as a bug.

 

Comment
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.