• You are here:  
  • Home
  • Q&A
  • Q&A
  • PowerBuilder
  • Numeric Value with 0 Precision Pulling Erroneous Decimals

Subscribe via RSS
Toggle Submenu
  1. Categories
  2. Recent
  3. Tags
View Replies (4)

Resolved Numeric Value with 0 Precision Pulling Erroneous Decimals

  1. How-to
0
Votes
Undo
  1. Matt Kluzak
    Matt Kluzak
  2. PowerBuilder
  3. Thursday, 26 August 2021 14:50 PM UTC

Hello,

 

I have an application that looks at a Sybase ASE 15.7 database with fields of Numeric(18,0). All values in the database are whole numbers. I recently noticed that in one datawindow the field would display properly as 1002000000002399, but after it was assigned to a variable of type double via datawindow.GetItemNumber() and then read sent to another datawindow, it would show up as 1002000000002400. I changed the variable from double to a decimal, and during debug I could see the value it was pulling was 1002000000002399.04. I have had other instances where the number is lightly lower, such as 100200000000219 being read as 100200000000218.9941. So this means I can't simply just truncate the decimals.

I am not sure if this only started happening after updating Sybase ASE 15.7 from SP103 to SP140, but that's when I noticed it. This is an existing application from when I started that always ran as a background job.

Other than modifying the field in every table and then throughout the application to be some other type, like BIGINT, is there a way I can fix this issue?

 

Thanks

The new webbrowser control
Orca still generate pbg files
Accepted Answer
John Fauss
John Fauss Accepted Answer Pending Moderation
  1. Thursday, 26 August 2021 15:11 PM UTC
  2. PowerBuilder
  3. # Permalink

I don't think the problem is in the database.

A variable of type Double in PB is a floating-point number that has 15 digits of precision.

If you need to store a whole number value of 18 digits, I suggest you use the PB Decimal datatype, which can store a numeric value with up to 28 digits.

Refer to the PB Help section titled "standard datatypes".

Comment
Chris Pollach @Appeon
  1. Chris Pollach @Appeon
  2. Thursday, 26 August 2021 15:40 PM UTC
Hi John ... Yep - Decimal is the way to go too IMHO (my $0.02)
  1. Helpful
Matt Kluzak
  1. Matt Kluzak
  2. Monday, 30 August 2021 16:34 PM UTC
In the end, I switched the variable type from double to decimal and then switched the function to get the value from datawindow.GetItemNumber() to datawindow.GetItemDecimal().
  1. Helpful
There are no comments made yet.
Responses (4)
Olan Knight
Olan Knight Accepted Answer Pending Moderation
  1. Thursday, 26 August 2021 21:08 PM UTC
  2. PowerBuilder
  3. # 1

Decimals are the way to go, but one other option is the LONGLONG datatype.

Comment
Arnd Schmidt
  1. Arnd Schmidt
  2. Thursday, 26 August 2021 21:22 PM UTC
As long(long) as you do not need 19+ digits ;-)

https://en.wikipedia.org/wiki/Integer_(computer_science)#Long_long
  1. Helpful
Matt Kluzak
  1. Matt Kluzak
  2. Friday, 27 August 2021 13:41 PM UTC
I tried it out with my variable as type longlong. The messagebox displaying datawindow.GetItemDecimal showed value 1002000000002399 as 1002000000002400, but then passing the value into the variable with datawindow.GetItemDecimal and datawindow.GetItemNumber both showed 1002000000002399 as 1002000000002399.



However, if I assign the value to the variable and then have the messagebox display the variable, it shows up properly as 1002000000002399. So having the messagebox display the datawindow.GetItemDecimal call directly causes it to be shown wrong.



I only need 18 digits, as the database field is Numeric(18,0), so longlong is viable. However, I think I will explore the decimal route as that is what the datawindow is currently using.
  1. Helpful
There are no comments made yet.
Arnd Schmidt
Arnd Schmidt Accepted Answer Pending Moderation
  1. Thursday, 26 August 2021 15:04 PM UTC
  2. PowerBuilder
  3. # 2

Hi Matt,

for decimals better use getItemDecimal () instead of getItemNumber().

hth

Arnd

Comment
There are no comments made yet.
Matt Kluzak
Matt Kluzak Accepted Answer Pending Moderation
  1. Thursday, 26 August 2021 19:52 PM UTC
  2. PowerBuilder
  3. # 3

I did try switching the variable to a Decimal. The strange thing is, if I have a messagebox display the value of datawindow.GetItemDecimal() it shows value 1002000000002400 instead of 1002000000002399, but stepping through debug if I assign datawindow.GetItemDecimal() to a Decimal variable its value shows properly as 1002000000002399. However, with value 1002000000000219, the messagebox  and the Decimal variable both display it properly. 

This inconsistency makes me rather nervous.

Comment
Load more comments
John Fauss
  1. John Fauss
  2. Thursday, 26 August 2021 22:23 PM UTC
OK - The next thing I suggest you try is running the app with the database driver tract enabled. SQLCA.DBMS="TRACE ASE" instead of "ASE", to see what value(s) the PB database driver is receiving from the database. It might be better if you have a small, sample application that pulls data from this table using the same DataWindow.
  1. Helpful 1
Matt Kluzak
  1. Matt Kluzak
  2. Friday, 27 August 2021 13:44 PM UTC
Are you meaning for me to set this in the applications config file? Currently, the entry is "DBMS=SYC Adaptive Server Enterprise". Would I change this to "DBMS=TRACE SYC Adaptive Server Enterprise"?
  1. Helpful
John Fauss
  1. John Fauss
  2. Friday, 27 August 2021 14:13 PM UTC
Correct. I've not done this for some time, but if memory serves, this will create a text file (called "C:\PBTrace.log", I think) showing all of the communications between the PB database driver and the Sybase ASE Database.
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.