Subscribe via RSS
Toggle Submenu
  1. Categories
  2. Recent
  3. Tags
View Replies (3)

Resolved Converting Real to String not correct

  1. Issue
0
Votes
Undo
  1. Roland Smith
    Roland Smith
  2. PowerBuilder
  3. Wednesday, 10 July 2019 14:18 PM UTC

It appears that when using the String() function to convert a Real to string, extra digits are added. I am seeing the same issue in the debugger.

I set a Real to:

2.5E38

Then I convert to String:

2.500000072189495E+38

I passed the Real to a dll written in C and converted it to string there and got:

2.500000E+38

As you can see, there are extra digits that shouldn't be there.

I'm thinking of submitting a bug report but thought I would get some feedback first.

 

How to upgrade from PowerBuilder .Net 12.5
XML Import woes
Responses (3)
René Ullrich
René Ullrich Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 05:15 AM UTC
  2. PowerBuilder
  3. # 1

FYI: From PB help:

Real: A signed floating-point number with six digits of precision ...

Comment
There are no comments made yet.
Miguel Leeuwe
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 00:15 AM UTC
  2. PowerBuilder
  3. # 2

In the past there were problems of getting "garbage" values in decimals when using the decimal datatype. For example assigning 3.000009 would result in something like 3.0000089999874 (just an unreal example).

The way to get around it, was to always use double types in the code.

In later versions of pb, I've noticed that now the type which behaves best is the decimal type and definitely not the real type.

Have you tried assigning the real to a decimal type before converting to string? (that is, if the value fits in a decimal of course).

Comment
There are no comments made yet.
Roland Smith
Roland Smith Accepted Answer Pending Moderation
  1. Wednesday, 10 July 2019 17:13 PM UTC
  2. PowerBuilder
  3. # 3

I found that I can use a format string to limit it to the correct number of places:

String(lr_realvalue, "0.000000E+00")

 

Comment
Armeen Mazda @Appeon
  1. Armeen Mazda @Appeon
  2. Wednesday, 10 July 2019 17:55 PM UTC
Thanks for sharing the solution.
  1. Helpful
Michael Kramer
  1. Michael Kramer
  2. Friday, 12 July 2019 12:35 PM UTC


Several PB system functions take double as the input datatype so you get "auto-convert" from real to double. That's what you see here. I had similar issue with LongLong at 16+ digits. They loose precision when calling the Mod function! Why? Because they auto-convert to double.

Solution (sometimes): Assign to DECIMAL which keeps precision (on the cost of performance I presume.)
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.