1. Sverre Tvedt
  2. PowerBuilder
  3. Tuesday, 19 November 2019 11:26 AM UTC

PB 2017 R3 build 1858  32 bit application

I have been doing a lot of web client refactoring lately, using JSON.

I some instances I send and receive huge JSON return data sets, of a complex, nested  nature and containing base64 encoded images.

One serious problem is that I cannot encode image blobs of more than up to about 12.5 Mb(13 or more always fails) size using Base64Encode. The result string from that function becomes Null. I have not tested decoding.

This problem is reproducible with a very simple piece of code:

ls_filepath = "c:\temp\volume_test.txt"
li_fileno = fileopen (ls_filepath,streammode!,read!, shared!)  // put 13 Mb here
ll_count = filereadex(li_fileno, lbl_data)   // long, blob
fileclose (li_fileno)
ls_base64  = iuo_coder.base64encode(lbl_data)  // becomes Null :(

The other problem, which may be related, is about retrieving and parsing large JSON data sets containing base64 encoded images. If the size is for instance about 18 mb (the limit may be the same as above, not tested), JsonPackage.Getvalue returns an ugly Null, preventing further parsing.  In this case the value is an object array, and the base64 string is one of the members of one array item.  The size is the only difference from working data sets. It works nicely if the size is not too high, and the system has been in production for almost a year, with high intensiity utilization, so I believe it is all about capacity limitations. Further the data does not have to be a base64 encoded image. Any type of base64 encoded document data behaves similarly.

This occurs in IDE and runtime.

pbcrypt170.dll is deployed to runtime environments.

regards

sverre

Sverre Tvedt Accepted Answer Pending Moderation
  1. Wednesday, 4 December 2019 07:03 AM UTC
  2. PowerBuilder
  3. # 1

It turns out that it is all caused  capacity limitations on httpclient.getresponsebody   and the string size handled by JsonPackage & JsonParser, a limitation said to be about 20Mb. When this happens, application memory seems to be overwritten and the application fails or crashes.

This also explains a number of strange instabilities I have experienced. May I suggest that these limitations are handled by means of error codes, as it is impossible to know the size of an incoming response body.

Since the cause of the problem now is known, I have marked the question as resolved.

 

 

Comment
There are no comments made yet.
Ken Guo @Appeon Accepted Answer Pending Moderation
  1. Wednesday, 20 November 2019 05:33 AM UTC
  2. PowerBuilder
  3. # 2

Hi Sverre,

I used the code you provided and created a test case using a file which exceeds 13 MB, even 70 MB and it works well.

Can you reproduce it using a small test case? If yes, it would be better if you could submit a ticket on our support ticketing system (https://www.appeon.com/standardsupport/search), please also upload the test case and the 12.5 MB file you used when you submit this bug so that the Appeon developers can analyze it and find out the cause of it more quickly.

  

Regards,

Ken

Comment
  1. Sverre Tvedt
  2. Wednesday, 20 November 2019 11:55 AM UTC
I now think the real problem is the JsonPackage.getvalue. It fails to perform when the size of the object array to be retrieved is too large. In fact this also causes memory leaks or other IDE internal errors that may be the real cause of the problem with Base64encode. I'll submit a bug report on this, supplying the Json string (verified with an independent tool). The problem is reproducible in version 2019 as well.

Originally I made my own Json parser object as a wrapper around JsonPackage & JsonParser, to get past the deficiencies of both, but combining the best of their methods. GetValue has until now been the last surviving useful function of JsonPackage.
  1. Helpful
  1. Sverre Tvedt
  2. Thursday, 21 November 2019 14:22 PM UTC
The problems seems to be primarily Getvalue or my use of it. When it malfunctions, base64encode starts returning null as well and PB IDE has to be restarted. I'm writing a replacement function in pure PB Script to see if this helps.
  1. Helpful
  1. Sverre Tvedt
  2. Friday, 22 November 2019 13:29 PM UTC
Replacement functions removed the symtoms, so I now abandon jsonpackage completely. However, working with the POS function with large Json strings is however slower and less effective.

Is there a way with JsonParser to get a similar result as with JsonPackage.Getvalue? That is, extract the complete string expression following an element name, be it object, string arrary or complex object array.
  1. Helpful
There are no comments made yet.
mike S Accepted Answer Pending Moderation
  1. Tuesday, 19 November 2019 18:57 PM UTC
  2. PowerBuilder
  3. # 3

report this as a bug:

https://www.appeon.com/standardsupport/search

and make sure you send a sample if you can.  it helps them fix it faster.

and as a workaround, use roland's: 

"I have examples of base64 encode/decode using Win API functions in this example:

http://www.topwizprogramming.com/freecode_cryptoapi.html

"

 

 

Comment
  1. Sverre Tvedt
  2. Wednesday, 20 November 2019 12:24 PM UTC
Done
  1. Helpful
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Tuesday, 19 November 2019 18:09 PM UTC
  2. PowerBuilder
  3. # 4

I have examples of base64 encode/decode using Win API functions in this example:

http://www.topwizprogramming.com/freecode_cryptoapi.html

 

Comment
  1. Sverre Tvedt
  2. Wednesday, 20 November 2019 12:29 PM UTC
That's what I originally had, but switched to the Coder object some time ago. It is possible that the encoding method is affected by something else, not being the root cause. See remark to Ken
  1. Helpful
There are no comments made yet.
Kevin Ridley Accepted Answer Pending Moderation
  1. Tuesday, 19 November 2019 17:30 PM UTC
  2. PowerBuilder
  3. # 5

Wondering if there's an outside chance this could have anything to do with the PBMaxBlobsize attribute?  Chris?

Comment
  1. Chris Pollach @Appeon
  2. Tuesday, 19 November 2019 19:10 PM UTC
Hi Kevin;

No, this parameter controls DBMS Blob access / updating. What Sverre is talking about is JSON parsing & packaging of data that includes large images encoded into Base64.

Regards ... Chris
  1. Helpful
  1. Kevin Ridley
  2. Tuesday, 19 November 2019 20:33 PM UTC
Ok didn't know it was only DB related. Oh well.
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Tuesday, 19 November 2019 16:12 PM UTC
  2. PowerBuilder
  3. # 6

You might have more success building a 64 bit executable. 32 bit is quite limited.

Comment
  1. Sverre Tvedt
  2. Wednesday, 20 November 2019 12:26 PM UTC
On the schedule for next year. A huge legacy application system is expected to be fine tuned to its 32 bit development environment.
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.
We use cookies which are necessary for the proper functioning of our websites. We also use cookies to analyze our traffic, improve your experience and provide social media features. If you continue to use this site, you consent to our use of cookies.