Subscribe via RSS
Toggle Submenu
  1. Categories
  2. Recent
  3. Tags
View Replies (3)

Resolved Any way to handle JSON with more than 20 MB?

  1. How-to
0
Votes
Undo
  1. Gerrit Bruns
    Gerrit Bruns
  2. PowerBuilder
  3. Wednesday, 20 April 2022 09:21 AM UTC

Hi,

is there any way to handle JSON with more than 20 MB? 

We want to create a Webservice in SnapDevelop, that sends or retrieves E-Mails. Our application sends the data to the Webservice and the Webservice sends the E-Mail or the retrieves them. The E-Mail can have attachments. 

Outgoing, we create the JSON using the JSONGenerator, which works, until the amount of data exceeds 20 MB. The JSON contains all information that is needed to send the E-Mail. The files are send as Base64 encoded strings inside the JSON. If the JSON would reslt in more than 20 MB, the result is an empty string, as described in the PB Help.

Same problem with incoming data.

The Webservice can handle JSON Data with more than 20 MB if i send them with Postman. 

Is there any way to do that with one reqest?

Any ideas how to do that?

 

Regards,

Gerrit Bruns

  1. JSON
  2. JSONGenerator
  3. JSONParser
  4. PowerBuilder
  5. REST
Cannot load the Rich Text Edit Control
20021 - Loading DLL failed during Appeon Web
Responses (3)
Andreas Mykonios
Andreas Mykonios Accepted Answer Pending Moderation
  1. Wednesday, 20 April 2022 10:11 AM UTC
  2. PowerBuilder
  3. # 1

This already json is compressed?

Andreas.

Comment
Load more comments
Gerrit Bruns
  1. Gerrit Bruns
  2. Wednesday, 20 April 2022 11:53 AM UTC
I have 32 GB in my computer, so that isn't a problem.

The mentioned limitation from PB is the problem.

At the moment i don't see a way to send or receive a request with more than 20 MB, or to handle a JSON with more than 20 MB.



Don't know if there would be a problem with 64 bit. The problem is, that it isn't easy to change our application to 64 bit, due to other 3rd party components. If we just compile it in 64 bit, many thinks won't work. We would have to change many things to make it work under 64 bit.



Regards, Gerrit
  1. Helpful
Chris Pollach @Appeon
  1. Chris Pollach @Appeon
  2. Wednesday, 20 April 2022 12:05 PM UTC
Hi Gerrit ... What about making the JSON "handler" a separate 64 bit App that can be called from your current PB App?
  1. Helpful
Gerrit Bruns
  1. Gerrit Bruns
  2. Wednesday, 20 April 2022 12:30 PM UTC
Hi Chris,

is there no problem with JSON larger than 20 MB in 64 bit? It isn't mentioned if the limitation is only for 32 bit.

If that is no problem, i could give it a try.

Not sure if that would be a good idea, since it is possible that the data contains information for the E-Mail Server authentication, like username and pwd.



What would be the best way to do that? Write everything in a file and then let that application handle the file, create the JSON based on the information and send the request?



Regards, Gerrit

  1. Helpful
There are no comments made yet.
Chris Pollach @Appeon
Chris Pollach @Appeon Accepted Answer Pending Moderation
  1. Wednesday, 20 April 2022 11:35 AM UTC
  2. PowerBuilder
  3. # 2

Hi Gerrit;

  Is there any way that you can have the web service "chunk" the 20M+ file? Say for example, into 3 x 10M chunks?

Regards ... Chris

Comment
Gerrit Bruns
  1. Gerrit Bruns
  2. Wednesday, 20 April 2022 12:14 PM UTC
Hi Chris,

the file isn't necessary a single file.

The data contains the basic information for the E-Mail and can contain multiple files. The problem is, that PB can't handle more than 20 MB.

So i don't see a way to send or receive information in one request, when it is more than 20 MB. One JSON with more than 20 MB and you have problem.



It should be possible to do that in multiple request, something like, send the base-data in one request, the attachments in multiple other request and then a third request to send the mail, or something like that. That would result in overhead for the multiple request and something that checks for not entirely finished requests, to prevent memory issues. And so on. One request for one action would be better, one request to send an E-Mail and one request to get an E-Mail.



Regards, Gerrit
  1. Helpful
Chris Pollach @Appeon
  1. Chris Pollach @Appeon
  2. Wednesday, 20 April 2022 14:55 PM UTC
The other alternative (like I have done with huge CSV files) is to use the FileRead() and FileWrite() commands (NOT the "EX" ones) and read & write large files in 32K "chunks" under PowerScript control. Then I could either break up the large files or process them chunk by chunk under PowerScript control. This approach allowed my PB Apps to easily handle super large data streams (100M+). Another "food for thought". ;-)
  1. Helpful
There are no comments made yet.
Ken Guo @Appeon
Ken Guo @Appeon Accepted Answer Pending Moderation
  1. Thursday, 21 April 2022 08:19 AM UTC
  2. PowerBuilder
  3. # 3

Hi Gerrit,

To avoid any abnormal issues with the memory, PB sets limits regarding all Json Object and supports 20 MB top or 100,000 data rows of json file. Please refer to the link below for details:
https://docs.appeon.com/pb2021/objects_and_controls/JSONPackage_object.html 

Therefore, to make sure PB run normally, I suggest that you try to keep the Json file within 20 MB. You can try to split the data in one big Json file into multiple small Json files.

Regards,
Ken

Comment
Andreas Mykonios
  1. Andreas Mykonios
  2. Thursday, 21 April 2022 09:01 AM UTC
Hi Ken. In this text it's stated that it is not recommended to use this object to process large data (20 MB or 100,000 data rows can be considered as large data based on our tests). It's not stated that there is an effective limit. If there is a limit, this should be documented...

Also Gerrit stated that he wants to create the json file with JSONGenerator, which I understand has the same limits with JSONPackage...

Andreas.
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.