1. David Motty
  2. PowerBuilder
  3. Tuesday, 3 September 2019 18:58 PM UTC

Hi!

 

                Just wondering if anyone would know if there are any plans (by Appeon) to allow the FileReadEx function to read files (into blobs) that are > than 300 M in size?  Currently, any attempt to do this just results in an error (my test file is 450M)…  I know that creates a very big object in memory, but the function is supposed to be able to take a blob as an argument and a blob is really big.  So actually, this function cannot really handle a blob (at least not big ones). 

               Our application has to read files from the users computer and store them into the database for other users to access.  Most of the files are small, but some files are 500M+ video files.  I understand that this issue can be worked around by taking chunks of the file and writing part of the file at a time to the database in each pass through a loop (which is what I have to do in my current version (2017R3))…  But I was wondering if this limitation will this be fixed in a future version?  Or will the workaround of breaking the file down into chunks, and updating database in a loop, still be necessary?  We try to avoid any manipulation of the files in any way as they may be needed as evidence and are not supposed to be manipulated (even though this is not really manipulation, and objection could be raised)...

 

Also, will the exact size limitation on FileReadEx be published, etc?

 

Thanks in advance for your time.

Dave...

Michael Kramer Accepted Answer Pending Moderation
  1. Wednesday, 4 September 2019 23:36 PM UTC
  2. PowerBuilder
  3. # 1

Hi,

This seems to be a reproducible issue. I could consistently repro in PB 2019, #2082.

To me it looks like PB runtime goes sour when file hits size approx. 300'000 kB.

 

TEST APP

Reads file into variable = data of type BLOB. I tested read into empty blob and tested overwriting existing data in blob. App deployed as separate executable started without PB IDE. Code:

fileNo = FileOpen(fileName, StreamMode!)
try
   bytes = FileReadEx(fileNo, ref data)
catch (RuntimeError ex)
   log("Runtime Error " + string(ex.Number) + " during FileReadEx")
   log("Reason: " + ex.GetMessage())
end try
FileClose(fileNo)

 

TEST Files

File #1 = 300'037 kB.
File #2 = 300'135 kB.

TEST Sequence

  1. Test using file #1 => Read succeeds every time.
  2. Test using file #2 => Read throws EXCEPTION every time. Error message from CATCH
    Runtime Error 21 during FileReadEx
    Reason: Bad runtime function reference at line 17 in clicked event of object cb_test2 of w_main.
  3. Repeat for file #1 => Success
  4. Repeat for file #2 => Failure
  5. . . . and so on . . .

 

QUESTION

Has this issue been reported to Appeon support so that there is a bug number where I can attach my test?

Otherwise I could report this issue myself.

HTH /Michael

Comment
  1. Roland Smith
  2. Thursday, 5 September 2019 13:22 PM UTC
Since FileReadEx returns a long, it should be able to read the max for a long - 2147483647 or 2GB.



You should go ahead and report a bug. If it is a duplicate, they can merge your info internally.
  1. Helpful
  1. Michael Kramer
  2. Thursday, 5 September 2019 16:04 PM UTC
I also expected 2GB as limit (well, 2 GB working memory limit) - but this fails so consistently at almost precisely 300.000 kB so I expect so other limit internal to the PB VM. The error "Bad runtime function" really confused me because I was successfully calling the same function (FileReadEx) both before and after.

True, I will report as bug. Probably in the weekend since work is quite busy. THX /Michael
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Wednesday, 4 September 2019 15:43 PM UTC
  2. PowerBuilder
  3. # 2

From the help PB2017 when you search on "memory management" and then click on the link "configuring memory management"

Configuring memory management

You can set the PB_POOL_THRESHOLD environment variable to specify the threshold at which the PowerBuilder memory manager switches to a different memory allocation strategy.

When most windows, DataWindows, DataStores, or other PowerBuilder objects are destroyed or reclaimed by the garbage collector, the PowerBuilder heap manager returns the memory allocated for each object to a global memory pool and records its availability on a global free list. The freed memory is not returned to the operating system. When a new object is created, PowerBuilder allocates blocks of memory from the global memory pool (if sufficient memory is available in the global free list) or from the operating system (if it is not) to a memory pool for the object.

When the memory required by an object exceeds 256KB, PowerBuilder uses a different strategy. It allocates subsequent memory requirements from the operating system in large blocks, and returns the physical memory to the operating system when the object is destroyed. It retains the virtual memory to reduce fragmentation of the virtual address space.

For most applications and components, the threshold of 256KB at which PowerBuilder switches to the "large blocks" strategy works well and reduces the memory required by an application when it is working at its peak level of activity. However, if you want to keep the overall physical memory usage of your application as low as possible, you can try setting a lower threshold.

The advantage of setting a low threshold is that the size of the global memory pool is reduced. The application does not retain a lot of memory when it is inactive. The disadvantage is that large blocks of memory are allocated for objects that require more memory than the threshold value, so that when the application is running at its peak of activity, it might use more virtual memory than it would with the default threshold.

Setting a low threshold can be beneficial for long-running client applications that use many short-lived objects, where the client application's memory usage varies from low (when idle) to high (when active). For multithreaded applications, such as servers, a higher threshold usually results in lower virtual memory utilization.

Logging heap manager output

You can record diagnostic output from the PowerBuilder heap manager in a file to help you troubleshoot memory allocation issues in your application. The PB_HEAP_LOGFILENAME environment variable specifies the name and location of the file.

If you specify a file name but not a directory, the file is saved in the same directory as the PowerBuilder executable.

If you specify a directory that does not exist, the file is not created.

By default, the log file is overwritten when you restart PowerBuilder. If you want diagnostic output to be appended to the file, set PB_HEAP_LOGFILE_OVERWRITE to false.

You can set the variables in a batch file that launches the application, or as system or user environment variables on the computer or server on which the application or component runs.

Comment
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Wednesday, 4 September 2019 15:41 PM UTC
  2. PowerBuilder
  3. # 3

From PB11 Application Techniques:

 

PowerBuilder

CHAPTER 3

Selected PowerScript Topics

Controlling when garbage collection occurs

Garbage collection occurs automatically in PowerBuilder, but you can use functions to force immediate garbage collection or to change the interval between reference count checks. Three functions let you control when garbage collection occurs: GarbageCollect, GarbageCollectGetTimeLimit, and GarbageCollectSetTimeLimit. For information about these functions, see the PowerScript Reference. For an example illustrating their use, see the Code Examples sample application, described in Chapter 1, “Using Sample Applications.”

Performance concerns

You can use tracing and profiling to examine the effect of changing the garbage collection interval on performance. For information about tracing and profiling, see the PowerBuilder User’s Guide.

Configuring memory management
You can set the PB_POOL_THRESHOLD environment variable to specify the threshold at which the PowerBuilder memory manager switches to a different memory allocation strategy. When most windows, DataWindows, DataStores, or other PowerBuilder objects are destroyed or reclaimed by the garbage collector, the PowerBuilder heap manager returns the memory allocated for each object to a global memory pool and records its availability on a global free list. The freed memory is not returned to the operating system. When a new object is created, PowerBuilder allocates blocks of memory from the global memory pool (if sufficient memory is available in the global free list) or from the operating system (if it is not) to a memory pool for the object. When the memory required by an object exceeds 256KB, PowerBuilder uses a different strategy. It allocates subsequent memory requirements from the operating system in large blocks, and returns the physical memory to the operating system when the object is destroyed. It retains the virtual memory to reduce fragmentation of the virtual address space. For most applications and components, the threshold of 256KB at which PowerBuilder switches to the “large blocks” strategy works well and reduces the memory required by an application when it is working at its peak level of activity. However, if you want to keep the overall physical memory usage of your application as low as possible, you can try setting a lower threshold.

Application Techniques

Comment
  1. Miguel Leeuwe
  2. Wednesday, 4 September 2019 19:57 PM UTC
Sorry Chris, I don't see any mention of that. EAServer?

Just google on PB_POOL_THRESHOLD and or PB_HEAP_LOGFILENAME

https://www.appeon.com/support/documents/appeon_online_help/pb2017r3/application_techniques/ch03s10.html

Not a word about EA Server anywhere as far as I see.
  1. Helpful
  1. Chris Pollach @Appeon
  2. Thursday, 5 September 2019 13:42 PM UTC
Yes unless you were using PB with EAS from way back when, those settings would not be an obvious connection. I think that the "gist" of the EAS statements have been removed from the current documentation. So the EAS association is no longer there, but the variables now seem to relate to PB instead of EAS. :-(

FYI: http://www.techno-kitten.com/Changes_to_PowerBuilder/New_In_PowerBuilder_9/New_In_PowerBuilder_9_0_2/PB902New_-_Environment_Variabl/pb902new_-_environment_variabl.html

  1. Helpful
  1. Miguel Leeuwe
  2. Thursday, 5 September 2019 15:42 PM UTC
Too bad there's not much information of what values have to / can be set :)

Anyway my guess is to leave it as is and let PB handle the memory. For big files, we'll just have to chunk'em up in big pieces.
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Wednesday, 4 September 2019 15:35 PM UTC
  2. PowerBuilder
  3. # 4

If I remember well, the problem with reading big files in one go, is that - even though you might have enough free memory in Windows - Powerbuilder doesn't access that memory directly.

Powerbuilder allocates chunks of memory in a (for me) totally non-transparent way.

Whenever PB finds it necessary, it'll increment that allocated memory or not.

Not sure if I can find that article still somewhere.

Comment
There are no comments made yet.
Michael Kramer Accepted Answer Pending Moderation
  1. Tuesday, 3 September 2019 21:18 PM UTC
  2. PowerBuilder
  3. # 5

Hi Dave, 

What is actually failing when your FileReadEx of 300+ MB fails?
Do you hit 2 GB virtual mem per process limit? Or is some other O/S resource exhausted?

I would expect some O/S process constraint could be reason for huge FileReadEx to fail. EX: There may be a temp file reader buffer that increases your resource consumption way beyond expectation.

Whenever I have to process large data sets (like reading huge file) I try to "stream" the data to stay within resource constraints. So I would always favor large file transfers to happen in chunks.

You may opt to chunk at 64 kB or at 50 MB. Your choice.

HTH /Michael

Comment
There are no comments made yet.
Chris Pollach @Appeon Accepted Answer Pending Moderation
  1. Tuesday, 3 September 2019 19:22 PM UTC
  2. PowerBuilder
  3. # 6

Hi David;

   Have you tried reading a large file when compiled into a 64bit App?

Regards ... Chris

Comment
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Tuesday, 3 September 2019 19:14 PM UTC
  2. PowerBuilder
  3. # 7

I have an example of using Windows API functions to read a file into a blob. See the of_readfile function. It also has of_writefile as well.

http://www.topwizprogramming.com/freecode_bcrypt.html

It should be able to read a file up to 4,294,967,295 bytes. That is 4GB, the max value for a ULong.

If that is too small, read the file in a loop and concatenate. There isn't an API function that can read more than that in one shot.

 

Comment
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.