1. Roland Smith
  2. PowerBuilder
  3. Monday, 8 May 2023 14:23 PM UTC

PB 2022

I'm allocating a blob to use for the output from a Windows API function:

lblb_Buffer = Blob(Space(lul_BufferSize), EncodingAnsi!)

Buffer size is 262,164,339.

The result is:

---------------------------
PowerBuilder application execution error (R0108)
---------------------------
Application terminated.

Error: Not enough memory to execute the operation at line 28 in function compress of object n_compression.
---------------------------
OK
---------------------------

 

This is a demo app just for the Windows API functions so nothing else is going on in the app.

Customer stated that the built-in CompressorObject/ExtractorObject are too slow so I put together an example of the Windows functions Compress/Decompress. They are fast but the above problem limits use.

Is there some other way to allocate a blob? This is a little under 256 MB which isn't that large when dealing with files.

Mark Goldsmith Accepted Answer Pending Moderation
  1. Wednesday, 10 May 2023 22:21 PM UTC
  2. PowerBuilder
  3. # 1

Hi Roland...I've been watching for an update on your ticket and Ken's latest reply yesterday still takes the position that the 2GB memory limit is the issue but looking at your latest update it feels like that may not be the case.


FWIW...I compiled a test app, 32 bit and then 64 bit, to see what my max would be on string length and number of strings that size. On the 32 bit app it was about 120MB and on the 64 bit app it was just over 1.053GB (only 20MB off PB's max string size). If I changed the 32 bit executable by turning on /LARGEADDRESSAWARE I was able to have a string as large as 1.053GB. I declared a variable string array and kept increasing the number of elements, filling each element of the array with 1GB of characters. Here's the code (not including variable declarations):

li_array_length = Integer(sle_array_length.Text)  //User-specified via sle
ll_string_size = Long(sle_string_size.Text)	//User-specified via sle
For li_counter = 1 to li_array_length
	mle_2.Text = String(li_counter)
	Yield()
	ls_spaces[li_counter] = Fill ('-', ll_string_size)  //approx. max string size for 64 bit exe in PB is 1,053,140,000 bytes
Next
MessageBox("", "Done.")
SetNull(ls_spaces)

In the 64 bit app, I tested as high as 25 elements in the array, each element 1GB in size and it handled it just fine. I didn't receive an error after 25 elements, I just decided not to test further. I have 32GB of ram on this machine and up around the 21st element it started paging to disk but other than that no issues. On the 32 bit side of things, it just seems to reach its maximum, then the memory error but never pages to disk.


You may be right, maybe the 32 bit limitation is not fixable but given what you were able to do, and that it appears the max string size for a 32 bit app can only hold a little over 10% of the stated maximum for a string under my configuration and the application used for testing, maybe there is something Appeon can do to improve this number. It may not be worth the effort though given there are workarounds. I was curious to see if the 32 bit app behaved differently on a 32 bit OS (versus SysWOW64), more so on the use of the pagefile, but ran out of time...maybe tomorrow.


All that being said, unless one does what you did to make it work or make the app large address aware, clearly a 64 bit app is the way to go if possible.

Regards...Mark

Comment
There are no comments made yet.
Benjamin Gaesslein Accepted Answer Pending Moderation
  1. Wednesday, 10 May 2023 11:26 AM UTC
  2. PowerBuilder
  3. # 2

I did some experimenting on this and it seems a basic 32-Bit PB app can allocate about 1,376,780,254 bytes of RAM. But not all at once. The only way I could come even close to that number was in incremental steps by allocating increasingly smaller blobs. The basic code block to create a single block looks like this. I used a new blob variable for each iteration.

ls_spaces = space( ll_spaces )
lblb_test = Blob(ls_spaces, EncodingUTF8!)
ls_spaces = ''

I used UTF8 because in case of spaces this will result in a blob exactly the size of ll_spaces in bytes. ANSI would probably work the same. I started out with one below the supposed maximum supported by Space (2147483646) but that would always error out with "Not enough memory to execute..."

Then I halved the amount with each step.

1073741823: "Not enough memory to execute..."

536870911: "Not enough memory to execute..."

268435455: NO error message so the string was created but it would just silently crash at the line creating the blob from the string.

134217727: This finally worked. But only three times, then I had to go down by half again. In the end it took the following to approach the apparent memory limit:

3x134217727
9x67108863
8x33554431
1x16777215
13x8388607
1x4194303
3x1048575

=~1.3GB RAM. After each allocation crashed I went down by half. No idea why at one point the 16MB allocation failed while 13 8MB blobs could still be created without issue. I mean there's obviously overhead from creating the string variable but it shouldn't be that much? I dunno.

 

I also tried compiling to 64bit and 64Bit PB Apps handle memory allocation a lot better but it STILL won't allow the supposed maximum for the Space function. The actual maximum seems to be 1053140931, because at 1053140932 it starts to give you the "Not enough memory to execute..." RTE.

In 64bit, the actually available RAM seems to be the limit, I could create 16 ~1GB blobs with no issues. Doing any concatenation etc. with strings this large still just silently crashes the app, though.

Comment
  1. Mark Goldsmith
  2. Wednesday, 10 May 2023 15:28 PM UTC
Hi Benjamin...I too did some experimenting and will be posting my reply shortly. I was wondering what version of PB you used for your testing and whether you created EXEs for both the 32 bit and 64 bit testing.

My results for the most part are similar to yours but do differ a bit, primarily on the 64 bit side of things, where I didn't receive a memory error at all and only crashed silently if I went beyond 1,053,140,000 bytes (I didn't test out as far as you did to the last byte) when executing string assignments. If I did go beyond that size then I would get the silent crash. I only experienced the "Not enough memory..." error if the app runs out of memory, which it never did under a 64 bit EXE (I'll elaborate in my reply).

As for "No idea why at one point the 16MB allocation...", not sure whether it would make a difference if you used SetNull(ls_spaces) and GarbageCollect(). It wouldn't surprise me if it didn't make a difference as it just feels like in the 32 bit environment memory management could be much improved.

Regards...Mark
  1. Helpful
  1. Benjamin Gaesslein
  2. Thursday, 11 May 2023 07:15 AM UTC
This was PB2019R3. Maybe later releases have improved in this a little.

Yes, I deployed both 32bit/64bit exes, running from the IDE crashes much earlier. It also makes a difference if the IDE is still running when starting a deployed app but not as much as running from IDE directly.
  1. Helpful
There are no comments made yet.
Andreas Mykonios Accepted Answer Pending Moderation
  1. Tuesday, 9 May 2023 06:45 AM UTC
  2. PowerBuilder
  3. # 3

Hi Chris.

In Roland's case string isn't longer than 256 MB. So it should work. The only issue I can think of is that space is unicode and will produce a UTF16-LE string. I don't know what will be the actual size of produced string in bytes, but I guess it will be lower that 2 GB. I'm right at my consumption?

Andreas.

Comment
  1. Roland Smith
  2. Tuesday, 9 May 2023 14:36 PM UTC
I moved the code into a PBX used by the app and it works fine in 32-bit. I just did a simple C malloc function call. The 2GB limit on 32-bit apps isn't the problem. I tried using a Byte array and it just disappears on:

lby_buffer[lul_BufferSize] = 0

There is a low level problem with very large variable declarations in general that probably isn't fixable.
  1. Helpful 1
  1. Miguel Leeuwe
  2. Wednesday, 10 May 2023 03:33 AM UTC
PB uses its own memory heap. There's not really a way of controlling it.

Please see also: https://www.appeon.com/standardsupport/search/view?id=201

  1. Helpful
  1. Miguel Leeuwe
  2. Wednesday, 10 May 2023 03:34 AM UTC
  1. Helpful
There are no comments made yet.
Andreas Mykonios Accepted Answer Pending Moderation
  1. Monday, 8 May 2023 14:33 PM UTC
  2. PowerBuilder
  3. # 4

Hi Roland.

From what I see fill has the same problem... I guess you should open a support case. Normally this should not happen. In help it's clearly stated that n can be up to  2,147,483,647.

I've tested in PB 2021...

Andreas.

Comment
  1. Benjamin Gaesslein
  2. Tuesday, 9 May 2023 06:56 AM UTC
PS: scratch that, now PB errors out at the Space function. Weird, that didn't happen the few times before this test run.
  1. Helpful
  1. Benjamin Gaesslein
  2. Tuesday, 9 May 2023 07:00 AM UTC
Apparently it depends on the amount of memory the app/ide is currently using. Space( 262164339 ) works if you're under a certain limit but trying to create a blob from this then fails because it trips a threshold. In that case there isn't even an error message, it just silently crashes.
  1. Helpful 1
  1. Andreas Mykonios
  2. Tuesday, 9 May 2023 08:38 AM UTC
Yes. This might be because another variable (memory allocation) takes places. I guess that running directly from an exe may have different result.

Andreas.
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.