1. Steen Jakobsen
  2. PowerBuilder
  3. Monday, 27 November 2017 15:16 PM UTC

Hi all,

 

The code below is eating up memory. (This example is eating 50 MB in 3 minutes)

GarbageGollect() does not work

 

The only 2 solutions are:

1. destroy and create datastore

2. MyDataStore.DataObject = "some dw"     and then  myDataStore.SetTransObject(MyTransaction)

But this is just not possible in our application.

What can I do ? (I'm in deep trouble)

Thanks in advance 

// Steen

-----------------------------

DataStore lds
lds = Create DataStore
lds.DataObject = "d_abc"
lds.SetTransobject(SQLCA)
 
int li_x
long ll_f
For li_x = 1 To 500
    lds.Retrieve('some id')
    For ll_f = 1 To lds.RowCount()  // 700 rows
         lds.SetItem(ll_f,"mycolname",'some data') // memory leak here!!
    Next
Next
MessageBox("","before destroy")
Destroy lds
 
// memory released
 
----------------------------
Armeen Mazda @Appeon Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 16:48 PM UTC
  2. PowerBuilder
  3. # 1

Guys,

There are reasonable workarounds and arguably better ways of writing the code.  PowerBuilder product is composed of many millions of lines of code that we didn't write and have to be very careful how we change so our customer's mission critical apps don't break when they upgrade, many of which are Fortune 500 companies. 

Plus we need to balance our R&D resources between fixing Sybase's old bugs that customers have tolerated (i.e. worked around) over the years with bringing long overdue features to market.  Hope you guys take a moment to look at this from our side as well that we are trying to do the best we can for PowerBuilder customers and their projects.

But even with that said we aren't passing the buck and have fixed or are currently releasing fixes for a number of such legacy bugs that Sybase ignored, such as bug ID # 150, 173, 186, 190, 199, 219, 234, 237, 256, 309, 314, 316, 341, 354, 381, 415, 437, 509, 535, 589, 717, 819, 896, 990, 1037, 1153, 1233, 1345, 1360, 1392, 1693, 1718, 1729, 1745, 1757, 1783, 1814, 1825, 1856, 1982, 1999, 2278, 2443, 2467, 2576, 2743, 2753, 2834, 3077, 3504, 3513.

Thanks,

Armeen Mazda
CEO, Appeon

Comment
  1. Miguel Leeuwe
  2. Thursday, 7 November 2019 18:11 PM UTC
Yes Armeen, you are very right on this. I might seem negative sometimes, but really do appreciate the effort Appeon is doing and the customer service is "unseen". Maybe it's because we have been suffering so much time with Sybase and SAP and now that there's a positive direction, we have become a bit demanding.

regards
  1. Helpful
  1. Miguel Leeuwe
  2. Thursday, 7 November 2019 18:13 PM UTC
Also, I now realize that this bug also happens in 12.6. Many more like this will show up, as now users want things to be fixed.

Before Appeon, we'd have to re-think twice before reporting a bug, as many times it would not "go anywhere".
  1. Helpful
  1. Armeen Mazda @Appeon
  2. Thursday, 7 November 2019 19:44 PM UTC
Thanks for your understanding, and we do sympathize with what customers went through during the Sybase days. We are doing our best to wash the bad taste out of people's mouth and rebuild goodwill with the PB community. For what its worth, if we introduce a regression bug (i.e. didn't exist in PB 12.6) it will be rare that we don't fix it because that code is usually would be written by us, and that's why in the support portal we have a specific bug classification called "Sybase (legacy) bug".

But even with that said we aren't passing the buck and have fixed or are currently releasing fixes for a number of such legacy bugs that Sybase ignored, such as bug ID # 150, 173, 186, 190, 199, 219, 234, 237, 256, 309, 314, 316, 341, 354, 381, 415, 437, 509, 535, 589, 717, 819, 896, 990, 1037, 1153, 1233, 1345, 1360, 1392, 1693, 1718, 1729, 1745, 1757, 1783, 1814, 1825, 1856, 1982, 1999, 2278, 2443, 2467, 2576, 2743, 2753, 2834, 3077, 3504, 3513.

The exception to this generalization would obviously be any third-party stuff that is in PowerBuilder, such as the RichTextEdit control (made by Text Control). For third-party stuff we rely on the third-party to fix. But Text Control is very good company, and in fact they are rolling out a 64-bit version with many new features that we hope will make it into PB 2019 R3.
  1. Helpful
There are no comments made yet.
John Fauss Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 15:22 PM UTC
  2. PowerBuilder
  3. # 2

Has anyone tried replacing the hundreds of SetItem calls performed inside of the inner loop with a single statement outside of the inner loop that sets the data values in a column for all rows at once?

Integer li_x
Long ll_rowcount, ll_f
String ls_columndata[], ls_emptyarray[]
DataStore lds

lds = Create DataStore
lds.DataObject = "d_abc"
lds.SetTransObject(SQLCA)

For li_x = 1 To 500
   // Flush, then populate the DataStore.
   lds.Reset()
   ll_rowcount = lds.Retrieve('some_id')
   if ll_rowcount > 0 then
      // Clear column data array, then load it with new column data values.
      ls_columndata = ls_emptyarray
      
      
For ll_f = 1 To ll_rowcount
         ls_columndata[ll_f] = 'some data'
      Next
     
      // Replace column data in the DataStore for all rows at once.

      lds.Object.mycolname.Primary.Current = ls_columndata
      // or
      //lds.Object.mycolname[1,ll_rowcount] = ls_columndata
   end if
Next

MessageBox("","Before destroy")
Destroy lds

Disclaimer: I have NOT tested the above code, and I do not claim it will solve the memory utilization issue. However, syntactically it looks like it should work correctly. The technique is documented in the DataWindow Reference, section 4.3.1 "Syntax for one or all data items in a named column".

HTH,
John

Comment
  1. Miguel Leeuwe
  2. Friday, 8 November 2019 00:42 AM UTC
Nice one, Yes this should work. I do have "mixed" experience with DATA though. Especially when getting full DATA and storing it in an any variable and - when user decides to not save - setting the primary buffer back using this stored any variable. I would sometimes get "chinese characters" in string fields. I should have some trivial explanation (somewhere) of SAP somewhere explaining why this sometimes happens.
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 08:11 AM UTC
  2. PowerBuilder
  3. # 3

Hi Steen,

I think that 3 possible workarounds were suggested when you reported bug 535.

One of them would be to do "

lds.resetupdate() //to release memory

If your MyDatastore would be a standard user object datastore, like n_ds of the pfc's, maybe a possible solution would be to do a ResetUpdate() in the RetrieveStart() event of the ancestor class of MyDataStore.

Just an idea, don't know if it would really work, this workaround.

regards,

Comment
  1. Miguel Leeuwe
  2. Thursday, 7 November 2019 08:57 AM UTC
I've tested my proposed solution and it works!

I've added these 2 lines of code to the retrievestart event of n_ds:



this.ResetUpdate()

return AncestorReturnValue



What is weird is that it does NOT work when doing the same thing in the retrieveEND event.

So my next move is to make sure all of my datatores are created as n_ds instead of datastore.

regards
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 02:42 AM UTC
  2. PowerBuilder
  3. # 4

This is the won't-fix bug I'm referring to, as Steen J. reported this before as a bug:

https://www.appeon.com/standardsupport/search/view?id=535

 

Comment
  1. Miguel Leeuwe
  2. Thursday, 7 November 2019 08:04 AM UTC
Ah, now I see that Yakov already mentioned this bug too..
  1. Helpful
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 02:12 AM UTC
  2. PowerBuilder
  3. # 5

The Retrieve function returns the number of rows so you should get it into a variable and use that in the for-next.

The RowCount function is being executed every time through the loop.

 

Comment
  1. Miguel Leeuwe
  2. Thursday, 7 November 2019 02:24 AM UTC
Hi Roland,

You're right, but this code is just meant as a how to reproduce the bug, is my guess. The problem of the memory leak won't be fixed by replacing the rowcount().
  1. Helpful
There are no comments made yet.
Miguel Leeuwe Accepted Answer Pending Moderation
  1. Thursday, 7 November 2019 02:00 AM UTC
  2. PowerBuilder
  3. # 6

Well, Appeon's answer on this reported bug has been clear: they won't fix it as they don't know how much other stuff they might break in doing so.

"Wow", is all I can say !

Comment
There are no comments made yet.
yakov werde Accepted Answer Pending Moderation
  1. Wednesday, 6 November 2019 16:11 PM UTC
  2. PowerBuilder
  3. # 7

Hello

Seems like we are encountering the same issue in multiple places when processing large data sets.   This is a likely cause for multiple reported failures.  Since "loop retrieve, setitem, update" is a common processing pattern code refactoring is not practical.  I saw the case https://www.appeon.com/standardsupport/search/view?id=535

Was a fix implemented in R3 or 2019?

 

Thanks

yakov.werde@iongroup.com

Comment
  1. Roland Smith
  2. Wednesday, 6 November 2019 23:57 PM UTC
Does 64bit help?
  1. Helpful
  1. yakov werde
  2. Thursday, 14 November 2019 20:40 PM UTC
Hi Roland, Thanks for the suggestion.

We ran the production code against a large customer dataset in 2017 R2 64 bit. Same abrupt termination results. Seems like the flaw is not address memory related.

We migrated to Appeon 2019 and ran the production code against a large customer dataset . Same abrupt termination results.

We are estimate at least about 2 months of work to refactor all the suspect code routine, with the expectation (but not an absolute guarantee) that using a different algorithm will avoid the ABEND.

Sure wish engineering could address the runtime issue!



Yakov
  1. Helpful
There are no comments made yet.
Chris Pollach @Appeon Accepted Answer Pending Moderation
  1. Monday, 27 November 2017 19:36 PM UTC
  2. PowerBuilder
  3. # 8

Hi Steen;

    Can you try the following ....

  1. Create a PB.INI file in your App's work folder
  2. Add the following to the INI ...

[DataStore Behavior]
UseHwnd=no

Let me know if that helps!

Regards ... Chris

 

Comment
  1. Chris Pollach @Appeon
  2. Tuesday, 28 November 2017 13:53 PM UTC
Hi Dan;



      Basically, this stops the PBVM from allocating very expensive control blocks and MS-Window handles for DataStores. The PBVM treats the DS like a DW Control and thinks that you might make it visible at some point - so it allocates graphic resources for the DS. The above setting tells the PBVM to ignore creating these resources as the DS will never be visible. So not only do you save MS-Window resources, it also makes your PB Apps run much faster where DS use is concerned.



    BTW: This fix goes way back to the PowerSoft days of PB 7 / 8.



Regards ... Chris



 

  1. Helpful
  1. Michael Kramer
  2. Tuesday, 28 November 2017 16:23 PM UTC
Hi Chris,



Can the DataStore object still print if it has no Windows handles? /Michael

  1. Helpful
  1. Chris Pollach @Appeon
  2. Tuesday, 28 November 2017 20:37 PM UTC
Hi Michael;



  A great question and ... Yes, it can!  



Regards ... Chris

  1. Helpful
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Monday, 27 November 2017 17:20 PM UTC
  2. PowerBuilder
  3. # 9

Try getting the rowcount into a long first instead of having it part of the For statement.

Comment
There are no comments made yet.
Govinda Lopez @Appeon Accepted Answer Pending Moderation
  1. Monday, 27 November 2017 16:42 PM UTC
  2. PowerBuilder
  3. # 10

Hi Steen,


Based on your example code, I infer that, you are using a nested "FOR" because you are doing some more processing in it. But, if this wasn't the case, I would suggest to fill that column in the query during the RETRIEVE. (ie: SELECT a, b, mycolname = 'some data' FROM mytable WHERE a BETWEEN 1 AND 500). That way you may be able to avoid the use of a FOR cycle of 350,000 iterations.

On the other hand, if you find that a particular part of the function is causing this, then I would recommend you open a support ticket for this case. It may be something with the internal processing of the info.

I hope this helps and please do share your results here.


Regards,

Comment
  1. Steen Jakobsen
  2. Monday, 27 November 2017 17:09 PM UTC
Hi Govinda,



Thanks for your quick reply :-)



 



The sample code is a simple way to repro the issue.



The application has 5000 objects and cannot be easily rewritten to workaround the bug.



using the sample code the issue is 100% visible. To my mind a huge memory management bug.



I will open a ticket. ;-)



 



Thanks - Steen



 

  1. Helpful
  1. Govinda Lopez @Appeon
  2. Monday, 27 November 2017 18:28 PM UTC
Thank you Steen,



 



I will review your ticket. Meanwhile, have you used this on other PowerBuilder versions as well? If so, did this happen there too?



 



 



Regards,

  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.
We use cookies which are necessary for the proper functioning of our websites. We also use cookies to analyze our traffic, improve your experience and provide social media features. If you continue to use this site, you consent to our use of cookies.