Anthony -

What everyone is saying is that the processing you have now needs to be recoded in SOME way. Retrieving 8 millioins rows of data in a client-server app is a *BAD* thing.

Options include:
1) Work the data in smaller batches.
2) Use a stored procedure to do the work you need done, THEN retrieve the final data.
3) Have the data processed in a Java application that you call from your PowerBuilder application. Then your app can examine the complete, processed data that is in the database. (BTW, this is how we handle a large data issue.)

Good Luck,

Olan

Hi Anthony;

  Sounds like you might be trying to perform some analysis on "Big Data". If that is the case, no client based application development tool will probably suffice unless you have a very large client PC machine with Gig's & Gig's worth of RAM on a super fast network. For smaller machines like you and your client's probably have on a lower speed network, I would only think about sending "summary" datum to ensure both network and PC optimal performance.

  My suggestion though would be to maybe re-think your approach in light of using Stored Procedures. Where the SP's would "cache" large result sets on the DB Server or pipe then into "temporary tables. From there, the SP's could produce "summary" data for your PB App(s) and/or your PB App could filter the datum from the DB temporary tables using very explicit "Where" clauses for a very "specific" result set.

Food for thought.

Regards ... Chris

Hi David

Ofcourse this a memory resource problem. I am trying to minimize data retrieval by Apply DB filter as you mentioned. 

Regards

Antony

Hi

Clearly this is a memory resource problem, but you do have to ask why? Why do you not use the DB to filter the rows before retrieval rather than apply a filter to 8M rows afterwards.

What are you doing with 8M rows of data, how are they being processed. I cannot believe a user is reviewing this number of rows so whatever you are doing could and should be done at the database layer for performance.

I think you need to question your application design when you start hitting these sorts of problems.

Could you now use a cursor to extract and process the data?

I hope that you will take these thought in the helpful manner they are intended, and consider your approach.

Kind regards

David

Hi Xaview,

8M is a *lot* of rows, as reminded by Miguel, this situation was already discussed in

https://community.appeon.com/index.php/qna/q-a/copying-datawindows/oldest#filter-sort

While retrieving those rows, what do you see in the Task Manager?

Filtering rows on top of Retrieve will add additional memory consumption, in fact this operation brings up the "DW Filter buffer".

Best,

.m

Hi Miguel 

Thank you very much for the advice, workaround and docs, It is working when I compile Application as 64 bit. But when I Try to apply some filter is not working. I have to try with web and IWARunner, any suggestion!!!.

Thanks in advance.

Regards

Antony

Ha, I just stumbled upon this Q&A where is spoken of the same memory limitations:

https://community.appeon.com/index.php/qna/q-a/copying-datawindows/oldest#filter-sort

Hi,

As far as I know, the only limit is the amount of free RAM memory that you have. Now, there is of course the little problem of powerbuilder 32 bit. I think it won't be able to use more then 2 GB and if you apply some "trick" maybe 3 or 4 GB.

So if you're getting close to 2GB when retrieving your datawindow with 8 million records, than RAM is your problem.

You can try to compile your application as 64bit (also use a 64 bit database client in that case and beware of MAPI and ActiveX that might stop working anywhere in your application).

Another option is this "trick" and be able to use 3 GB instead of 2GB:

https://www.brucearmstrong.org/2013/03/nice-tool-for-increasing-memory-for-32.html

Here's another interesting read: http://pbdj.sys-con.com/node/2187602 

I've printed the pages and attached as pdf.