Roland, we found a way to read in large chunks of data. We selected 20,000 rows at a time to read. Here the source is dw_source, but the same principle should work with a FileReadEx (1, 20000)

STRING      ls_NULL[],  ls_SOURCE_DATA[], ls_status

    // Init the data for the next group of rows
    ls_SOURCE_DATA  = ls_NULL

    // Expand the target DW_INPUT's storage capacity
    ls_status = dw_input.Modify ("datawindow.storagepagesize='LARGE'")

   LOOP
       // Grab the next group of rows into the buffers
       ls_SOURCE_DATA = dw_source.Object.SOURCE_DATA  [ll_row_start, ll_row_end]

      << process data in the string array>>

       // Save the analyzed and possibly updated data into DW_INPUT
       dw_input.Object.SOURCE_DATA  [ll_target_start,ll_target_end]  = ls_SOURCE_DATA
  END LOOP


This works very well, but with REALLY big files we apparently run out of space, and the entire app just vanishes.


Good Luck,

Olan

Hi Roland,

If any good: All I know is that PB has it's own "intermediate" layer of memory allocation. The way it's being assigned is non-transparent and arbitrary. Even if you have 1GB of available RAM, it still might blow up on reading only 400MB.

FileReadEx() ( you say "readFileEx()"? I guess that's the Windows API) has a parameter that allows you to set how much data you want to read. So maybe a solution would be to read your file in a loop of 100MB chunks). It still doesn't guarantee PB would allocate enough memory for the blob you assign the data to, but it's worth the try.

regards.