Subscribe via RSS
Toggle Submenu
  1. Categories
  2. Recent
  3. Tags
View Replies (4)

Resolved ImportFile : large datablock locks up memory

  1. Advice
0
Votes
Undo
  1. John Brown
    John Brown
  2. PowerBuilder
  3. Monday, 14 June 2021 13:42 PM UTC

 

All - I've searched and not found an answer to this solution.  Apology if there's a thread on this  ( feel free to point me that direction ).

Working on a new application ... User has text based files that we load for editing.

 

We were using ImportFile() for faster processing ( as opposed to line by line ), but some of the files are large ( 12k rows or more - I think roughly 1.8mb ).  It freezes loading ... but eventually the application will come back.

But closing the window ... it locks up the application and never comes back.  In fact, I have to kill the IDE.

I've tried playing with some of the concepts of Garbage collection, etc., but nothing has worked at this point.

 

I'm not sure what I'm doing wrong. LOL

 

Any help greatly appreciated it.

- Thanks, John

Select varchar(max) field on SQLServer return the ...
DNS-less ODBC Connection for PostgreSQL
Responses (4)
Olan Knight
Olan Knight Accepted Answer Pending Moderation
  1. Monday, 14 June 2021 14:17 PM UTC
  2. PowerBuilder
  3. # 1

According to the HELP, the ImportFile() datawindow command does have a file size limit. They tell you this because of the error code 15:
                      -15 -- File size exceeds limit

Sadly, I could not find where the size limit was specified in the HELP, but my fuzzy memory vaguely remembers it being something like [ 32,768 X 255 ] bytes.

Options include:
1. Multiple import files
    - Limit the import to X number of rows, save, then import into file #2, and so on.
      When complete, combine the various import files

2. Import into a blob

3. Import into a temp table in the database
    - Like option #1, import chunks of data at a time, then combine the data into a single file.

4. Row by Row manual import
    - Slow by comparison, but if it works....  :/


Comment
There are no comments made yet.
Mark Goldsmith
Mark Goldsmith Accepted Answer Pending Moderation
  1. Monday, 14 June 2021 16:27 PM UTC
  2. PowerBuilder
  3. # 2

Hi John,

In addition to Olan's suggestions, you could also try:

1) DW.ImportString() using a FileReadEx() call first; I'm not sure that ImportString() has a size limit as there is no stated error message pertaining to size but you may still run into the issue with FileReadEx() which will then bring you back to splitting the file or maybe by reading it into a Blob could give you a higher ceiling.

2) Use an ODBC Text driver connection

HTH...regards,

Mark

Comment
There are no comments made yet.
Matt Balent
Matt Balent Accepted Answer Pending Moderation
  1. Monday, 14 June 2021 16:33 PM UTC
  2. PowerBuilder
  3. # 3

You could try to read the file into a string via FileOpen and FileReadEx and then using ImportString into your datawindow.  Not sure it would make any difference.

Comment
There are no comments made yet.
Olan Knight
Olan Knight Accepted Answer Pending Moderation
  1. Monday, 14 June 2021 22:12 PM UTC
  2. PowerBuilder
  3. # 4

I saw that Chris had replied to a similar issue; here is his post:

   The PB IDE install defaults to a 32K maximum data stream for BLOB and Long Textual data. For the native SS connection or connecting via ODBC you need to set the following in your PB App's DBParm connection settings in the Transaction Object, as follows ...

  • SQLCA.DBParm = "PBMaxBlobSize=nnnnnnn"  // Binary
  • SQLCA.DBParm = "PBMaxTextSize=nnnnnn"    // Text

Note1: Both settings can co-exit in the DBParm field of your transaction object.

Note2: You can effect the same in an ODBC connection by modifying the PBODB190.ini file and adding the above settings to that control file.

   Please check the above keywords in the PB Help file for more detailed information.

HTH

Regards ... Chris

Comment
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.