1. Constantin Bergatt
  2. PowerBuilder
  3. Thursday, 17 October 2019

Hello everyone,

we are currently in the process of evaluating, if patching is a viable way of updating our product (PowerBuilder application).

Recently I found a way to create and apply patches using the "diffUtils" and "patch" tool from the "GnuWin32" toolbox.

But during research and testing I found that even smallest changes within the PowerBuilder source code will lead to differences for all PBD files of our application, which does increase the size of the resulting patch file a lot.

E.g. lately we had kind of an emergency hotfix for one of our customers, which were only fixing an error within a single global function in one of our PBL files. But as a consequence when comparing the resulting PBD files with the ones from the predecessor version the patch/diff file did amount to about 400 MB (!!!).

So here are my questions :

  1. Does anyone have some experience in patching PowerBuilder applications?
    1. If yes, which tools or workflows did you use?
    2. Did you try other tools? And if yes, why did you choose the other approach/tool?
  2. Why the big difference in PBD files although the made changes were comparatively minor?
    1. Is it related to the PBD file format?
    2. Does with each new build the order of things within all used PBD files get changed a lot? If yes, why?
    3. Is it possible to somehow configure PowerBuilder or its compiler to keep the order of things in general and only make as few changes as possible?

Many thanks in advance for all your efforts!

Best Regards,

Constantin Bergatt

Roland Smith Accepted Answer Pending Moderation
0
Votes
Undo

The PBL file contains source code and compiled code versions of each object. The two versions are stored separately in the library.

The PBD file and the PBL file have an identical internal format. When a PBD file is generated, it create an empty PBD file and it copies the existing compiled code from the PBL into the PBD. The rebuild step is what recompiles objects.

If even one object has been changed since the last time a PBD was generated, the order of objects in the files will be different. Because of the 'library' format, when an object is saved or regenerated, the space used by the old version is marked unused and then the new version is placed in the first location that is large enough for it to fit.

Another point is that objects are stored in blocks so there are block headers and all the objects will be stored in several blocks with no guarantee that the blocks are in order or even contiguous. There are different types of blocks, some are 512 bytes, some 3072 bytes and some are variable length.

 

Comment
There are no comments made yet.
  1. Friday, 18 October 2019
  2. PowerBuilder
  3. # 1
mike S Accepted Answer Pending Moderation
0
Votes
Undo

the patch.pbd method:

DO NOT do a full build.  You are ONLY building the patch. 

Make sure patch is at top of lib list.  you can do this either at runtime, or have an empty patch.pbd as part of your original build.  I don't like the latter method because it will require you to replace the pbd during deployment and that requires the user to have security rights to program directory to do that, or requires a process outside of the application to update it.

copy any changed object in the patch.pbl - then build just the patch.pbl.  use orcascript to do the build. 

I copy the patch.pbl into a build directory, then build from there:

you must use a PBL that holds the application object, not the pbd.  (i think that is a bug!)   

you don't have to use pbds for the rest of the app, but it makes sense to build it that way to ensure you didn't accidently update any pbls other than the patch.  

I wish there was a way in orcascript to regen all objects in just the specified pbl (the patch pbl).  You can write orcascript to regen a specified object, but not all objects in the pbl.

TEST TEST TEST TEST.   if it breaks it will break obviously with a run time error.

deployment methods of the patch is a whole other topic.  

 

SAMPLE ORCASCRIPT:

start session

file copy "patch.pbl" "build\patch.pbl"


set liblist "build\patch.pbl; \source\main.pbl; library1.pbd; library2.pbd;"
set application "C:\SOURCE\main.pbl" ""

build library "build\patch.pbl" "" pbd
end session

 


start session
file copy "build\patch.pbd" "..\patch.pbd"  -- copy the build pbd to a spot you can test it
end session

 

Comment
No rebuilds is the most important point. Build individual libraries only.

It would be nice if the Project Painter offered the option of 'No Rebuild'.
  1. Roland Smith
  2. Friday, 18 October 2019
There are no comments made yet.
  1. Friday, 18 October 2019
  2. PowerBuilder
  3. # 2
Constantin Bergatt Accepted Answer Pending Moderation
0
Votes
Undo

Hello everyone,

first of all many thanks for all your contributions!

I think you helped me a lot in getting a better understanding of the problem.

So for a very limited range of issues the patch.pbd method can be used. I did try to do some research on this topic, but did not find some concrete way how to do it.

  1. Can someone give me some more details on that?
    1. How is the workflow of this approach?
    2. Do you do add a patch.pbd, containing the required changes and files, to the used target, do a full build, deploy the application and add it to the faulty application using ORCA script?
    3. Will all the references within the other PBD files be correct when simply adding the patch.pbd on top?
    4. Or will ORCA script be used to kind of rebuild the target while "patching"?
    5. Is it possible to instead of importing/adding a patch.pbd to add an empty patch.pbd to the application and add objects based on their exported source code?
    6. How would an ORCA script look like for doing such?

Further, sorry as it is the most promising approach to us, I would like to elaborate more on my approach on doing binary patches :

  1. First of all, I don't want to only patch a single PBD file.
  2. My approach would be to get the binary diff of the "old" and the "new" version (to be honest my current idea is to get the binary diff of the whole folder, containing the PowerBuilder application, C# components, etc.) and use it for patching.
    1. So as I am taking all of the PowerBuilder application (PBD files, EXE file, etc.) into account when creating the difference, I would assume the references of the patched application should still be intact and valid.
    2. I mean, my idea is to just assume each "iteration" of the PowerBuilder application is an binary image and as our application is very big and complex, to only having to transfer changes within the binary images would decrease the amount of data required for an update by a lot. Wouldn't it?

Last but not the least, as someone mentioned RTPatch :

  1. Does someone have experience with this tool?
  2. Doing some research I found that this tool also provides kind of an whole update mechanism?
  3. Can anyone give me some more details on that?

Regards,

Constantin

 

Comment
Constantin,

Although I don't have current experience with RTPatch, our company used it for about a decade in the past.

Unless they've changed how it works, you create a patch file list, which can contain individual files, wildcards, or whole directories, between an original version & new version. You then run the patch builder, and get a resulting .rtp file. This file is distributed with an exe to apply the patch. I haven't looked at their docs lately, but I'm pretty sure it can also be used to register Active-X objects, and make other system changes if necessary. Versioning can be used as well, where a patch is built from other patches (to allow patching of more than one version). The .RTP files are compressed, and more than that, they do some pretty smart matching of source/destination files to keep the resulting patch data to a bare minimum. It's all very flexible, and is still used by quite a few companies.



HTH,

Brad
  1. Brad Mettee
  2. Friday, 18 October 2019
Hi Everyone;

FWIW: I think that this would make a great presentation topic at the next Elevate 2020 conference (just a thought)! ;-)

Regards ... Chris

  1. Chris Pollach
  2. Friday, 18 October 2019
There are no comments made yet.
  1. Friday, 18 October 2019
  2. PowerBuilder
  3. # 3
Chris Pollach Accepted Answer Pending Moderation
0
Votes
Undo

Hi Roland;

   This is the "safety rule " ... make sure that you rebuild every PBD for every Object that has been affected in the "hotfix". Of course, Mr Murphy can still get into the picture here with "human" dependencies. Basically ... patch at your own risk but, OK if your pragmatic about the process.

   Another technique I use is to always create a "patch" PBL as the first library in the App's library list. For example MyApp_Patches.PBL. When you deploy a complete version, the patch library is empty. However, when you replace it with an updated PBL with patched objects, they are loaded first because the library is before any current objects you are patching. Thus, all the ancestors and descendants in the patch PBL take precedence at Open/Create time. Just another "food for thought" approach.

Regards ... Chris

 

Comment
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 4
Michael Kramer Accepted Answer Pending Moderation
0
Votes
Undo

Hey, General reply to all the patch-PBD replies/comments >>

Whenever you call SetLibraryList it will ALWAYS prefix the resulting library list with the name of the executable itself. Example:

string desiredList, actualList

desiredList = "myCode.pbd," + GetLibraryList( )
SetLibraryList(desiredList)
actualList = GetLibraryList( )

When called several times, this is what happens to the actual library list:

  1. W:\Hat\app.exe,first.pbd,next.pbd    <== Initial library list before first SetLibraryList
  2. W:\Hat\app.exe,myCode.pbd,W:\Hat\app.exe,first.pbd,next.pbd
  3. W:\Hat\app.exe,myCode.pbd,W:\Hat\app.exe,myCode.pbd,W:\Hat\app.exe,first.pbd,next.pbd

I also tried this

int i

for i = 1 to 3
   SetLibraryList( GetLibraryList( ) )
next

MessageBox( GetApplication( ).DisplayName, GetLibraryList( ) )

My resulting library was:

W:\Hat\app.exe,W:\Hat\app.exe,W:\Hat\app.exe,W:\Hat\app.exe,first.pbd,next.pbd

I verified that this I current behavior in PB 2019.
See how .EXE keeps adding?

My learnings:

  1. Whatever classes and DW objects are in the .EXE file can NEVER be superseded by inserting new library to front of list.
    1. Because: System functionality ensures .EXE is always first

  2. You must strip off .EXE reference each time you modify the library list.
    1. If you don't then your library list  will continue to grow until app crash simply from multiple, identical references to the executable

  3. Wrap SetLibraryList call into a custom function of_AddFirstInLibraryList( string newLibrary ).
    1. This wrapper gets library list and strips off the .EXE reference which is always first in list.
    2. Then adds + "," to front of list.
    3. Calls SetLibraryList will knowing that .EXE ends up in front once more.

HTH /Michael

Comment
If you deploy it so that the library containing the application object generates a PBD, the EXE only contains the bootstrapper code and any images that are assigned to the EXE via a PBR file.
  1. Roland Smith
  2. Thursday, 17 October 2019
True, and that's how you need to do it. Otherwise you lose chance to patch.

App object itself instantiates before your PowerScript runs so that object class can't be replaced by fresh library list. Everything else can.
  1. Michael Kramer
  2. Thursday, 17 October 2019
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 5
Roland Smith Accepted Answer Pending Moderation
0
Votes
Undo

Where I work we have what we call a 'hotfix'. Basically we generate a PBD file for each library that we had to change an object in to fix the bug.

This is only used for urgent bug fixes that can't wait until the next full release which are usually 3 times a year.

We haven't had any problems using this method.

 

Comment
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 6
Chris Pollach Accepted Answer Pending Moderation
1
Votes
Undo

Hi Constantin;

  FWIW: Never, ever roll out fixes this way as you are taking your App's "stability" & throwing it out the window (so to speak). When you change any Ancestor object in PB it can totally change the descendant's P-Code. Especially, base address + offset to an internal resource. Which the PB run-time uses to locate the starting code address of executable code and / or variables. When ancestor changes are made to methods, events and instance variables, the descendants P-Code addressing will be significantly altered.

  So unless you are building a PBD that contains only images (or other external resources), PB object descendants at the "concrete" level where "guaranteed" no ancestors were altered, or the PBD only includes patched objects (like a "WIP" PBL in the Apps library list) - you can never trust the objects in that one regenerated PBD to not literally "take out" your entire App at run time.

  Been there, done that ... hurts too much just to think about how many times I "shot myself in the foot" so to speak trying that - LOL! Just my $0.02.

Regards ... Chris

Comment
Thank you Chris, my Gut told me not to do this years ago. What you have described is a systems administrator's worst nightmare. Memory offsets being out of alignment! ugh!



Really is the complete set of exe & pbds that big?
Yes, to be 100% safe - all PBD's or DLL's.
  1. Chris Pollach
  2. Wednesday, 23 October 2019
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 7
0
Votes
Undo

The problem I have with rolling out a single PBD or other patching method is that you cannot be sure what version someone is running.There is always a risk that the wrong PBD version is deployed.

One thing we have also noticed is that when you compile with PBDs then the complier is less rigorous about missing objects and invalid references. If you compile everything to a single large exe the compiler will error on missing references and objects. I personally prefer this method of compiling and deployment as it is only a single file, you either have the right version of not. The down side is obviously it can be quite a large file.

I'd be very interested to see where this discussion thread takes us...

Cheers

David

 

Comment
"always a risk that the wrong PBD version is deployed"



yes, this can be huge a problem, but there are ways to handle it such as providing a patch build date or number, and checking in the database what should be running.



I think it really depends on the application usage on whether you patch or not, and the approach to take if you do patch. if the application is used just by one company, then a patch probably doesn't make much sense.
  1. mike S
  2. Thursday, 17 October 2019
Correct me if I'm wrong, but the PBD files do not have a version number.



The only way I can see you knowing what version the PBD truly is, is to have a function within each PBD that returns the version number and check these against the DB. However, the problem with that is you need a unique function name for each PDB and it relies on human intervention to change the version number. Relying on developers, in my long experience, is bound to bite you at some point ;-)



Also having now read all the additional comments about patching, it does sound rather risky. I would in any case always ship the entire build for the sake of a few MB in distribution and save agro later.



Just my opinion.
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 8
Brad Mettee Accepted Answer Pending Moderation
0
Votes
Undo

In the past we used RTPatch (by PocketSoft). The patch files it built were fairly small (as we were still using modems at the time, this was critical). 

The reason PBD files differ so much is partly that the recompile of even one function can change the resulting code a lot, and/or cause a location shift. Compiling also embeds compile date/time for each object, which can create large numbers of differences, and in binary data, this can be significant in re-aligning the diffs as the file size increases. Patch progs designed specifically for binary data can make fairly small patch files vs a text differ.

With the speed of internet being relatively high, why not just create zip files, or a small updater like NSIS or MSI, to distribute patches?

 

 

Comment
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 9
Michael Kramer Accepted Answer Pending Moderation
0
Votes
Undo

Hi Constantin.

Yes, PowerBuilder's release unit is the PB library - the PowerBuilder "package".
Don't try to release smaller units.

Many PB code bases are 20+ years old. They started before the software patterns movement - so modern best practices didn't inform original architecture.

Cyclic dependencies are what kills ability to deliver small patches! Common coding styles are culprits. Examples:

  • App managers like PFC's gnv_App.
    They often refer to specific windows/DataWindows/menus while app manager itself being referenced from all over the app.
  • DataWindow controls as parameters
    This enables hardcoding dependency on column names, datatypes, grouping/sorting, etc.
    Every window/nvo/etc;. may this may hardcode depend on DataWindows on other windows/nvos/etc.
  • Bloated menu scripts
    Menu depends on window classes to open the windows.
    Menu also reaches into DataWindows to obtain data that are then passed to other clasess
    AND - windows and user objects customize the menu and hence depend on menu themselves.
  • Window coordination
    Windows coordinate action like save to DB or fill in Word document or open web app by reaching into each others internal structure to obtain relevant data.
  • IN GENERAL :: Lack of adherence to the dependency inversion principle
  • IN GENERAL :: Lack of dependency injection

In such structured app you can only create single-library patches if you never modify the API of any class or any DataWindow object. So you can release change to existing scripts but never a new script anywhere.

These days we should all consider "serviceability" (incl. patching) when we design new apps or modify existing apps. Consider the boy scout's rule: Always leave the camp ground a little cleaner than you found it! Or for code:

Always leave the code a little cleaner/better structured than you found it.
Over time your app becomes easier to patch.

HTH /Michael

Comment
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 10
mike S Accepted Answer Pending Moderation
1
Votes
Undo

I've never tried the approach you are taking, i use the patch.pbd method:

You can set your library list at runtime to add a patch.pbd at the top of your library list.  Put any changed objects in the patch.pbd, or any objects that need to be recompiled due to changes in other objects.

if you only change datawindow or script (not adding /deleting/changin definition of instance/global variables, methods, or events) then this method works great. If you do have to change something like adding an instance variable, then you must test all objects that call/use the object that was changed.  You may need to add the calling object to your patch file as well (regen the object).  

Comment
If we have an Emergency Bug Fix (EBF), we use the patch-PBD method as well. Of course, eventually we'll put out a new release.



Having that PBD at the top of the hierarchy chain of PBDs means that anything placed within it will override the same object in any subsequent PBD of the application. The warning Mike listed above are acacurate; changin ancestor code in a common object is NOT something for which you would normally use this type of patching.

  1. Olan Knight
  2. Thursday, 17 October 2019
We have a rule to add instance variables to the end when making emergency changes. The compiled code refers to instance variables by location, not by name.
  1. Roland Smith
  2. Thursday, 17 October 2019
instance variables at the end used to work for me in 12.5x and prior. I have seen some problems with doing that in PB2017.
  1. mike S
  2. Thursday, 17 October 2019
There are no comments made yet.
  1. Thursday, 17 October 2019
  2. PowerBuilder
  3. # 11
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.