This tutorial discusses how to interface traditional, Windows-hosted PowerBuilder 32-bit and 64-bit applications with the Windows Application Programming Interface (API), also known as the WinAPI. Much of the information presented here also applies to interfacing with DLL’s created in-house or by third-party vendors, because the interface mechanism in PB is the same. The focus, however, will be on interfacing to the WinAPI.
The tutorial is presented in four parts. Part one reviewed External Function Declarations (EFD’s) in PB and discussed issues related to interfacing with the WinAPI. The second part examined the calling conventions in 64-bit Windows and important differences between the 32-bit and 64-bit environments, particularly as it pertains to structures. Part three looks at several factors that can affect the interfacing of PB applications with the WinAPI, such as null values, PowerBuilder datatypes not supported by Windows, the PB Any datatype, unbounded arrays, nested structures and arrays of nested structures. The tutorial concludes in part four beginning with the description of an available free PB sample application and non-visual object that can dynamically determine the memory size and layout of a structure in either 32-bit or 64-bit environments. Part four also contains a list of coding tips & techniques to help you develop PB applications that interface with the WinAPI, and mapping tables to assist in the translation between many common WinAPI datatypes and the standard PB datatypes.
Why So Much Emphasis About Structures?
A considerable portion of this tutorial series is concerned with structures, so you may ask: “Why is so much emphasis being placed on structures?”
Frankly, it’s because the WinAPI makes heavy use of structures. There are literally thousands of functions in the WinAPI. I’ve never attempted to count them, but I found a listing of the functions in Windows’ kernel32.dll (just one of scores of Windows system DLL’s) and this list contained nearly 2,000 functions. I’m confident in stating that a significant number of them use one or more structures. In order to be able to effectively interface PB applications with the WinAPI in both 32-bit and 64-bit implementations, the PB developer needs to understand how information in structures is laid out and exchanged between PB and Windows. This becomes even more important when using a WinAPI structure where one of the structure members needs to specify the size of the structure in memory.
In addition to the general rules and conditions that were covered in part two of the tutorial, there are several additional factors that can influence structure layout and size:
- Structure members that reference PB objects.
- Null structure members.
- The use of PB datatypes not recognized by Windows.
- The PB Any datatype.
- Empty unbounded arrays.
- Differences in alignment and length of nested structures between 32-bit & 64-bit.
- Nested structure arrays.
We’ll examine the effect each of these can have on structure layout and size.
Structure Members That Reference PB Objects
What happens to the in-memory representation of a structure when a structure member is a PowerBuilder object reference variable… a variable that is not a standard PB datatype or structure, such as a DataWindow control?
Quite simply, PowerBuilder omits it! This is what my research shows.
Of course, a reference to a PB object conveys no information to the Windows operating system or to an external, third-party application – so presumably, it should not come as a surprise that PB omits object references from the traditional in-memory representation of a structure. Now, obviously, the object reference variable(s) are not tossed away completely, because we know for example that structures containing object references can be passed within PowerBuilder to newly-opened windows via the PowerObjectParm property of the Message object. Somehow, PB manages object references in structures differently than it manages standard datatypes. Exactly how it does this is, well… a mystery.
The important thing to recognize and remember is that this is what PB does. Using this knowledge, we conclude that any structure used to convey information to an external function should use only variables and/or arrays of standard PB datatypes or nested structures for its members… except the Any datatype and, maybe a few others, too. More on this topic soon.
Null Structure Members
When a member of a PowerBuilder structure is null, two questions come to mind: (1) What happens to the in-memory representation of the structure, and (2) is this identical or different from how Windows works? The answers may surprise you.
Note: Let’s temporarily postpone (for a little while longer!) any discussion of the PB Any datatype in structures, because its behavior is markedly different from the other standard PB datatypes.
What Is A Null Value in PB and in Windows?
The PowerScript language permits a variable of almost any datatype to be null. What does this mean and what happens to the in-memory representation of a structure member variable when it is null?
A null variable in PowerBuilder is generally considered to mean “the value is undefined”. In other words, you cannot assume, use or depend on the value in a null PB variable… it is unusable. However, even when its value is null, a variable exists in memory insomuch that space gets allocated for it. And, since a variable exists and every bit in memory must be either zero or one, every variable actually has a value regardless of whether that value is “usable” (i.e., null) or not.
Consider integer-type datatypes; all available bits are used to define a variable’s value, so it seems reasonable to assume that PB must track whether every variable is null by using another mechanism, possibly a “variable descriptor block” or something similar.
Windows, however, works differently. If a value is null, all bits are zero. For example, a variable that represents a pointer (memory address) contains zero if it is null. The PB developer has to recognize and understand this fundamental difference between PB and Windows. In the Windows API, argument parameters can be null to signify a particular condition or a set of conditions. When preparing the argument values to be passed to a Windows API function, you should explicitly set the value to zero if Windows expects to receive what it recognizes as null.
But what exactly does PB place in a structure when a member variable is null? Research and experiments show the memory representing a null structure member variable in PB contains the most recent (non-null) value. For example, if a structure member is of type Integer and that member is assigned the value 5, then subsequently set to null, the two bytes of memory allocated for use by that structure member will contain 0x0005. This is understandable if we assume PB keeps track of whether a variable is null separately from the variable itself.
Important: You must explicitly set variables to zero in PB if Windows expects to receive what it recognizes as null. The SetNull PowerScript function does not appear to affect any of the bits in the memory allocated for a PB variable.
Null Structure Member Arrays
In general, an array is a collection of variables of the same type laid out consecutively in memory. Therefore, the behavior described in the preceding section on null structure member variables applies to each element of a structure member array, also.
Using the SetNull PowerScript function on an array element is the same as using it on a (non-array) variable, so it behaves the same way. Furthermore, if you use the SetNull() function on the entire structure member array, it leaves the memory used by the entire array the same as if you issued SetNull on each array element individually.
Null Nested Structures
A structure can contain a structure. Here’s a tossup question and a chance to earn some Bonus Points:
What happens to the in-memory representation of a null structure member that is a nested structure?
Trick question! You cannot use the SetNull PowerScript function on a structure or an auto-instantiate object (the PB compiler flags the attempt as an error), so a structure in PB can never be null. The non-structure members within a main or nested structure, however, can be null.
The Use of PB Datatypes Not Recognized by Windows
Windows does not recognize or understand how to interpret the values in all standard PB datatypes. You should not attempt to pass the following PB datatypes to the WinAPI:
- Any (discussed in the next section)
- A reference variable for any type of object or class (discussed earlier)
Internally, Windows considers all types of visual controls to be a sub-classed form of a “window”. If a WinAPI function needs access to a PB window (or a visual control in a PB window), it will require it’s handle. The Handle PowerScript function returns the Windows handle for a PB window or visual control. Note that at the time this tutorial is written that the PowerScript Handle() function returns a value of type Long even though a Windows handle is actually a LongLong in a 64-bit process. (Note: There is an open enhancement request to Appeon to address this oversight.)
As mentioned earlier, it is interesting to note that when a PB structure contains a member that is a reference variable for an object or class (such as a window), examination of the memory used to store the contents of the structure shows that no memory is allocated in the structure for that reference variable. We can only postulate that PB keeps track of object/class structure members separately from those of the standard PB datatypes – not that this affects or pertains to interacting with the WinAPI… but it is nonetheless interesting.
DataWindowObjects Are Not Windows Controls
Because a DataWindow control (DWC) is a visual control in a window or user object, Windows considers it to be another type of sub-classed window. Thus, a DWC has a Windows handle and that handle can be obtained via the Handle PowerScript function.
To the Windows operating system, the visual content inside of a DWC is unknown because the DataWindow engine, not Windows, renders the display. None of the DataWindowObjects drawn by the DataWindow engine are visual controls drawn by the Windows operating system (they are just a rendered bitmap facsimile thereof) and hence, they do not have Windows handles nor can they be manipulated by WinAPI functions. This is one reason why automated testing software has difficulty interfacing effectively with PowerBuilder applications.
The PB Any Datatype
Enough procrastinating…let’s discuss Any’s!
The PowerBuilder Any datatype can be a powerful tool for the PB developer because it can mimic the behavior of all other PB datatypes. Although similar to the Variant datatype in Visual Basic, the PB Any datatype is unique to PowerBuilder and is therefore not designed to be compatible with the Windows API or other external interfaces. Generally, you should not use variables or structure members of type Any with EFD’s. A closer look at how Any’s are managed in memory as variables and as structure members will illustrate why.
The PB Any Variable in Memory
My experiments show that a PB Any variable occupies eight bytes in a 32-bit environment. The first four bytes appear to be a structure containing bit flags and code bytes. For example, there is a bit used to designate when the Any variable is null and a code byte or bytes that identifies the actual datatype of the value stored in the Any variable. The last four bytes contain a single value that can be interpreted in one of two possible ways. If the actual datatype is one that can be represented in four bytes or less (such as a Boolean, an UnsignedInteger, a Long or a Real), then these last four bytes contain the actual data value. However, if the actual data value requires more than four bytes, then the value in the last four bytes contain a pointer to another memory location where the actual data value is stored.
In a 64-bit environment, a PB Any variable occupies 12 bytes of memory; the first four bytes contain the same structure as the 32-bit version of Any, but the “value” portion now occupies eight bytes to accommodate the potentially larger pointer value (memory address). Actual data values of four bytes or less continue to be stored in the eight-byte “value” portion of the Any variable. The sample application described in part four of this tutorial includes a button that performs a series of tests involving variables and structure members of type Any, and the first test displays how different types of data are represented in an Any variable in memory.
The PB Any Variable as A WinAPI Function Argument Parameter
When used carefully, variables of type Any can be used to pass an argument value into a WinAPI function. This is especially true for the various integer datatypes. The PB compiler knows the datatype of every argument parameter to a DLL function, thanks to the EFD. Whenever possible, PB automatically converts an integer-type value in an Any variable into the needed datatype.
One of the Any-related tests that can be performed from the sample application illustrates this feature. The GetSysColor WinAPI function accepts a Windows system color ID code as a PB Long and returns the encoded RGB color value as a PB Long. In a test performed in the sample application, the same system color value is requested by using a variable of type Any containing each of the following types of integer values: Byte, UnsignedInteger, UnsignedLong, LongLong, Decimal and as an additional test, String. All of these datatypes in the Any variable work correctly. except for String since there is no automatic conversion attempted between the String and Long datatypes.
This (sort of) works – Syntactically, you can do this. The BIG question is: Should you do this?
In my opinion, you should not use Any variables to pass information to external functions. The technique depends on an automatic, behind-the-scenes data conversion that will not be obvious to anyone looking at your code. It’s better to use a variable of the argument parameter’s datatype in the first place. If an Any needs to be used in your application, first assign the value in the Any variable to a variable of the argument parameter’s datatype, then pass the value in that variable to the external function.
The PB Any Structure Member in Memory
An interesting thing occurs to the in-memory representation of a structure when it contains a member of type Any: The actual datatype of the Any structure member is placed into the structure! Of course, the real Any (variable) is tucked away out of sight, but PB can access it anytime access to it is needed. PB keeps the value of a structure member of type Any in sync with the actual data value held in the “behind the scenes” Any variable.
If a structure or a pointer to a structure is to be passed to a WinAPI function, there should be no problems associated with using a structure member of type Any, right?
Well, there can be a problem. Consider this hypothetical example: Take a small, whole-number value, such as 14. The value 14 can be rightfully assigned to seven different integer PB datatypes: Byte, Integer, UnsignedInteger, Long, UnsignedLong, Longptr and LongLong (even though the value 14 can be assigned to a variable of type Decimal, it is not an integer). If you assign the value 14 to a structure member of type Any and the WinAPI function expects that structure member to be of datatype LongLong, what happens? What datatype does the Any structure member believe it contains?
The PB compiler appears to consider small, whole-number (i.e., “hard-coded”) constants to be of type Long unless the value is too large/small (in a negative number sense) to be contained within four bytes (you can verify this easily by using the ClassName PowerScript function on the Any structure member after assigning it the value 14). Therefore, in this hypothetical example, the in-memory representation of the structure will contain the value 14 in four bytes, not eight (used for a LongLong), causing the incorrect number of bytes to be removed from the call stack during execution.
A work-around for this issue is to first assign the structure member of type Any a value from a declared variable (or another structure member) of the desired datatype. For our example, this would be a variable of type LongLong. You can also use the LongLong(value) PowerScript function in this case. Performing this action “sets” the datatype code in the Any variable first, then the value 14 can be assigned and the value will be treated as a LongLong value. Of course, if you’re going to go to that much trouble, why not simply define the structure member as a LongLong to begin with and avoid the added complexity and hassle?
The Null PB Any Structure Member in Memory
We mentioned earlier that internally within a PB Any variable, a bit flag in the four-byte structure that precedes the value portion of the Any variable designates whether the value is null. Additionally, if the Any structure member has ever held a non-null value, then that Any structure member “knows” what type of data it most recently held because its datatype code has been set.
What does a structure member of type Any contain when the member is set to null? Well, it depends primarily on whether the Any structure member has ever held a non-null value. If it has, then one of two things happen:
- If the value in the Any structure member is of a datatype that can be represented in four bytes or less, then the memory used by that structure member will continue to contain the same pattern of bits as it did before the structure member was set to null.
- If the pre-null value of the Any structure member is of a datatype that requires more than four bytes, then the memory used by that structure member are cleared (all bits are set to zero).
The third test involving Any’s performed by the sample application illustrates this behavior.
Even more interestingly, if the Any structure member has never held a non-null value (the datatype code in the Any's internal structure has never been set), PB has no way to determine what length of empty value should reside in the in-memory representation of the structure…so it omits it entirely! It’s pretty safe to assume that a WinAPI function expecting a structure to contain a particular member is going to produce an error when some of the bytes in the structure are missing! This is yet another reason to never use the Any datatype when interfacing to external methods via EFD’s.
The Any Array as A Structure Member
An array of datatype Any can be used to contain a collection of just about anything you can think of in the PowerBuilder realm. Since internally, every Any variable is a fixed size (eight or twelve bytes, depending on the process bitness), PB can efficiently manage the contents of an array of type Any even if the array contains a variety of datatypes.
Placing the actual data value represented in the Any variable when the Any is a structure member presents problems, however. If a structure member is an array of type Any, each element of the array could conceivably be “converted” into one to eight bytes, depending on the actual datatype of each element. There is no way that an external method in another DLL could correctly decode the stream of bits/bytes representing the array or the structure. PowerBuilder’s solution (which is not really a true solution) is to simply omit the Any array from the in-memory representation of the structure. Poof! It doesn’t exist (in the structure's allocated memory). So, not only should you not use an array of type Any in a structure to be used by an external function, you can’t!
Empty Unbounded Arrays in a Structure
An empty unbounded array is an array that contains no elements and has no preset limit on the number of elements it may contain. When a structure contains an empty unbounded array, experiments show that the in-memory representation of the structure contains a single, all zero-bits value of the appropriate size for that datatype.
For example, if a structure member is defined as an unbounded array of type Integer and the UpperBound PowerScript function verifies the array contains no elements, the in-memory representation of the structure will contain 0x0000 (two bytes) at the proper offset from the start of the structure. Thus, an empty, zero-element array of type Integer will appear to an external DLL method as a single-element Integer containing the value zero.
This does not occur just for the integer datatypes… it happens for all standard PB datatypes.
Boundary Alignment of Structures & Nested Structures (Review)
In part two I described how the boundary alignment for structures is determined by examining all members of the structure and using the largest boundary alignment of the members. The boundary alignment and size of the structure determines if any supplemental bytes of padding at the end of the structure are needed. Padding may also be needed ahead of nested structures as well as after them.
None of this occurs in the rare case where single-byte alignment (progma_pack( 1 )) is in effect, but the WinAPI uses only the normal, eight-byte alignment (progma_pack( 8 )) convention.
Nested Structure Arrays
A structure member that is an array of say, four nested structures is equivalent to four nested structures laid out consecutively in memory. For each one of the four nested structures, the starting position of the structure in memory is determined by the ending address of the previous one (including any padding needed to preserve structure alignment). All subsequent structure array elements will utilize the same boundary alignment of the first structure array element.
Structures are an important component of the WinAPI, so a complete understanding of PB structures and how they are laid out in memory is valuable. When interacting with the WinAPI, PowerBuilder null values need to be avoided because of the difference in how PB and Windows each represent null values. Windows does not recognize or support some of the standard PB datatypes. We showed why the PB Any datatype can be but, should not be used with external functions. How PowerBuilder manages empty unbounded arrays may surprise you – more importantly, it may surprise Windows if these are used with the WinAPI. In the final part of the tutorial, an available free sample application and non-visual object that can dynamically determine the memory size and layout of a structure in either 32-bit or 64-bit environments is described. The sample application also performs some tests that validate many of the behaviors described in this tutorial. Part four additionally lists several tips and techniques that can be used to develop PB code for interfacing with the WinAPI in both 32-bit and 64-bit executables. An appendix for translating many common WinAPI datatypes to PB datatypes is included.