1. John Niespodzianski
  2. PowerBuilder
  3. Wednesday, 29 May 2019 21:31 PM UTC

We're using PB 2017 R3 and making SOAP-based web service calls via the httpclient.

From our client system, we call an internal API that we host that serves as a wrapper for a 3rd party API to process credit card payments.

Everything has been working fine until the past week - the vendor API has had problems, which has resulted in all kinds of issues throughout the chain.

Long story short - we have our httpclient timeout set to 15 seconds.  If our intermediary web service takes greater than 15 seconds to respond, the httpclient appears to timeout correctly (our expected message pops, etc).  However, in the IIS logs we are seeing up to 3 requests initiated from the client system -- spaced 15 seconds a part.

Since this is payment logic, we have no looping or retry logic in place -- everything either works 100% and we post a payment, or we stop everything and roll back to prevent double payments, etc.

Has anyone seen anything like this with the httpclient?  I wouldn't imagine that it would retry the request a 2nd or 3rd time if the timeout value is reached - but that's exactly what it looks like.  We're seeing the multiple requests both in the IIS logs and via WireShark, even though we're only calling SendRequest() once.

Thanks,

John

Tom Jiang @Appeon Accepted Answer Pending Moderation
  1. Thursday, 30 May 2019 02:11 AM UTC
  2. PowerBuilder
  3. # 1

Hi John,

It seems to be a bug. Please open a Support Ticket for this issue (at https://www.appeon.com/standardsupport) and also attach a simple PB app to the ticket as a test case.


Many thanks in advance!

Regards,

Tom Jiang

Comment
  1. John Niespodzianski
  2. Friday, 31 May 2019 20:02 PM UTC
Additional info on this - in case it benefits others. We'll be opening a support ticket as suggested. I'm also going to test with PB 2019 to see if the httpclient behaves differently.



When we hit against an endpoint via http -- we get one request (it times out at 15 seconds as expected, and there's no additional requests in the IIS logs, web service logs, or in the network traffic via WireShark).



When we hit the same endpoint via https -- we get the extra requests at whatever interval the httpclient.timeout is set at. So when we bump it from 15 seconds to 30 seconds, we see the extra requests 30 seconds apart.



I also switched back to the old SOAP proxy approach (instead of httpclient) and kept all other variables the same. Whether http or https it times out and only issued one request.



So it does appear the httpclient has an issue when issuing a SendRequest (Post via HTTPS) & the timeout is reached.

Our temporary workaround is to set the httpclient timeout > the web service timeout, so we'll react to that timeout before the client timeout is reached.
  1. Helpful
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Thursday, 30 May 2019 10:29 AM UTC
  2. PowerBuilder
  3. # 2

An auto-retry might be a useful feature, not in your case since you are using it for credit cards.

Comment
There are no comments made yet.
Vipin Dwivedi Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 19:49 PM UTC
  2. PowerBuilder
  3. # 3

Hi Tom,

Did you find if this is a httpclient bug in PB2017R3. We are planning to make some changes in Timeout in our application to not wait longer at client side. We are running PB2017R3 build 1858.

 

Please confirm.

Thanks, Vipin

Comment
There are no comments made yet.
Vipin Dwivedi Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 19:50 PM UTC
  2. PowerBuilder
  3. # 4

Hi John,

Did you get this fixed?

 

Thanks, Vipin

Comment
There are no comments made yet.
John Niespodzianski Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 20:35 PM UTC
  2. PowerBuilder
  3. # 5

Vipin, we opened a support ticket for this bug with Appeon.  To my knowledge it has not been fixed yet.

Our workaround is to extend our timeout beyond the API's timeout.

So if the API times out at 60 seconds, we set our timeout at 65 seconds to ensure no duplicate requests are posted.  Not ideal, but we can live with it while the issue gets addressed.

Thanks,

John

Comment
There are no comments made yet.
Kevin Ridley Accepted Answer Pending Moderation
  1. Thursday, 11 July 2019 20:47 PM UTC
  2. PowerBuilder
  3. # 6

Might be worth looking into if they have a REST API as well.

Comment
  1. John Niespodzianski
  2. Thursday, 11 July 2019 21:07 PM UTC
We're actually consuming a REST API from the vendor, but it's wrappered by a SOAP based API internally. This was done many years ago before REST support directly existed in PB. Long-term we're going to rewrite the internal API as a RESTful service; I have not had the chance to test the RESTClient to see if the same issue exists there as it does in the HTTPClient.
  1. Helpful
  1. Aires Manlig
  2. Tuesday, 17 December 2019 18:52 PM UTC
I'm using PB 2019 Build 2082. A RestClient object refuses to timeout even when the Timeout property is set. But an HttpClient object will timeout accordingly. Any idea on the situation with RestClient?
  1. Helpful
There are no comments made yet.
Jeff Kandt Accepted Answer Pending Moderation
  1. Thursday, 12 September 2019 16:59 PM UTC
  2. PowerBuilder
  3. # 7

John- Did a bug get submitted? I'm not finding one via the Standard Support portal.

I think we're seeing this -- or a very similar -- problem, PB 2017 R3. In our case it's a restful service and we're posting to an https (basic authentication) service.

I have a question about your scenario though: If I'm understanding correctly, the client times out after 15 seconds as expected .. but the service STILL receives 2 additional calls over the next 30 seconds? So even though the flow of execution has returned to you, there's still some background process running and issuing retries?  If so, that's terrifying...

I still have a lot or testing to do, but here's what I think I'm seeing so far:

In our case, in the debugger it steps into the SendRequest() function and comes out 90 seconds later with a "-1" error (timeout was set to 100 seconds). Meanwhile, during those 90 seconds the receiving service gets 3 duplicate requests exactly 30 seconds apart.

This is a service we use to submit orders, so we end up placing three identical orders -- not good!

In our case I believe the service (or something in between -- we have a complex environment with a middleware layer in between), seems to be timing out after exactly 30 seconds. In SoapUI at 30 seconds I get "org.apache.http.NoHttpResponseException: The target server failed to respond" which is a different error than the one you get when SoapUI itself times out. My theory is that the HTTPClient responds to this exception by retrying three times before returning "-1" (not "-2" as would be expected for timeout).

I'm not sure I understand the theory behind your proposed workaround of increasing the client timeout, and in fact I was going to try the opposite of what you suggest: I was hoping that setting the HTTPClient timeout to LOWER than the service timeout would cause the client to give up before getting and responding to that first "NoHttpResponse" exception and thus avoid retries. But what you're describing makes it sound like the retries may be unavoidable...

I've had problems getting HTTPClient to honor the timeout we set, but will be experimenting with that next. Any additional advice or clarity would be appreciated!

 

Comment
There are no comments made yet.
John Niespodzianski Accepted Answer Pending Moderation
  1. Thursday, 12 September 2019 17:58 PM UTC
  2. PowerBuilder
  3. # 8

Hi Jeff, I did open Bug 2826 for this issue.

If we set the HTTPClient timeout greater than the web service timeout, everything is handled gracefully.  The service returns a failed response and we're able to handle in our TRY/CATCH logic.

If the HTTPClient timeout is less than the server timeout, it will keep retrying the request at the specified client interval until it gets a response from the web service.  

From your example, it sounds like your HTTPClient is set to a 30 second timeout so it keeps reissuing the call every 30 seconds (which is the bug we're seeing).

Note that we only see this when it's an HTTPS request; we didn't observe this behavior when sending an HTTP request.  Since we're dealing with credit card data, HTTP is not an option.  So we had to use the workaround of having the server timeout before the client to prevent the possibility of duplicate (or triplicate) calls.

Thanks,

John

Comment
  1. Chris Pollach @Appeon
  2. Thursday, 12 September 2019 18:05 PM UTC
Hi John;

No action on that ticket. I just pinged Engineering on this. The problem is in PB2017R3 & PB2019GA currently.

Regards ... Chris
  1. Helpful
  1. John Niespodzianski
  2. Thursday, 12 September 2019 18:08 PM UTC
Thanks Chris - much appreciated!



-John
  1. Helpful
There are no comments made yet.
Jeff Kandt Accepted Answer Pending Moderation
  1. Thursday, 12 September 2019 21:08 PM UTC
  2. PowerBuilder
  3. # 9

Ugh, well shortening the timeout does indeed do what you've just described: With timeout set to 20 seconds, it didn't actually emerge from SendRequest() for 60 seconds, after sending three requests. (It did return a -1 return code -- is that what you're seeing?)

But whereas setting a long timeout works for you, it doesn't for me, and I think that's because of something specific to the server I'm talking to.

In your case setting a long timeout allows the Client to receive a proper timeout message from the server that it processes as a failure and doesn't cause it to retry. But the way our server times out is by sending this weird disconnect message after 30 seconds which also causes HTTPClient to retry three times.

So what's the weird thing my server is doing?

I used Wireshark while testing with SoapUI to determine that there was no network traffic between client and server during those 30 seconds, and that at exactly 30 seconds the server replied and that's when SoapUI failed. Unfortunately, because this is https, the traffic is encrypted so I can't actually see what the server is sending at 30 seconds, I just know SoapUI interprets is as a "NoHttpResponseException". But at least this appears to rule out something on the client causing the 30 second timeout.

A little bit of research on the internet indicates that http servers like Apache often have an "idle session timeout" which defaults to 30 seconds. To conserve connection pool resources. if a session is active but idle for 30 seconds, it hangs up on the client to return the connection to the pool, and that comes back as a NoHttpResponseException.

I believe I read that HTTPClient is a wrapper over Microsoft's WinHTTP API? I poked around the documentation a bit and didn't find any reference to auto-retry functionality like this. Did the Appeon developers add it themselves? Either way, for a GET or PUT, retrying might be fine, but for a POST a retry is a terrible idea, and having no control over this behavior puts me between a rock and a hard place.

As I can tell I'm left with NO workaround at all within the HTTPClient object, and will need to pursue either giving up on HTTPClient and using some other HTTP object, or trying to get configuration changes (which will effect the whole enterprise and may raise concerns about server resources) made at the server, to accommodate PowerBuilder.

 

Comment
There are no comments made yet.
John Niespodzianski Accepted Answer Pending Moderation
  1. Thursday, 12 September 2019 21:24 PM UTC
  2. PowerBuilder
  3. # 10

I have not tested the same scenario with the RestClient; our current web service is SOAP based so until it's rewritten I'm locked into the HTTPClient.  

If/when time permits I'll write a simple REST service to try to mimic the same test scenario and report back.

Comment
  1. Jeff Kandt
  2. Thursday, 12 September 2019 22:58 PM UTC
I'd be thrilled if RestClient was a viable workaround, but my understanding is that it's specialized to pipe json response data directly into a datawindow -- GET only. In fact I don't see that the RestClient can even do a POST -- there's no SendRequest() method.

We're consuming complex json documents that need to be manually parsed, so haven't been able to use that object even for GETs.

The old PostURL doesn't support TLS 1.2, so I guess I'm looking for a third-party control. (Any suggestions are welcome.)
  1. Helpful
  1. John Niespodzianski
  2. Friday, 13 September 2019 14:33 PM UTC
Jeff, in PB 2019 the RESTCLIENT was enhanced to allow GET/POST/PUT/PATCH/DELETE requests.



https://community.appeon.com/index.php/articles-blogs/tutorials-articles/2-powerbuilder/244-rest-enhancements-in-powerbuilder-2019
  1. Helpful
  1. Jeff Kandt
  2. Tuesday, 17 September 2019 21:12 PM UTC
Ah thanks, didn't realize that. I wouldn't be surprised if RestClient is based on the same code under the covers as HTTPClient, which we know hasn't been fixed in PB2019.

At any rate, we're in late testing for a big project right now, and taking on a PB upgrade (from 2017 R3) probably isn't palatable right now. If we're going to need to switch to another HTTP object anyway, we'll try some of the third-party ones first.
  1. Helpful
There are no comments made yet.
Roland Smith Accepted Answer Pending Moderation
  1. Monday, 16 September 2019 13:20 PM UTC
  2. PowerBuilder
  3. # 11

I have two code examples that may be of use:

This one uses the WinHTTP API functions:

http://www.topwizprogramming.com/freecode_winhttp.html

This one uses the Msxml2.XMLHTTP COM control:

http://www.topwizprogramming.com/freecode_xmlhttp.html

It is coded to expect XML but that could be removed easily.

Comment
  1. Jeff Kandt
  2. Tuesday, 17 September 2019 21:13 PM UTC
Thanks Roland, your WinHTTP was already at the top of my list, I'll check the other one out too.
  1. Helpful
There are no comments made yet.
Tom Jiang @Appeon Accepted Answer Pending Moderation
  1. Tuesday, 17 September 2019 10:46 AM UTC
  2. PowerBuilder
  3. # 12

Dear Everyone, 

The HTTPClient had implemented an auto-retry feature which must be causing this issue. The developer is analyzing the issue and will provide a solution as soon as possible. 

Regards,

Tom Jiang

Comment
  1. Jeff Kandt
  2. Tuesday, 17 September 2019 21:16 PM UTC
Thank you Tom! If you have any idea about a timeline and whether a fix will be provided in a '17 R3 patch, that would help me plan our path forward.
  1. Helpful
  1. Tom Jiang @Appeon
  2. Wednesday, 18 September 2019 10:20 AM UTC
Hi Jeff, I will submit a request for fixing it in a 17 R3 patch. I will update you when I get any feedback.



  1. Helpful
There are no comments made yet.
Jeff Kandt Accepted Answer Pending Moderation
  1. Monday, 23 September 2019 14:41 PM UTC
  2. PowerBuilder
  3. # 13

FYI our issue was resolved with a workaround provided by Appeon support.

In our case, setting SecurityProtocol to "5" (TLS 1.2) solved the problem.

When HTTPClient has a specific protocol set for its "securityprotocol" attribute, the retries no longer occur and the object reliably times out at the specified interval.

Apparently this is a bug in how the object implements automatic detection of the security protocol when the default value, "0", is not changed. The default represents "Automatic" and according to the documentation "It will detect and use the secure protocol in the following order: TLS 1.2, TLS 1.1, SSL 2.0/SSL 3.0/TLS 1.0."

So apparently the bug is that it is misinterpreting the server disconnect that occurs in our environment after 30 seconds as a security protocol failure which caused it to retry again with the next protocol.

The only remaining (minor) issue is that although the SendRequest() now times out at exactly the specified number of seconds, the return code is still "-1" (General Error), not "-4" as documented for a Timeout. But it looks like we can reliably infer a timeout by measuring how long it takes for the SendRequest() to return the error -- if the -1 is returned after a duration that's within a half-second of the timeout setting, it probably timed out.

Comment
  1. John Niespodzianski
  2. Monday, 23 September 2019 20:57 PM UTC
Jeff, thanks for the update - we're going to test this to see if explicitly setting to TLS 1.2 will give us the expected result.



John
  1. Helpful
There are no comments made yet.
Tom Jiang @Appeon Accepted Answer Pending Moderation
  1. Wednesday, 7 October 2020 01:15 AM UTC
  2. PowerBuilder
  3. # 14
Update: The issue has been fixed in the latest PB 2017 R3 and PB 2019 R2.
Comment
  1. Chris Pollach @Appeon
  2. Monday, 7 June 2021 16:46 PM UTC
Hi Vipin;

Just to add to Armeen's post ... your PB App sets the timeout, for example



HttpClient lnv_HttpClient

lnv_HttpClient = Create HttpClient

//Set timeout to 100 seconds

lnv_HttpClient.Timeout = 100



However, if you always want to impose a 100 second timeout application wide standard, the I would suggest creating an HTTPClient "Ancestor" and then set the timeout in there to 100 seconds. Then all descendant HTTPClient object classes will follow the 100 timeout setting by default. Food for thought. ;-)

Regards ... Chris
  1. Helpful
  1. Vipin Dwivedi
  2. Monday, 7 June 2021 22:27 PM UTC
Thank you Chris, we are also doing the same thing. The code which I showed you is base code which all descendent objects are calling. I am trying with/without SecureProtocol along with Timeout but system SendRequest is does not timeout after time set in Timeout. I tried with 100 second as well as 5 sec but result is same.



Also why PB has set default Timeout to 60 while DOTNET has 100 seconds?



Do I have to upgrade to new version to make it work properly?
  1. Helpful
  1. Chris Pollach @Appeon
  2. Monday, 7 June 2021 23:12 PM UTC
Yes, you'll need to get up to build 2703 (MR01) for PB2019R3 or 1915 for R2 as Armeen mentioned.

I suspect that Appeon Engineering team just took an educated guess on the default timeout based on their initial testing. However once you create your own ancestor, you can set it to the same as .Net. That's the beauty of OO programming. :-)

Regards ... Chris
  1. Helpful
There are no comments made yet.
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.