1. John Niespodzianski
  2. PowerBuilder
  3. Wednesday, 29 May 2019

We're using PB 2017 R3 and making SOAP-based web service calls via the httpclient.

From our client system, we call an internal API that we host that serves as a wrapper for a 3rd party API to process credit card payments.

Everything has been working fine until the past week - the vendor API has had problems, which has resulted in all kinds of issues throughout the chain.

Long story short - we have our httpclient timeout set to 15 seconds.  If our intermediary web service takes greater than 15 seconds to respond, the httpclient appears to timeout correctly (our expected message pops, etc).  However, in the IIS logs we are seeing up to 3 requests initiated from the client system -- spaced 15 seconds a part.

Since this is payment logic, we have no looping or retry logic in place -- everything either works 100% and we post a payment, or we stop everything and roll back to prevent double payments, etc.

Has anyone seen anything like this with the httpclient?  I wouldn't imagine that it would retry the request a 2nd or 3rd time if the timeout value is reached - but that's exactly what it looks like.  We're seeing the multiple requests both in the IIS logs and via WireShark, even though we're only calling SendRequest() once.

Thanks,

John

Who is viewing this page
Jeff Kandt Accepted Answer Pending Moderation
1
Votes
Undo

FYI our issue was resolved with a workaround provided by Appeon support.

In our case, setting SecurityProtocol to "5" (TLS 1.2) solved the problem.

When HTTPClient has a specific protocol set for its "securityprotocol" attribute, the retries no longer occur and the object reliably times out at the specified interval.

Apparently this is a bug in how the object implements automatic detection of the security protocol when the default value, "0", is not changed. The default represents "Automatic" and according to the documentation "It will detect and use the secure protocol in the following order: TLS 1.2, TLS 1.1, SSL 2.0/SSL 3.0/TLS 1.0."

So apparently the bug is that it is misinterpreting the server disconnect that occurs in our environment after 30 seconds as a security protocol failure which caused it to retry again with the next protocol.

The only remaining (minor) issue is that although the SendRequest() now times out at exactly the specified number of seconds, the return code is still "-1" (General Error), not "-4" as documented for a Timeout. But it looks like we can reliably infer a timeout by measuring how long it takes for the SendRequest() to return the error -- if the -1 is returned after a duration that's within a half-second of the timeout setting, it probably timed out.

Comment
Jeff, thanks for the update - we're going to test this to see if explicitly setting to TLS 1.2 will give us the expected result.



John
  1. John Niespodzianski
  2. Monday, 23 September 2019
There are no comments made yet.
  1. Monday, 23 September 2019
  2. PowerBuilder
  3. # 1
Tom Jiang Accepted Answer Pending Moderation
0
Votes
Undo

Dear Everyone, 

The HTTPClient had implemented an auto-retry feature which must be causing this issue. The developer is analyzing the issue and will provide a solution as soon as possible. 

Regards,

Tom Jiang

Comment
Thank you Tom! If you have any idea about a timeline and whether a fix will be provided in a '17 R3 patch, that would help me plan our path forward.
  1. Jeff Kandt
  2. Tuesday, 17 September 2019
Hi Jeff, I will submit a request for fixing it in a 17 R3 patch. I will update you when I get any feedback.



  1. Tom Jiang
  2. Wednesday, 18 September 2019
There are no comments made yet.
  1. Tuesday, 17 September 2019
  2. PowerBuilder
  3. # 2
Roland Smith Accepted Answer Pending Moderation
0
Votes
Undo

I have two code examples that may be of use:

This one uses the WinHTTP API functions:

http://www.topwizprogramming.com/freecode_winhttp.html

This one uses the Msxml2.XMLHTTP COM control:

http://www.topwizprogramming.com/freecode_xmlhttp.html

It is coded to expect XML but that could be removed easily.

Comment
Thanks Roland, your WinHTTP was already at the top of my list, I'll check the other one out too.
  1. Jeff Kandt
  2. Tuesday, 17 September 2019
There are no comments made yet.
  1. Monday, 16 September 2019
  2. PowerBuilder
  3. # 3
John Niespodzianski Accepted Answer Pending Moderation
0
Votes
Undo

I have not tested the same scenario with the RestClient; our current web service is SOAP based so until it's rewritten I'm locked into the HTTPClient.  

If/when time permits I'll write a simple REST service to try to mimic the same test scenario and report back.

Comment
I'd be thrilled if RestClient was a viable workaround, but my understanding is that it's specialized to pipe json response data directly into a datawindow -- GET only. In fact I don't see that the RestClient can even do a POST -- there's no SendRequest() method.

We're consuming complex json documents that need to be manually parsed, so haven't been able to use that object even for GETs.

The old PostURL doesn't support TLS 1.2, so I guess I'm looking for a third-party control. (Any suggestions are welcome.)
  1. Jeff Kandt
  2. Thursday, 12 September 2019
Jeff, in PB 2019 the RESTCLIENT was enhanced to allow GET/POST/PUT/PATCH/DELETE requests.



https://community.appeon.com/index.php/articles-blogs/tutorials-articles/2-powerbuilder/244-rest-enhancements-in-powerbuilder-2019
  1. John Niespodzianski
  2. Friday, 13 September 2019
Ah thanks, didn't realize that. I wouldn't be surprised if RestClient is based on the same code under the covers as HTTPClient, which we know hasn't been fixed in PB2019.

At any rate, we're in late testing for a big project right now, and taking on a PB upgrade (from 2017 R3) probably isn't palatable right now. If we're going to need to switch to another HTTP object anyway, we'll try some of the third-party ones first.
  1. Jeff Kandt
  2. Tuesday, 17 September 2019
There are no comments made yet.
  1. Thursday, 12 September 2019
  2. PowerBuilder
  3. # 4
Jeff Kandt Accepted Answer Pending Moderation
0
Votes
Undo

Ugh, well shortening the timeout does indeed do what you've just described: With timeout set to 20 seconds, it didn't actually emerge from SendRequest() for 60 seconds, after sending three requests. (It did return a -1 return code -- is that what you're seeing?)

But whereas setting a long timeout works for you, it doesn't for me, and I think that's because of something specific to the server I'm talking to.

In your case setting a long timeout allows the Client to receive a proper timeout message from the server that it processes as a failure and doesn't cause it to retry. But the way our server times out is by sending this weird disconnect message after 30 seconds which also causes HTTPClient to retry three times.

So what's the weird thing my server is doing?

I used Wireshark while testing with SoapUI to determine that there was no network traffic between client and server during those 30 seconds, and that at exactly 30 seconds the server replied and that's when SoapUI failed. Unfortunately, because this is https, the traffic is encrypted so I can't actually see what the server is sending at 30 seconds, I just know SoapUI interprets is as a "NoHttpResponseException". But at least this appears to rule out something on the client causing the 30 second timeout.

A little bit of research on the internet indicates that http servers like Apache often have an "idle session timeout" which defaults to 30 seconds. To conserve connection pool resources. if a session is active but idle for 30 seconds, it hangs up on the client to return the connection to the pool, and that comes back as a NoHttpResponseException.

I believe I read that HTTPClient is a wrapper over Microsoft's WinHTTP API? I poked around the documentation a bit and didn't find any reference to auto-retry functionality like this. Did the Appeon developers add it themselves? Either way, for a GET or PUT, retrying might be fine, but for a POST a retry is a terrible idea, and having no control over this behavior puts me between a rock and a hard place.

As I can tell I'm left with NO workaround at all within the HTTPClient object, and will need to pursue either giving up on HTTPClient and using some other HTTP object, or trying to get configuration changes (which will effect the whole enterprise and may raise concerns about server resources) made at the server, to accommodate PowerBuilder.

 

Comment
There are no comments made yet.
  1. Thursday, 12 September 2019
  2. PowerBuilder
  3. # 5
John Niespodzianski Accepted Answer Pending Moderation
0
Votes
Undo

Hi Jeff, I did open Bug 2826 for this issue.

If we set the HTTPClient timeout greater than the web service timeout, everything is handled gracefully.  The service returns a failed response and we're able to handle in our TRY/CATCH logic.

If the HTTPClient timeout is less than the server timeout, it will keep retrying the request at the specified client interval until it gets a response from the web service.  

From your example, it sounds like your HTTPClient is set to a 30 second timeout so it keeps reissuing the call every 30 seconds (which is the bug we're seeing).

Note that we only see this when it's an HTTPS request; we didn't observe this behavior when sending an HTTP request.  Since we're dealing with credit card data, HTTP is not an option.  So we had to use the workaround of having the server timeout before the client to prevent the possibility of duplicate (or triplicate) calls.

Thanks,

John

Comment
Hi John;

No action on that ticket. I just pinged Engineering on this. The problem is in PB2017R3 & PB2019GA currently.

Regards ... Chris
  1. Chris Pollach
  2. Thursday, 12 September 2019
Thanks Chris - much appreciated!



-John
  1. John Niespodzianski
  2. Thursday, 12 September 2019
There are no comments made yet.
  1. Thursday, 12 September 2019
  2. PowerBuilder
  3. # 6
Jeff Kandt Accepted Answer Pending Moderation
0
Votes
Undo

John- Did a bug get submitted? I'm not finding one via the Standard Support portal.

I think we're seeing this -- or a very similar -- problem, PB 2017 R3. In our case it's a restful service and we're posting to an https (basic authentication) service.

I have a question about your scenario though: If I'm understanding correctly, the client times out after 15 seconds as expected .. but the service STILL receives 2 additional calls over the next 30 seconds? So even though the flow of execution has returned to you, there's still some background process running and issuing retries?  If so, that's terrifying...

I still have a lot or testing to do, but here's what I think I'm seeing so far:

In our case, in the debugger it steps into the SendRequest() function and comes out 90 seconds later with a "-1" error (timeout was set to 100 seconds). Meanwhile, during those 90 seconds the receiving service gets 3 duplicate requests exactly 30 seconds apart.

This is a service we use to submit orders, so we end up placing three identical orders -- not good!

In our case I believe the service (or something in between -- we have a complex environment with a middleware layer in between), seems to be timing out after exactly 30 seconds. In SoapUI at 30 seconds I get "org.apache.http.NoHttpResponseException: The target server failed to respond" which is a different error than the one you get when SoapUI itself times out. My theory is that the HTTPClient responds to this exception by retrying three times before returning "-1" (not "-2" as would be expected for timeout).

I'm not sure I understand the theory behind your proposed workaround of increasing the client timeout, and in fact I was going to try the opposite of what you suggest: I was hoping that setting the HTTPClient timeout to LOWER than the service timeout would cause the client to give up before getting and responding to that first "NoHttpResponse" exception and thus avoid retries. But what you're describing makes it sound like the retries may be unavoidable...

I've had problems getting HTTPClient to honor the timeout we set, but will be experimenting with that next. Any additional advice or clarity would be appreciated!

 

Comment
There are no comments made yet.
  1. Thursday, 12 September 2019
  2. PowerBuilder
  3. # 7
Kevin Ridley Accepted Answer Pending Moderation
0
Votes
Undo

Might be worth looking into if they have a REST API as well.

Comment
We're actually consuming a REST API from the vendor, but it's wrappered by a SOAP based API internally. This was done many years ago before REST support directly existed in PB. Long-term we're going to rewrite the internal API as a RESTful service; I have not had the chance to test the RESTClient to see if the same issue exists there as it does in the HTTPClient.
  1. John Niespodzianski
  2. Thursday, 11 July 2019
There are no comments made yet.
  1. Thursday, 11 July 2019
  2. PowerBuilder
  3. # 8
John Niespodzianski Accepted Answer Pending Moderation
0
Votes
Undo

Vipin, we opened a support ticket for this bug with Appeon.  To my knowledge it has not been fixed yet.

Our workaround is to extend our timeout beyond the API's timeout.

So if the API times out at 60 seconds, we set our timeout at 65 seconds to ensure no duplicate requests are posted.  Not ideal, but we can live with it while the issue gets addressed.

Thanks,

John

Comment
There are no comments made yet.
  1. Thursday, 11 July 2019
  2. PowerBuilder
  3. # 9
Vipin Dwivedi Accepted Answer Pending Moderation
0
Votes
Undo

Hi John,

Did you get this fixed?

 

Thanks, Vipin

Comment
There are no comments made yet.
  1. Thursday, 11 July 2019
  2. PowerBuilder
  3. # 10
Vipin Dwivedi Accepted Answer Pending Moderation
0
Votes
Undo

Hi Tom,

Did you find if this is a httpclient bug in PB2017R3. We are planning to make some changes in Timeout in our application to not wait longer at client side. We are running PB2017R3 build 1858.

 

Please confirm.

Thanks, Vipin

Comment
There are no comments made yet.
  1. Thursday, 11 July 2019
  2. PowerBuilder
  3. # 11
Roland Smith Accepted Answer Pending Moderation
0
Votes
Undo

An auto-retry might be a useful feature, not in your case since you are using it for credit cards.

Comment
There are no comments made yet.
  1. Thursday, 30 May 2019
  2. PowerBuilder
  3. # 12
Tom Jiang Accepted Answer Pending Moderation
0
Votes
Undo

Hi John,

It seems to be a bug. Please open a Support Ticket for this issue (at https://www.appeon.com/standardsupport) and also attach a simple PB app to the ticket as a test case.


Many thanks in advance!

Regards,

Tom Jiang

Comment
Additional info on this - in case it benefits others. We'll be opening a support ticket as suggested. I'm also going to test with PB 2019 to see if the httpclient behaves differently.



When we hit against an endpoint via http -- we get one request (it times out at 15 seconds as expected, and there's no additional requests in the IIS logs, web service logs, or in the network traffic via WireShark).



When we hit the same endpoint via https -- we get the extra requests at whatever interval the httpclient.timeout is set at. So when we bump it from 15 seconds to 30 seconds, we see the extra requests 30 seconds apart.



I also switched back to the old SOAP proxy approach (instead of httpclient) and kept all other variables the same. Whether http or https it times out and only issued one request.



So it does appear the httpclient has an issue when issuing a SendRequest (Post via HTTPS) & the timeout is reached.

Our temporary workaround is to set the httpclient timeout > the web service timeout, so we'll react to that timeout before the client timeout is reached.
  1. John Niespodzianski
  2. Friday, 31 May 2019
There are no comments made yet.
  1. Thursday, 30 May 2019
  2. PowerBuilder
  3. # 13
  • Page :
  • 1


There are no replies made for this question yet.
However, you are not allowed to reply to this question.