I'm come upon a wierd problem with java HttpClient library.
Specifically the library automatically retries my request (POST requests)
even before the response is received from the server. Moreover the weirder problem
is that this only happens on specific hosts (machines).
So the end result is if a post request succeeds, then there may be an exact same
post request coming to the server which the server can't handle. Now, I do want
the retry behavior, but it should behave intuitively.
Anyone faced this kind of problem before, or is there a way to configure
http client to wait for a specific time before retrying. I'm not sure what going
wrong here.
Do you have a methodretryhandler set for your HttpClient? As in:
DefaultMethodRetryHandler retryhandler = new DefaultMethodRetryHandler(10, true);
client.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, retryhandler);
That is where retries would originate and you could debug and see what response headers it's receiving if any, etc.
Have you tried using a firefox http monitor or ethereal or similar to look through your http requests and responses and ensure that what you believe is happening is actually hapening?
Related
This question is maybe more a tip for people to search a solution if they have the same problem (as I found the solution eventually).
I had an application that does some HTTP requests with a local server (a mix of GET/POST with JSON content in the request/response bodies). The server is a third-party application, and after I upgraded it to a recent version, my Delphi app was no longer working.
It turned out that it was now hanging on the statement:
IdHTTP.Post("URL", "Payload", "BytesStreamResult");
As a manual POSTMAN request was still working, it had to be on the Delphi client side.
Further isolating the issue showed that the HTTP POST request did get an HTTP 200 response with valid HTTP response headers, but then was getting stuck reading the response body. It was hanging on:
IOHandler.ReadLn
When I compared the headers with the POSTMAN response, I noticed that 'Transfer-Encoding: chunked' was missing in the Delphi response.
Finally, I noticed the code related to TIdHTTP's hoKeepOrigProtocol option, which is not set by default.
So, my POST request was "downgraded" to an HTTP 1.0 request, and I guess this now made the (updated) server to respond differently (I'm not an RFC expert, but I guess 'chunked' is maybe an HTTP 1.1 option only).
After setting this option, everything worked like before (and indeed, the response was now read as "chunked" in Delphi).
Summary:
Shouldn't hoKeepOrigProtocol be the default option? (why punish good citizens for those that are not...)
Can we intercept this? Now my POST is assuming upfront a streamed response and thus it hangs because the server doesn't write anything to the buffer.
What would that high-level code look like? As it seems a mix of interpreting the header response headers and then deciding if more response reading is required.
(it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes...)
TIdHTTP supports non-chunked responses (which yes, is an HTTP 1.1 feature), so the hanging would have to be caused by the server sending a malformed response (a bug that should be reported to the server author).
When reading a non-chunked and non-MIME response, TIdHTTP does not use IOHandler.ReadLn to read the response's body, as you claim. Only when reading the response's headers.
But, since you did not show what the response actually looks like, nobody can explain for sure exactly why the hang occurs.
Shouldn't hoKeepOrigProtocol be the default option?
At the time the option was first introduced, no. There were plenty of buggy HTTP 1.1 servers around that downgrading to HTTP 1.0 was warranted.
However, that was many years ago. Nowadays, HTTP 1.1 is much more mature, and such buggy servers are rare. So, feel free to submit a change/pull request to Indy's GitHub repo if you feel the default behavior should be changed.
Can we intercept this?
No. The behavior you describe is most likely caused by a bug in the HTTP server. Either it is not sending all of the data it should be, or else the response is likely malformed in a way that makes TIdHTTP expect more data than is actually being sent. Either way, all you can do is assign a non-infinite timeout to TIdHTTP.
it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes.
Indy is designed to use infinite timeouts by default. You can assign custom timeouts to TIdHTTP's ConnectTimeout and ReadTimeout properties.
Setting this prevent the HTTP protocol downgrade:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoKeepOrigProtocol];
This is, of course, dependant upon how the server processes the protocol specification, and if it results in issues or not.
I call a http request,The reponse is html,but gatling get the response is incomplete.What should I do
I think a part of I need that is gatling supported resources.It is under the tag 'table'.
The server may not be returning the complete response due to an error or a problem with the server-side code. In this case, you should check the server logs to see if there are any errors, and you should also check the HTTP response headers to see if there are any indications of what went wrong.
The HTTP request may be failing or being blocked by a firewall or other network security device. In this case, you should check the network logs to see if the request is being sent and received successfully, and you should also check any network security settings to ensure that the request is not being blocked.
The HTML response may not be well-formed or may be missing some elements, such as the 'table' element you mentioned. In this case, you should validate the HTML using a tool such as the W3C HTML Validator, and you should also check the HTML source to ensure that all required elements are present.
User issue, as concluded on the Gatling community forum.
I am attempting to use the REST api of Jenkins. Jenkins requires a POST request to a URL to delete a job. This results in the following:
I tell my chosen Client to send a POST to the appropriate URL.
The client sends a POST and authorizes itself with username and password.
Jenkins deletes the job.
Jenkins returns a "302 - Found" with the location of folder containing the deleted job.
Client automatically sends a POST to the location.
Jenkins answers with "200 - OK" and the full HTML of the folder page.
This works just fine with Postman (unless I disable "Automatically follow redirects" of course).
Jersey however keeps running into a "404" at step 5 because I blocked anonymous users from viewing the folder in question. (Or a "403" if I blocked anonymous users altogether.)
Note that the authentication works in step 1 because the job has been deleted successfully!
I was under the impression that Jersey should use the given authentication for all requests concerning the client.
Is there a way to actually make this true? I really don't want to forbid redirects just to do every single redirect myself.
To clarify: The problem is that while Jersey follows the redirect, but fails to authenticate itself again, leading to the server rejecting the second request.
Code in question:
HttpAuthenticationFeature auth = HttpAuthenticationFeature.basicBuilder()
.credentials(username, token)
.build();
Client client = ClientBuilder.newBuilder()
.register(auth)
.build();
WebTarget deleteTarget = client.target("http://[Jenkins-IP]/job/RestTestingArea/job/testJob/doDelete")
Response response = deleteTarget.request()
.post(null);
EDIT: The "302-Found" only has 5 headers according to Postman: Date, X-Content-Type-Options ("nosniff"), Location, Content-Length (0) and Server. So neither any cookies nor any tokens that Postman might use and Jersey disregard.
Question loosely related to this one - if I were able to log the second request I might be able to understand what's happening behind the scenes.
EDIT2: I have also determined that the problem is clearly with the authentication. If I allow anonymous users to view the folder in question, the error disappears and the server answers with a 200.
I found the answer with the help of Paul Samsotha and Gautham.
TL;DR: This is intended behavior and you have to set the System property http.strictPostRedirect=true to make it work or perform the second request yourself.
As also described here, HttpURLConnection decided to not implement a redirect as it is defined in the HTTP standard but instead as many browsers implemented it (so in laymans terms, "Do it like everyone else instead of how it is supposed to work"). This leads to the following behavior:
Send POST to URL_1.
Server answers with a "302 - Found" and includes URL_2.
Send GET to URL_2, dropping all the headers.
Server answers with a "404 - Not Found" as the second request does not included correct authentication headers.
The "404" response is the one received by the code, as steps 2 and 3 are "hidden" by the underlying code.
By dropping all headers, the authentication fails. As Jersey uses this class by default, this lead to the behavior I was experiencing.
I got an eccentric problem. I am trying to automate visiting a web site by using WebRequest and WebClient. I have observed all the post request header key-value pairs and posted data string in Firebug(request Header and Post tab). Then I simulated such request with WebRequest and put all the header parameter and posted data there. However when I do GetResponse() from this request instance, I always got an error page back that says some sessionID is short of.
Actually I have taken care to put previously(first step to open the Logon page) responded session cookie in the Header's cookie field for the request. And I can get the correct response back by simulating requesting the logon page(the 1st page), but cannot get through this authentication page. My post data is like userid=John&password=123456789&domain=highmark.And the authentication page request that carried out by browser succeeds every time.
Am I missing something in the request that may not be shown by firebug.If yes, can you give me some recommendation for the tools that may examine the entire request sent by browser.
I have solved this issue. The problem is I set the httpWebRequest instance's AllowAutoRedirect=true. Thus the effect is when I got the first response from the server, the httpWebRequest would continually to make another request asking for a different url that is replied in the response header's Location field.
The defect of HttpWebRequest class is when it is getting redirected, it does not include the Set-Cookies(Response's Header Field)'s cookies in the next request header, thus the server would deny such page request and may redirect again to another different page.
And the httpWebRequest.GetResponse() method only return the last responsed page back under the setting AllowAutoRedirect=true. And I got the totally different response than I expected.
Also in this solving process, I need to thank to a distinguish Http Traffic examining tool:IEInspector Http Analyzer(http://www.ieinspector.com/httpanalyzer/). The great feature of this tool is it can examine not only the http traffic from browser but also what your process's httpWebRequest made. And also it can display in text format the raw stream of those request and response. Although it is a commercial software, you can try it for 15 days. I am quite happy with what it tells me(in well-formed details) and I like to buy it as well.
Our app makes a lot of HTTP requests and we are facing this issue with both inets-5.5.1 and 5.3.2.
Basically our receive clause for the response is trying to match the request id returned in httpc:request call
and we see that the Request Id match fails
We gave seen this mismatch in all 3 receive clauses for
stream_start, stream and stream_end
What we observed after a lot of trial and error is that if the same pid makes the http requests, the responses get kind of muddled up but if we spawn a separate process for the httpc:request, it is better . We also tried using a separate httpc profile to isolate the current process requests from other process requests. But even after this we have seen a lot of occurrences of this faulty behavior.
This is occurring with a lot of our http requests. Has anyone faced this ?
Thanks
Suma
This may not be a direct solution, but....
I suggest you try a much more "heavy duty" http client called ibrowse.
(if it is not too late for your project!)
Inets httpd and httpc are better for "simple HTTP tasks", so you may need Yaws, Mochiweb e.t.c if you need to do "duty grade" HTTP jobs instead of inets httpd
wish u success!