I have an application that was doing call to Azure Application Gateway and it was failing when the following rule was enforced:
RuleId: 200003
Description: Multipart Request Body Strict Validation
The call was to a PUT endpoint, passing 2 files in the body. Disabling this rule makes the call work.
Can't find a proper description about this rule. Just wondering if someone could provide more info about it? Is it safe to disable it?
I had this issue and MS support response was this is a bug and to disable the rule. See their response below.
I went through the WAF logs, and it seems we are hitting rule 200003 matching your request.
After investigation, there is basically a bug with this rule that is a new multipart formatting body check in 3.2. (it was only added a couple weeks ago).
What our expert says about is the following:
“This rule it's looking at the stream of the file and trying to determine if the formatting of the multipart request is valid. If it thinks it's not, the body check fails and the rest of the request can't be scanned. This particular rule causes a lot of false positives and should be disabled if that is the case here.
The 200002/200003/200004 family of rules are very difficult to workaround because they are looking at the formatting of the request, however, how the request is formatted is almost impossible to predict because the streaming of a file this way get encoded into a bunch of random chars, and if those chars break the typical formatting boundaries of a multipart request these rules will trigger. But.. since you can't predict how a file will get encoded during a stream, it's almost impossible to "fix" from a code /request level. Disable the rule the profit.“
Related
This question is maybe more a tip for people to search a solution if they have the same problem (as I found the solution eventually).
I had an application that does some HTTP requests with a local server (a mix of GET/POST with JSON content in the request/response bodies). The server is a third-party application, and after I upgraded it to a recent version, my Delphi app was no longer working.
It turned out that it was now hanging on the statement:
IdHTTP.Post("URL", "Payload", "BytesStreamResult");
As a manual POSTMAN request was still working, it had to be on the Delphi client side.
Further isolating the issue showed that the HTTP POST request did get an HTTP 200 response with valid HTTP response headers, but then was getting stuck reading the response body. It was hanging on:
IOHandler.ReadLn
When I compared the headers with the POSTMAN response, I noticed that 'Transfer-Encoding: chunked' was missing in the Delphi response.
Finally, I noticed the code related to TIdHTTP's hoKeepOrigProtocol option, which is not set by default.
So, my POST request was "downgraded" to an HTTP 1.0 request, and I guess this now made the (updated) server to respond differently (I'm not an RFC expert, but I guess 'chunked' is maybe an HTTP 1.1 option only).
After setting this option, everything worked like before (and indeed, the response was now read as "chunked" in Delphi).
Summary:
Shouldn't hoKeepOrigProtocol be the default option? (why punish good citizens for those that are not...)
Can we intercept this? Now my POST is assuming upfront a streamed response and thus it hangs because the server doesn't write anything to the buffer.
What would that high-level code look like? As it seems a mix of interpreting the header response headers and then deciding if more response reading is required.
(it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes...)
TIdHTTP supports non-chunked responses (which yes, is an HTTP 1.1 feature), so the hanging would have to be caused by the server sending a malformed response (a bug that should be reported to the server author).
When reading a non-chunked and non-MIME response, TIdHTTP does not use IOHandler.ReadLn to read the response's body, as you claim. Only when reading the response's headers.
But, since you did not show what the response actually looks like, nobody can explain for sure exactly why the hang occurs.
Shouldn't hoKeepOrigProtocol be the default option?
At the time the option was first introduced, no. There were plenty of buggy HTTP 1.1 servers around that downgrading to HTTP 1.0 was warranted.
However, that was many years ago. Nowadays, HTTP 1.1 is much more mature, and such buggy servers are rare. So, feel free to submit a change/pull request to Indy's GitHub repo if you feel the default behavior should be changed.
Can we intercept this?
No. The behavior you describe is most likely caused by a bug in the HTTP server. Either it is not sending all of the data it should be, or else the response is likely malformed in a way that makes TIdHTTP expect more data than is actually being sent. Either way, all you can do is assign a non-infinite timeout to TIdHTTP.
it didn't do anything specific regarding time-outs, either. I have the impression it hangs forever, or at least > 10 minutes.
Indy is designed to use infinite timeouts by default. You can assign custom timeouts to TIdHTTP's ConnectTimeout and ReadTimeout properties.
Setting this prevent the HTTP protocol downgrade:
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoKeepOrigProtocol];
This is, of course, dependant upon how the server processes the protocol specification, and if it results in issues or not.
I have a Jmeter script where during replay, Post request is displaying as Get request and the parameters in the request are not sent to the server. Due to this, correlations are failing at this request.
One of the parameters in the request is ViewState with so many characters. Is this large parameter value causing the above issue? How to proceed now?
Most probably you're sending a malformed request therefore instead of properly responding to a POST request you're being redirected somewhere (most probably to Login page)
Use View Results Tree listener in HTML or Browser mode to see what page you're hitting in the reality
With regards to the ViewState, "so many characters" is not a problem, the problem is that these are not random characters. ViewState is being used for client-side state management and if you fail to provide the proper value you won't be able to move further so you need to design your test as follows:
Open first page
Extract ViewState using a suitable Post-Processor
Open second page
here you need to pass viewstate from the step 1 along with other parameters
More information: ASP.NET Login Testing with JMeter
Also don't forget to add HTTP Cookie Manager to your Test Plan
What I'm able to understand is the request may be getting redirected. This happens usually when the server expects a unique request. If you recorded the request, you may be possibly using older headers that carry old cookie information. Check your headers and then reconstruct the request.
Make sure you are not using old cookies anywhere. remove that cookie part from HTTP Header Manager everywhere.
I'm working with a company on lead delivery, and they sent me some info regarding a Ping Post form setup. I've built hundreds of HTML forms processed by PHP (ie. sending an email/etc), but never something that would Ping a url, then return a value. The value it returns is XML.
Here's the purpose of the process:
I send a lead (form data) using the form with a particular zip code
This company parses that info, decides if it wants to "buy" it
Returns XML saying "Approved" or "Denied"
If "approved", I then post the data, and if "denied", I can do whatever I want
What is a common PHP method for doing this? I can research the code and put something together, just need to know what structure or PHP methods would work?
Thanks in advance.
You should be looking into RESTful Web Services.
here's a few examples that might help you
http://markroland.com/blog/restful-php-api/
http://coreymaynard.com/blog/creating-a-restful-api-with-php/
I did not create these examples, just what I found on Google.
I used file_get_contents(url) to handle the posting. The url contains inputs from the HTML form added as a query string, and the response is in XML which gets handled with simplexml_load_file().
As far as I understand your question what you need is to make an HTTP POST request and parse the incoming XML data.
I would rather not use file_get_contents() on remote servers - there are some potential security issues and it was missing some features the last time I checked. I strongly recommend cURL for remote HTTP/HTTPS communication.
Depending on the API you are posting to you might be able to use the SOAPclient class, but from the look of the response you got all you need is XML parser or Simple XML.
Anyway if you just need to check if a certain keyword (like Approved or Denied) is present you can use a simple string matching like this
if(strpos($response,'<STATUS>APPROVED</STATUS')!==false){
//approved
}
...
Wondering if I should ALWAYS use the respond_to/format.xxx block in ALL of my actions in ALL of my controllers. This came up because I realized that, for apps using only HTML response format (no respond_to block), I could send some other type of request (say XML) and get a valid response. Is this considered insecure? Feels to me like it is, and it seems that the solution would be to ALWAYS add the respond_to/format.xxx block to EVERY action on EVERY controller even if you are just using HTML only. That way, any non-html requests will get rejected (as they should).
So, is there ever a situation where it would be OK to leave out the respond_to/format.xxx bits for any given action? Even if you were just doing a simple, standard, HTML-only app? It seems like different versions of rails code generators have generated different things regarding this over time. Just wondering what is considered best practice, and when you can get away without using it.
And if the answer is "Yes, you should use it all the time in every action to be secure", then would it not be considered boilerplate at that point? And shouldn't it be pushed down into rails somehow so we don't have to write it out over and over again every single time?
I don't think this has any impact on the security of your application. Presuming that the inbound HTTP request is well-formed and the client is authenticated to make the request, you as the server don't really care what the client receives. In fact, the client could be passing in all sorts of headers and parameters that you totally ignore and it's up to the client to handle unexpected formats or errors. This is integral to the underlying architecture of HTTP.
In any case, you can set the routes up to not accept the .format parameter, in which case a request for .xml will result in a 404 or other type of error.
If you only use one format, don't the respond_to block, it clutters code with unnecessary artifacts.
I'm working a custom SpringSecurityFilter for my Grails application and I'm trying to use the commons upload library to process the request. I'm able to process the request in the filter but once it gets to my controller, none of the values are available.
Can the HttpRequest only be processed once by the upload library? I'm guessing it's cleaning up the temp files. Is there a way to keep them around so they can be processed again at the controller level?
I need to interrogate a form parameter for the security (due to the client I can't add it to the http headers) but once I get the value, it seems to wipe the request for further processing.
Yes. A Request can only be parsed once.
I saw this answer on Apache's FAQ page for FileUpload.
Question: Why is parseRequest() returning no items?
Answer: "This most commonly happens when the request has already been parsed, or processed in some other way. Since the input stream has aleady been consumed by that earlier process, it is no longer available for parsing by Commons FileUpload."
Reference: http://commons.apache.org/fileupload/faq.html