I have a situation where users are sending multiple parallel requests in a very short time frame, both GET and POST requests. We have a healthy mix of user agents, but only iOS clients are causing the duplication. It seems that the User Agent is non-standard. I tried to find browsers with similar user agents, and the only one I could find is Ghostery (which hides the browser/version), but I can't replicate the behaviour observed. Here's a list of user agent strings collected over the last week or so:
Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13F69
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14E304
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Mobile/14F89
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14E277
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13B143
As mentioned, these can produce parallel requests, even including when the requests result in 302 redirects, each redirect is followed. We usually notice this behaviour because we'll get database integrity errors due to attempts to create users with duplicated usernames, obviously since one of the parallel requests is successful, and the other fails. One of the most complex examples would be this string of requests from our apache log:
[19/May/2017:12:18:01 +0000] "GET /product/1/ HTTP/1.0" 200
[19/May/2017:12:18:12 +0000] "POST /product/1/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 500
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
The start of this flow would require the user to press a Buy button, which includes JS to disable the button on click, to reduce the chance of double-clicks. The form itself doesn't load without JS enabled, so we can be sure that JS is enabled on the browser.
Notice that the gap between the requests is virtually non-existent, and there are actually 4 parallel requests initially. One of them hits an HTTP 500 response, however 3 of the parallel requests return 302 and all appear to be followed all the way through to the final 200 response.
In this instance the user proceeded to complete their purchase legitimately, so I don't think any funny business ensued.
So I'm really unsure of how to handle this problem. I'm guessing I simply need to accept that this can happen with iOS clients. But I can't replicate the issue and I'm worried it might lead to a bad user experience in some cases.
Update
I updated the request list to include the initial entry point of the GET to load the form, and POST where it was submitted. Note only a single POST was received in this instance, yet, the redirect seems to have been followed 4 times.
Here's the HTML / JS involved in the initial submit:
HTML:
<form method="post" action="/product/1/" id="buy_form">
...
<button type="submit" id="buy_btn">Buy Now</button>
</form>
JS:
$("#buy_form").submit(function(){
$("#buy_btn").prop('disabled', true);
});
Related
I am trying to call some urls with special characters in it. But it does not work.
This works:
GET .../rest/validation/checknameunique/?className=lomnido.Template&rename=true&name=Templaa%3Ea
This not: PUT ../rest/template/rename/526/Templaa%3Ea
There I get a 400 back from grails.
In the NGINX Log there is this entry
213.162.73.171 - - [22/Apr/2022:13:16:32 +0000] "PUT /rest/template/rename/28484/Bla%3Eaa HTTP/1.1" 400 2307 "https://mytest.com/configuration/template/28484" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"
When I debug this, the request does not reach the Security Interceptor (all requests go through this).
What is wrong here?
Best regards,
Peter
I'd like to parsing my apache access log, but I guess that there are something wrong in my grok pattern.
Following logs are what I want to analyze.
"message": "221.251.246.139 - - [14/Sep/2020:04:56:04 +0000] \"POST /services/api/agent/liveview_image_upload HTTP/1.1\" 200 45 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko\" 6019 234 1618 \"-\""
And here's what I've been parsing using that message.
%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})) %{NUMBER:response} (?:%{NUMBER:bytes})
According to my parsing process, It can parse up to \"-\". I want to parsing rest of my messages.
How can I parsing the rest? And are the things I've done so far correct?
I have an access.log nginx with cookie:
99.20.231.22 www.carite.com - [01/Dec/2015:03:00:10 -0600] "GET /?mode=_ajax&_imod[]=i159330&make=Mercedes-Benz&_=1448960297171 HTTP/1.1" 200 1182 "http://www.carite.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" "PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; mod-compare-box=%7B%22vehicles%22%3A%7B%22v11279294%22%3A%7B%22vuid%22%3A%2211279294%22%2C%22isCompared%22%3Afalse%7D%7D%2C%22compareAll%22%3Atrue%2C%22cookieLifeTime%22%3A30%2C%22cookiePath%22%3A%22%5C%2F%22%7D; _ga=GA1.2.10339867.1448960182; _gali=make; _gat_a1=1; _gat_a2=1; _gat_a3=1; _gat_a4=1; usy46gabsosd=collserve__-2_1448960382693_8786" 80 0.295
Can I specify Yandex-tank get cookie from access log and add it to every yandex-tank request?
Also I need get header "Host:" from access log instead of specify it in load.ini like:
headers = [Host: www.carite.com]
You have two options:
to make stepper read cookies along with uri from access.log (it
should be done around there
https://github.com/yandex/yandex-tank/blob/master/yandextank/stepper/missile.py#L213)
make a separate file from access.log, in https://yandextank.readthedocs.org/en/latest/tutorial.html#uri-style-uris-in-file format. Headers are overriden on the go, so you can redefine headers anywhere
For example it could be like this:
[Host: www.carite.com]
[Cookie: PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; ...]
/?mode=_ajax&imod[]=i159330&make=Mercedes-Benz&=1448960297171
...
[Host: example.com]
[Cookie: myowncookie=1]
/something
...
I would advice to use the 2nd way as an easiest one
Trying to set up a simple Dart HttpServer:
import 'dart:io';
void main() {
HttpServer.bind(InternetAddress.ANY_IP_V4, 80).then((server) {
server.listen((HttpRequest request) {
request.response.write('Hello, world.');
request.response.close();
print(new DateTime.now());
print(request.connectionInfo.remoteAddress);
print(request.method);
print(request.headers.toString());
print("--------------");
});
});
print("listing....");
}
When hitting localhost from a browser (Chrome), it appears as if the incoming request is handled twice:
listing....
2013-11-07 15:19:24.478
InternetAddress('127.0.0.1', IP_V4)
GET
host: localhost:80
connection: keep-alive
cache-control: max-age=0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
accept-encoding: gzip,deflate,sdch
accept-language: en-US,en;q=0.8
--------------
2013-11-07 15:19:24.554
InternetAddress('127.0.0.1', IP_V4)
GET
host: localhost:80
connection: keep-alive
accept: */*
user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
accept-encoding: gzip,deflate,sdch
accept-language: en-US,en;q=0.8
--------------
Those two requests look almost identical, except for the accept header. Doesn't look like the browser is firing up the request twice:
So, why does the request get handled twice?
EDIT: Dart SDK version 0.8.10.6_r30036
You don't output what is requested (the url member of the request instance) and that is the difference between the two requests.
The first request requests the file you try to open, probably /. The second request is issued internally by the browser and requests favicon.ico to display the icon in the address bar / tab title.
I've been checking my production.log today and there's a number of requests hitting my site that appear to be malicious, but I'm confused as to how they're even getting to us.
For example:
Processing PublicController#unknown_request (for 217.23.4.13 at 2009-11-09 09:15:52) [GET]
Parameters: {"anything"=>["results.aspx"], "action"=>"unknown_request", "first"=>"200", "controller"=>"public", "q"=>"\"bbs/cbbs.cgi?\" intitle:\"Book\" intext:\"2008\" site:.uz ", "count"=>"200", "FORM"=>"PERE"}
Completed in 16ms (View: 12, DB: 0) | 200 OK [http : // search . live .com /results.aspx?q=%22bbs/cbbs.cgi%3F%22%20intitle%3A%22Book%22%20intext%3A%222008%22%20site%3A.uz%20&count=200&first=200&FORM=PERE]
These are happening every 30 seconds or so. Obviously, PublicController/Unknown_request is my controller/action 404 error.
The access log shows these requests as:
217.23.4.13 - - [09/Nov/2009:09:57:25 +1000] "GET http://search.live.com/results.aspx?q=%22en-gb.html%22%20intitle%3A%22Home%22%20intext%3A%222006%22%20site%3A.mn%20&count=200&first=400&FORM=PERE HTTP/1.1" 200 3626 "-" "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.1$
How are these requests even hitting my site? Does anyone have any ideas?
I think this might be the same problem you're having: http://penguinpetes.com/b2evo/index.php?p=567&more=1&c=1&tb=1&pb=1
Basically, live/bing are doing some sort of testing that involves going to your site looking like someone searched something completely irrelevant to the content you have.