I am trying to call some urls with special characters in it. But it does not work.
This works:
GET .../rest/validation/checknameunique/?className=lomnido.Template&rename=true&name=Templaa%3Ea
This not: PUT ../rest/template/rename/526/Templaa%3Ea
There I get a 400 back from grails.
In the NGINX Log there is this entry
213.162.73.171 - - [22/Apr/2022:13:16:32 +0000] "PUT /rest/template/rename/28484/Bla%3Eaa HTTP/1.1" 400 2307 "https://mytest.com/configuration/template/28484" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"
When I debug this, the request does not reach the Security Interceptor (all requests go through this).
What is wrong here?
Best regards,
Peter
Related
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
Error after running the code - (failed 2 times): 503 Service Unavailable.
I tried many ways but failed. Please help me. Thanks in advance!
Check url by "curl" first. like,
curl -I "https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3"
then, you can see 503 response.
HTTP/2 503
In other words, your request is wrong.
you have to find proper request.
Chrome DevTools will help you. like
I think that user-agent ( like browser ) must be needed.
curl 'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36' \
--compressed
so... It may work,
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
user_agent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
I'm trying to ingest my AWS ALB logs into Splunk. After all, I could search my ALB logs in Splunk. But still the events are not properly parsing. Did anyone had similar issue or any suggestion?
Here is my prop.conf
[aws:alb:accesslogs]
SHOULD_LINEMERGE=false
FIELD_DELIMITER = whitespace
pulldown_type=true
FIELD_NAMES=type,timestamp,elb,client_ip,client_port,target,request_processing_time,target_processing_time,response_processing_time,elb_status_code,target_status_code,received_bytes,sent_bytes,request,user_agent,ssl_cipher,ssl_protocol,target_group_arn,trace_id
EXTRACT-elb = ^\s*(?P<type>[^\s]+)\s+(?P<timestamp>[^\s]+)\s+(?P<elb>[^\s]+)\s+(?P<client_ip>[0-9.]+):(?P<client_port>\d+)\s+(?P<target>[^\s]+)\s+(?P<request_processing_time>[^\s]+)\s+(?P<target_processing_time>[^\s]+)\s+(?P<response_processing_time>[^\s]+)\s+(?P<elb_status_code>[\d-]+)\s+(?P<target_status_code>[\d-]+)\s+(?P<received_bytes>\d+)\s+(?P<sent_bytes>\d+)\s+"(?P<request>.+)"\s+"(?P<user_agent>.+)"\s+(?P<ssl_cipher>[-\w]+)\s*(?P<ssl_protocol>[-\w\.]+)\s+(?P<target_group_arn>[^\s]+)\s+(?P<trace_id>[^\s]+)
EVAL-rtt = request_processing_time + target_processing_time + response_processing_time
Sample data
https 2020-08-20T12:40:00.274478Z app/my-aws-alb/e7538073dd1a6fd8 162.158.26.188:21098 172.0.51.37:80 0.000 0.004 0.000 405 405 974 424 "POST https://my-aws-alb-domain:443/api/ps/fpx/callback HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.2840.91 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:ap-southeast-1:111111111111:targetgroup/my-aws-target-group/41dbd234b301e3d84 "Root=1-5f3e6f20-3fdasdsfffdsf" "api.mydomain.com" "arn:aws:acm:ap-southeast-1:11111111111:certificate/be4344424-a40f-416e-8434c-88a8a3b072f5" 0 2020-08-20T12:40:00.270000Z "forward" "-" "-" "172.0.51.37:80" "405" "-" "-"
Using transforms is pretty straightforward. Start with a stanza in transforms.conf.
[elb]
REGEX = ^\s*(?P<type>[^\s]+)\s+(?P<timestamp>[^\s]+)\s+(?P<elb>[^\s]+)\s+(?P<client_ip>[0-9.]+):(?P<client_port>\d+)\s+(?P<target>[^\s]+)\s+(?P<request_processing_time>[^\s]+)\s+(?P<target_processing_time>[^\s]+)\s+(?P<response_processing_time>[^\s]+)\s+(?P<elb_status_code>[\d-]+)\s+(?P<target_status_code>[\d-]+)\s+(?P<received_bytes>\d+)\s+(?P<sent_bytes>\d+)\s+"(?P<request>.+)"\s+"(?P<user_agent>.+)"\s+(?P<ssl_cipher>[-\w]+)\s*(?P<ssl_protocol>[-\w\.]+)\s+(?P<target_group_arn>[^\s]+)\s+(?P<trace_id>[^\s]+)
Then refer to the transform in props.conf
[aws:alb:accesslogs]
TIME_PREFIX = https\s
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TRANSFORMS-elb = elb
EVAL-rtt = request_processing_time + target_processing_time + response_processing_time
We are using Nylas api to get Access token for different type of Email account like Gmail, Outlook.. But We couldn't authenticate for Gmail.
let myURL = URL(string: getNylasAuthUrl())
let userAgent = getUserAgentParams()
webView.customUserAgent = userAgent
let myRequest = URLRequest(url: myURL!)
webView.load(myRequest)
got below error
Finally found a way, by setting User-Agent, we could do authentication for gmail from post
Tried below User-agents but didn't help
let userAgent = "Mozilla/5.0 (Apple \(Utils.getDeviceModel()) ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
let userAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
let userAgent = "Mozilla/5.0 (Google) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
Finally, I found the working user-agent.
let userAgent = "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X)
AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14F89 Safari/602.1"
If you want to Google auth via Webview, use this user-agent especially for getting access token.
I have a situation where users are sending multiple parallel requests in a very short time frame, both GET and POST requests. We have a healthy mix of user agents, but only iOS clients are causing the duplication. It seems that the User Agent is non-standard. I tried to find browsers with similar user agents, and the only one I could find is Ghostery (which hides the browser/version), but I can't replicate the behaviour observed. Here's a list of user agent strings collected over the last week or so:
Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13F69
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14E304
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/603.2.4 (KHTML, like Gecko) Mobile/14F89
Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) Mobile/14E277
Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13B143
As mentioned, these can produce parallel requests, even including when the requests result in 302 redirects, each redirect is followed. We usually notice this behaviour because we'll get database integrity errors due to attempts to create users with duplicated usernames, obviously since one of the parallel requests is successful, and the other fails. One of the most complex examples would be this string of requests from our apache log:
[19/May/2017:12:18:01 +0000] "GET /product/1/ HTTP/1.0" 200
[19/May/2017:12:18:12 +0000] "POST /product/1/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /queue/buy/1862/ HTTP/1.0" 500
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /en/purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:12 +0000] "GET /purchasing/buy/ HTTP/1.0" 302
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
[19/May/2017:12:18:13 +0000] "GET /purchasing/orders/ HTTP/1.0" 200
The start of this flow would require the user to press a Buy button, which includes JS to disable the button on click, to reduce the chance of double-clicks. The form itself doesn't load without JS enabled, so we can be sure that JS is enabled on the browser.
Notice that the gap between the requests is virtually non-existent, and there are actually 4 parallel requests initially. One of them hits an HTTP 500 response, however 3 of the parallel requests return 302 and all appear to be followed all the way through to the final 200 response.
In this instance the user proceeded to complete their purchase legitimately, so I don't think any funny business ensued.
So I'm really unsure of how to handle this problem. I'm guessing I simply need to accept that this can happen with iOS clients. But I can't replicate the issue and I'm worried it might lead to a bad user experience in some cases.
Update
I updated the request list to include the initial entry point of the GET to load the form, and POST where it was submitted. Note only a single POST was received in this instance, yet, the redirect seems to have been followed 4 times.
Here's the HTML / JS involved in the initial submit:
HTML:
<form method="post" action="/product/1/" id="buy_form">
...
<button type="submit" id="buy_btn">Buy Now</button>
</form>
JS:
$("#buy_form").submit(function(){
$("#buy_btn").prop('disabled', true);
});
I have an access.log nginx with cookie:
99.20.231.22 www.carite.com - [01/Dec/2015:03:00:10 -0600] "GET /?mode=_ajax&_imod[]=i159330&make=Mercedes-Benz&_=1448960297171 HTTP/1.1" 200 1182 "http://www.carite.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" "PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; mod-compare-box=%7B%22vehicles%22%3A%7B%22v11279294%22%3A%7B%22vuid%22%3A%2211279294%22%2C%22isCompared%22%3Afalse%7D%7D%2C%22compareAll%22%3Atrue%2C%22cookieLifeTime%22%3A30%2C%22cookiePath%22%3A%22%5C%2F%22%7D; _ga=GA1.2.10339867.1448960182; _gali=make; _gat_a1=1; _gat_a2=1; _gat_a3=1; _gat_a4=1; usy46gabsosd=collserve__-2_1448960382693_8786" 80 0.295
Can I specify Yandex-tank get cookie from access log and add it to every yandex-tank request?
Also I need get header "Host:" from access log instead of specify it in load.ini like:
headers = [Host: www.carite.com]
You have two options:
to make stepper read cookies along with uri from access.log (it
should be done around there
https://github.com/yandex/yandex-tank/blob/master/yandextank/stepper/missile.py#L213)
make a separate file from access.log, in https://yandextank.readthedocs.org/en/latest/tutorial.html#uri-style-uris-in-file format. Headers are overriden on the go, so you can redefine headers anywhere
For example it could be like this:
[Host: www.carite.com]
[Cookie: PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; ...]
/?mode=_ajax&imod[]=i159330&make=Mercedes-Benz&=1448960297171
...
[Host: example.com]
[Cookie: myowncookie=1]
/something
...
I would advice to use the 2nd way as an easiest one