I have an access.log nginx with cookie:
99.20.231.22 www.carite.com - [01/Dec/2015:03:00:10 -0600] "GET /?mode=_ajax&_imod[]=i159330&make=Mercedes-Benz&_=1448960297171 HTTP/1.1" 200 1182 "http://www.carite.com/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1" "PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; mod-compare-box=%7B%22vehicles%22%3A%7B%22v11279294%22%3A%7B%22vuid%22%3A%2211279294%22%2C%22isCompared%22%3Afalse%7D%7D%2C%22compareAll%22%3Atrue%2C%22cookieLifeTime%22%3A30%2C%22cookiePath%22%3A%22%5C%2F%22%7D; _ga=GA1.2.10339867.1448960182; _gali=make; _gat_a1=1; _gat_a2=1; _gat_a3=1; _gat_a4=1; usy46gabsosd=collserve__-2_1448960382693_8786" 80 0.295
Can I specify Yandex-tank get cookie from access log and add it to every yandex-tank request?
Also I need get header "Host:" from access log instead of specify it in load.ini like:
headers = [Host: www.carite.com]
You have two options:
to make stepper read cookies along with uri from access.log (it
should be done around there
https://github.com/yandex/yandex-tank/blob/master/yandextank/stepper/missile.py#L213)
make a separate file from access.log, in https://yandextank.readthedocs.org/en/latest/tutorial.html#uri-style-uris-in-file format. Headers are overriden on the go, so you can redefine headers anywhere
For example it could be like this:
[Host: www.carite.com]
[Cookie: PHPSESSID=ebg5n89m9pc1iamekii1qra5k0; chooseStoreNotificationShown=1; dfa_visit=1448960180633603603; dfa_visitor=1448960180633796491; ...]
/?mode=_ajax&imod[]=i159330&make=Mercedes-Benz&=1448960297171
...
[Host: example.com]
[Cookie: myowncookie=1]
/something
...
I would advice to use the 2nd way as an easiest one
Related
I am trying to use Microsft-Clarity https://clarity.microsoft.com/ in my Electron App. I have added their integration steps in Head section of my app's main page. On running the app, in network section it is shoiwng network call to https://d/clarity.ms/collect but it is giving 400 bad request error in the response.
I am using Electron version 13.x
Is it possible to use Clarity in Electron app or not? Or do I need to make any specific changes to support it?
I have added the following code in head section of page
<script type="text/javascript">
(function(c,l,a,r,i,t,y){
c[a]=c[a]||function(){(c[a].q=c[a].q||[]).push(arguments)};
t=l.createElement(r);t.async=1;t.src="https://www.clarity.ms/tag/"+i;
y=l.getElementsByTagName(r)[0];y.parentNode.insertBefore(t,y);
})(window, document, "clarity", "script", "<Project_id>");
</script>
POST request is being made for the following URL:
https://d/clarity.ms/collect
Request header has following value
authority: d.clarity.ms
method: POST
path: /collect
scheme: https
accept: application/x-clarity-gzip
accept-encoding: gzip, deflate, br
accept-language: en-US
content-length: 285595
cookie: MUID=XXXXXXXXXXXXXXXXXXXXX
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: cross-site
user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) XXXXXX/XXXXX Chrome/91.0.4472.164 Electron/13.6.9 Safari/537.36
Getting the following response for the request
{
"type":"https://tools.ietf.org/html/rfc7231#section-6.5.1",
"title":"Bad Request",
"status":400,
"traceId":"00-xxxxxxxxxxxxxxx-xxxxxxxxxxxx-00"
}
I am trying to call some urls with special characters in it. But it does not work.
This works:
GET .../rest/validation/checknameunique/?className=lomnido.Template&rename=true&name=Templaa%3Ea
This not: PUT ../rest/template/rename/526/Templaa%3Ea
There I get a 400 back from grails.
In the NGINX Log there is this entry
213.162.73.171 - - [22/Apr/2022:13:16:32 +0000] "PUT /rest/template/rename/28484/Bla%3Eaa HTTP/1.1" 400 2307 "https://mytest.com/configuration/template/28484" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36"
When I debug this, the request does not reach the Security Interceptor (all requests go through this).
What is wrong here?
Best regards,
Peter
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
Error after running the code - (failed 2 times): 503 Service Unavailable.
I tried many ways but failed. Please help me. Thanks in advance!
Check url by "curl" first. like,
curl -I "https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3"
then, you can see 503 response.
HTTP/2 503
In other words, your request is wrong.
you have to find proper request.
Chrome DevTools will help you. like
I think that user-agent ( like browser ) must be needed.
curl 'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36' \
--compressed
so... It may work,
import scrapy
class oneplus_spider(scrapy.Spider):
name='one_plus'
page_number=0
user_agent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
start_urls=[
'https://www.amazon.com/s?k=samsung+mobile&page=3&qid=1600763713&ref=sr_pg_3'
]
def parse(self,response):
all_links=[]
total_links=[]
domain='https://www.amazon.com'
href=[]
link_set=set()
href=response.css('a.a-link-normal.a-text-normal').xpath('#href').extract()
for x in href:
link_set.add(domain+x)
for x in link_set:
next_page=x
yield response.follow(next_page, callback=self.parse_page1)
def parse_page1(self, response):
title=response.css('span.a-size-large product-title-word-break::text').extract()
print(title)
I'm using workbox with workbox-webpack-plugin to cache some assets.
My current config is supposed to cache two files: a .js and a .css.
Both files are being cached properly, but the problem is that the browser still downloads them from the network, and I have no idea why.
Here's the workbox plugin in my webpack config that generates the service worker:
new GenerateSW({
swDest: 'service-worker.js',
importWorkboxFrom: 'local',
chunks: ['myChunk'],
skipWaiting: true,
clientsClaim: true,
ignoreUrlParametersMatching: [/.*/],
cacheId: 'myCacheId',
}),
Here's the generated service worker:
/**
* Welcome to your Workbox-powered service worker!
*
* You'll need to register this file in your web app and you should
* disable HTTP caching for this file too.
* See https://xxx
*
* The rest of the code is auto-generated. Please don't update this file
* directly; instead, make changes to your Workbox build configuration
* and re-run your build process.
* See https://xxx
*/
importScripts("workbox-v3.6.3/workbox-sw.js");
workbox.setConfig({modulePathPrefix: "workbox-v3.6.3"});
importScripts(
"precache-manifest.14645da973669ef1d2247d1863e806bd.js"
);
workbox.core.setCacheNameDetails({prefix: "myCacheId"});
workbox.skipWaiting();
workbox.clientsClaim();
/**
* The workboxSW.precacheAndRoute() method efficiently caches and responds to
* requests for URLs in the manifest.
* See https://xxx
*/
self.__precacheManifest = [].concat(self.__precacheManifest || []);
workbox.precaching.suppressWarnings();
workbox.precaching.precacheAndRoute(self.__precacheManifest, {
"ignoreUrlParametersMatching": [/.*/]
});
And precache manifest:
self.__precacheManifest = [
{
"revision": "9b22d66a17276ac21d45",
"url": "myChunk.js"
},
{
"revision": "9b22d66a17276ac21d45",
"url": "myChunk.css"
}
];
The precaching actually works, but it looks like the fetch event is not intercepted by the service worker. If I try to download the files directly from the address bar in Chrome, the file is loaded correctly from the service worker. But when it's loaded from the script tag of my page, it's still being downloaded from the network.
Here are the request header when loaded from the script tag:
GET /assets/myChunk.js?1546600720154 HTTP/1.1
Host: localhost:5000
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: */*
Referer: http://localhost:5000/xxx
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,fr-BE;q=0.8,fr;q=0.7,te-IN;q=0.6,te;q=0.5
Cookie: visitor_id=f86c312d-76e2-468d-a5c5-45c47fa3bbdc
Any help would be great!
As per the HTTP traffic snippet you've posted, your <script> tag results in a request for /assets/myChunk.js?1546600720154. The 1546600720154 query parameter bit is causing a mismatch when attempting to match that request against the precached URLs.
I'd suggest one of two things:
Configure webpack to add a content hash as part of your URLs, and use that in favor of the time-based URL parameters for cache-busting. Workbox should be able to just read those hashed URLs as-is.
Continue using the time-based URL query parameter for cache-busting, but configure Workbox to ignore those parameters when determining whether there's a match against the precached URLs. You can do that by using the ignoreUrlParametersMatching option. It takes an array of RegExps, and something like ignoreUrlParametersMatching: [/^\d+$/] would be enough to tell Workbox to ignore any query parameters that consist entirely of numbers.
I'd probably go with option 1 if you could.
There are several ways to change the user agent header send by UIWebView. However the value of window.navigator.userAgent retrieved by JS on any page always contains the same value, something like "Mozilla/5.0 (iPad; U; CPU OS 4_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Mobile/8C134".
Is there any way to change that value without having to modify the HTML response (and insert some JS)?
Thanks in advance!