I am trying to lookup a zipcode from usps using curl like so:
curl "https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=11111&zip="
This url does return normal output from browser.
One needs to specify the User-Agent header (using the -A option) so that the request resembles a request from a browser. Also, due to the presence of ! in the URL, one has to use ' instead of ":
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0" \
'https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=10001&zip='
Related
I am trying to trying to get the packets which contain application layer payloads like HTTP from a given pcap file.
I have tried using http in the Wireshark display filter. My doubt is that, is it the right way to get the http payload from the pcap file. Please help me on this.
Saving HTTP packets
To filter for http traffic in tshark, you would use a display filter (-Y). This is sample output showing what that would look like:
$ tshark -r input.pcap -Y http
25 1.051399 10.8.143.109 → server-13-35-127-122.sfo5.r.cloudfront.net HTTP
630 GET /online HTTP/1.1 0c:8d:db:90:cf:38 ← 6c:96:cf:d8:7f:e7
34 1.078368 server-13-35-127-122.sfo5.r.cloudfront.net → 10.8.143.109 HTTP
404 HTTP/1.1 304 Not Modified 6c:96:cf:d8:7f:e7 ← 0c:8d:db:90:cf:38
This shows them output as text (the default). To output them to a new file, use the -w flag:
$ tshark -r input.pcap -Y http -w modified.pcap
Export files
You can also export certain types of plaintext objects from tshark
$ output_folder="files"
$ tshark -r input.pcap --export-object http,$output_folder
$ ls $output_folder
example.png example.html ...
This article will walk you through generating a packet capture from which you can then export HTTP files.
I'm currently trying to setup a standalone rails 6.0.0 API fresh install (rails new xxx --api) + sidekiq (that includes Rack::Protection)
After setting-up Rack::Cors, i get the following 403 error on any non-GET request :
WARN -- : attack prevented by Rack::Protection::AuthenticityToken
To my understanding this is related to CSRF tokens, which don't make sense for a json API.
Here is a copy of the request I'm doing :
curl 'http://127.0.0.1:5100/searches' -H 'Accept: application/json,
text/plain, /' -H 'Referer: http://localhost:8080/' -H 'Origin:
http://localhost:8080' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/76.0.3809.132 Safari/537.36' -H 'Sec-Fetch-Mode: cors' -H
'Content-Type: application/x-www-form-urlencoded' --data '{query:
'test'}' --compressed
The 2 possible solutions that I cannot solve :
disable rack-protection
get a token in a controller (ActionController::API) to pass to the requests
In ApplicationController try protect_from_forgery with: :null_session, or
Rails.configuration.middleware.insert_before 0, Rack::Cors do
allow do
origins "*"
end
end
in config/initializers/rack-cors.rb
Before Chrome makes a cross-domain AJAX call it makes an OPTIONS check like this:
curl \
'https://fubar.com/users/sign_in' \
-X OPTIONS \
-H 'Access-Control-Request-Method: POST' \
-H 'Origin: http://snafu.com' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' \
-H 'Access-Control-Request-Headers: content-type' \
--compressed \
--insecure \
--verbose
(I added --insecure and --verbose for testing.)
I can see this request in the Apache logs but it doesn't get to Rails.
127.0.0.1 - - [27/Jul/2018:09:22:44 -0400] "OPTIONS /users/sign_in HTTP/1.1" 200 -
If I remove either the Access-Control-Request-Method or Origin headers then it does pass the request to Rails.
Something about the combination of these two headers seems to be causing Apache to handle the request itself and not give Rails a chance to process it.
I am not setting any headers or defining any rewrite rules in the Apache config; it's basically a vanilla install.
I'm not able to find any documentation or configurations explaining why this would be happening and how to prevent it.
1) OPTIONS HTTP call at 'https://fubar.com/users/sign_in' performed from chrome browser with request headers 'Access-Control-Request-Method: POST' and 'Origin: http://snafu.com'
2) the server at 'https://fubar.com/users/sign_in' receives the request
STEP 1
The routing is handles from either nginx or apache webserver which will first apply their own config rules. This settings are included in the file inside /etc/apache2 or /etc/nginx
For example with nginx you can define rules to add_header to the http response or settings to redirect to another url
for example
add_header "Access-Control-Allow-Origin: '*'"
this will add the header "Access-Control-Allow-Origin: '*'" to all responses.
If for example you apply this response to all OPTIONS requests, then all subsequent http request will be whitelisted from any http origin
STEP 2
once the nginx/apache redirection rules are applied, the rails router receives the request and redirects to your controller.
Here you can still add any header you want inside the controller, you can redirect OPTIONS request to a specific controller action, which can add a specific header, be careful to not add twice the same header as that can cause issues.
In this action you can rewrite the Access-Control-Allow-Origin header in the response from the OPTIONS request to whitelist only specific origin domains (you just need to write a routing rule which is applied only to OPTIONS requests)
The origin domain is written in the header of the request
request headers 'Access-Control-Request-Method: POST' and 'Origin: http://snafu.com'
Using HTTParty, I'm able to read some pages fine, other on the same site strangely give a 404 unless I set the proper session headers. So I'm trying to get and set them via HTTParty.
This works:
HTTParty.get 'https://www.instagram.com/explore/locations/24993086/pfriem-family-brewers/'
This gives a 404:
HTTParty.get 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
Curl also gives a 404 for that:
url -I https://www.instagram.com/explore/locations/295648950/trio-salon-spa/
...unless I set all the headers that Chrome sends:
curl -I 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/' -H 'pragma: no-cache' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9,da;q=0.8,fr;q=0.7' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: no-cache' -H 'authority: www.instagram.com' -H 'cookie: shbid=15682; csrftoken=7FQ5Z0SZikfkYiS02bcTRotjAYEvdooD; mid=Wk7bTgAEAAHaOKTW39cyGMXo8vLj; ds_user_id=2055054912; mcd=3; sessionid=IGSC07d4796576bd227dda2a5353ee0365bbd6a6f6b2da7567d57a83cde58c0ae870%3AOvd5knFMpZUQaZpRyr0QkeMitWBFnGDP%3A%7B%22_auth_user_id%22%3A2055054912%2C%22_auth_user_backend%22%3A%22accounts.backends.CaseInsensitiveModelBackend%22%2C%22_auth_user_hash%22%3A%22%22%2C%22_platform%22%3A4%2C%22_token_ver%22%3A2%2C%22_token%22%3A%222055054912%3AhdEI59s33u2BM3M2f8p2ZkSkZ9HeZR5Y%3A7359e774fd121f9726db15a24e660d43a0f464b4733bbbaa31f619aec3f433ba%22%2C%22last_refreshed%22%3A1524728608.4418663979%7D; rur=FTW; fbm_124024574287414=base_domain=.instagram.com; fbsr_124024574287414=9pBDQeojfCbPmhlXZHwx_OGhduHFlQusvBdewwiZDY4.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImNvZGUiOiJBUUJVTEZmWHBfYS1mdVYyMmhKZEhnV3Bvb3dKVXppV3oxVVhxcDBGelFnTmpnNmpRZjNDY1ZZV2xybWZYSE5JV0dxeVUtNE9UaDBFUUdMbFJFWVVwZzZPYngxYmdxbUxHLV9pVGl5U3hGa1JxbGRJRExITHV5V09WVEVsbjlHUFhsTmRCQUNheUdYMmh4ZWcxajJEcERwczZ2X282aWRUcWd6UmYxaHE2VjVObEFGX0w5bFM1RXI5bHg3b2c1bWk0ak9OdmVCcVpLTG5nY1llM0NnWHVtOWdWQWJTVi1SdWpQU2J1UmhHNFdaS0xweWtPVEYzdmhsUlVaT2FLZ3FXcFo2TXFXY2xqSTM0T3JWZjR3dzFyY3J0S0RkdVh0Qk5zellKRG9weU1IdG1kVERUVGVrZmwyUHhpRzZsSmRkUVpSbWc0MTNMWERHQ0ZORDRVRS13OUhvMyIsImlzc3VlZF9hdCI6MTUyNDc1ODM3NCwidXNlcl9pZCI6IjExNTU2MzM2MjkifQ; urlgen="{\"time\": 1524728608\054 \"65.157.26.82\": 209}:1fBjJI:P27GLkp5R5uijWSi-SEkHW-Mo0c"' --compressed
So, I'm trying to do a basic get with HTTP, read the session, and make another request with it, but it's not working properly:
require 'httparty'
url = 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
get_response = HTTParty.get(url)
cookie_hash = CookieHash.new
get_response.get_fields('Set-Cookie').each { |c| cookie_hash.add_cookies(c) }
get_response_cookie = parse_cookie(get_response.headers['Set-Cookie'])
post_response = HTTParty.get( url, headers: {'Cookie' => cookie_hash.to_cookie_string } )
How can I get this to work?
I'm attempting to gather XML data automatically using curl, and my command so far is
curl -E keyStore.pem -d 'uid=myusername&password=mypassword&active=y&type=F' 'https://www.aidaptest.naimes.faa.gov/aidap/XmlNotamServlet HTTP/1.1/' -k
but it keeps on giving me a "Your browser sent a query this server could not understand." error.
I'm pretty sure that it's connecting since it's not rejecting me, but I don't know how to properly format the POST. Here's some documentation they gave me for the format of the POST request.
POST <URL>/aidap/XmlNotamServlet HTTP/1.1
Content-type: application/x-www-form-urlencoded
Content-length: <input_parameter’s length>
<a blank line>
<input_parameter>`
input_parameter is uid, password, and location_id bit and is correct
Am I doing this correctly from what you can see?
Something like this should do it.
curl -E keyStore.pem -v -X POST -i --header Content-Type:application/x-www-form-urlencoded
-d 'uid=myusername&password=mypassword&active=y&type=F' 'https://www.aidaptest.naimes.faa.gov/aidap/XmlNotamServlet'
I don't think you need HTTP/1.1 at the end of the URL in the command line. And even if you needed it, you certainly don't need the final / character before the closing single quote.