Using HTTParty, I'm able to read some pages fine, other on the same site strangely give a 404 unless I set the proper session headers. So I'm trying to get and set them via HTTParty.
This works:
HTTParty.get 'https://www.instagram.com/explore/locations/24993086/pfriem-family-brewers/'
This gives a 404:
HTTParty.get 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
Curl also gives a 404 for that:
url -I https://www.instagram.com/explore/locations/295648950/trio-salon-spa/
...unless I set all the headers that Chrome sends:
curl -I 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/' -H 'pragma: no-cache' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9,da;q=0.8,fr;q=0.7' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: no-cache' -H 'authority: www.instagram.com' -H 'cookie: shbid=15682; csrftoken=7FQ5Z0SZikfkYiS02bcTRotjAYEvdooD; mid=Wk7bTgAEAAHaOKTW39cyGMXo8vLj; ds_user_id=2055054912; mcd=3; sessionid=IGSC07d4796576bd227dda2a5353ee0365bbd6a6f6b2da7567d57a83cde58c0ae870%3AOvd5knFMpZUQaZpRyr0QkeMitWBFnGDP%3A%7B%22_auth_user_id%22%3A2055054912%2C%22_auth_user_backend%22%3A%22accounts.backends.CaseInsensitiveModelBackend%22%2C%22_auth_user_hash%22%3A%22%22%2C%22_platform%22%3A4%2C%22_token_ver%22%3A2%2C%22_token%22%3A%222055054912%3AhdEI59s33u2BM3M2f8p2ZkSkZ9HeZR5Y%3A7359e774fd121f9726db15a24e660d43a0f464b4733bbbaa31f619aec3f433ba%22%2C%22last_refreshed%22%3A1524728608.4418663979%7D; rur=FTW; fbm_124024574287414=base_domain=.instagram.com; fbsr_124024574287414=9pBDQeojfCbPmhlXZHwx_OGhduHFlQusvBdewwiZDY4.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImNvZGUiOiJBUUJVTEZmWHBfYS1mdVYyMmhKZEhnV3Bvb3dKVXppV3oxVVhxcDBGelFnTmpnNmpRZjNDY1ZZV2xybWZYSE5JV0dxeVUtNE9UaDBFUUdMbFJFWVVwZzZPYngxYmdxbUxHLV9pVGl5U3hGa1JxbGRJRExITHV5V09WVEVsbjlHUFhsTmRCQUNheUdYMmh4ZWcxajJEcERwczZ2X282aWRUcWd6UmYxaHE2VjVObEFGX0w5bFM1RXI5bHg3b2c1bWk0ak9OdmVCcVpLTG5nY1llM0NnWHVtOWdWQWJTVi1SdWpQU2J1UmhHNFdaS0xweWtPVEYzdmhsUlVaT2FLZ3FXcFo2TXFXY2xqSTM0T3JWZjR3dzFyY3J0S0RkdVh0Qk5zellKRG9weU1IdG1kVERUVGVrZmwyUHhpRzZsSmRkUVpSbWc0MTNMWERHQ0ZORDRVRS13OUhvMyIsImlzc3VlZF9hdCI6MTUyNDc1ODM3NCwidXNlcl9pZCI6IjExNTU2MzM2MjkifQ; urlgen="{\"time\": 1524728608\054 \"65.157.26.82\": 209}:1fBjJI:P27GLkp5R5uijWSi-SEkHW-Mo0c"' --compressed
So, I'm trying to do a basic get with HTTP, read the session, and make another request with it, but it's not working properly:
require 'httparty'
url = 'https://www.instagram.com/explore/locations/295648950/trio-salon-spa/'
get_response = HTTParty.get(url)
cookie_hash = CookieHash.new
get_response.get_fields('Set-Cookie').each { |c| cookie_hash.add_cookies(c) }
get_response_cookie = parse_cookie(get_response.headers['Set-Cookie'])
post_response = HTTParty.get( url, headers: {'Cookie' => cookie_hash.to_cookie_string } )
How can I get this to work?
Related
I'm currently trying to setup a standalone rails 6.0.0 API fresh install (rails new xxx --api) + sidekiq (that includes Rack::Protection)
After setting-up Rack::Cors, i get the following 403 error on any non-GET request :
WARN -- : attack prevented by Rack::Protection::AuthenticityToken
To my understanding this is related to CSRF tokens, which don't make sense for a json API.
Here is a copy of the request I'm doing :
curl 'http://127.0.0.1:5100/searches' -H 'Accept: application/json,
text/plain, /' -H 'Referer: http://localhost:8080/' -H 'Origin:
http://localhost:8080' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/76.0.3809.132 Safari/537.36' -H 'Sec-Fetch-Mode: cors' -H
'Content-Type: application/x-www-form-urlencoded' --data '{query:
'test'}' --compressed
The 2 possible solutions that I cannot solve :
disable rack-protection
get a token in a controller (ActionController::API) to pass to the requests
In ApplicationController try protect_from_forgery with: :null_session, or
Rails.configuration.middleware.insert_before 0, Rack::Cors do
allow do
origins "*"
end
end
in config/initializers/rack-cors.rb
Before Chrome makes a cross-domain AJAX call it makes an OPTIONS check like this:
curl \
'https://fubar.com/users/sign_in' \
-X OPTIONS \
-H 'Access-Control-Request-Method: POST' \
-H 'Origin: http://snafu.com' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' \
-H 'Access-Control-Request-Headers: content-type' \
--compressed \
--insecure \
--verbose
(I added --insecure and --verbose for testing.)
I can see this request in the Apache logs but it doesn't get to Rails.
127.0.0.1 - - [27/Jul/2018:09:22:44 -0400] "OPTIONS /users/sign_in HTTP/1.1" 200 -
If I remove either the Access-Control-Request-Method or Origin headers then it does pass the request to Rails.
Something about the combination of these two headers seems to be causing Apache to handle the request itself and not give Rails a chance to process it.
I am not setting any headers or defining any rewrite rules in the Apache config; it's basically a vanilla install.
I'm not able to find any documentation or configurations explaining why this would be happening and how to prevent it.
1) OPTIONS HTTP call at 'https://fubar.com/users/sign_in' performed from chrome browser with request headers 'Access-Control-Request-Method: POST' and 'Origin: http://snafu.com'
2) the server at 'https://fubar.com/users/sign_in' receives the request
STEP 1
The routing is handles from either nginx or apache webserver which will first apply their own config rules. This settings are included in the file inside /etc/apache2 or /etc/nginx
For example with nginx you can define rules to add_header to the http response or settings to redirect to another url
for example
add_header "Access-Control-Allow-Origin: '*'"
this will add the header "Access-Control-Allow-Origin: '*'" to all responses.
If for example you apply this response to all OPTIONS requests, then all subsequent http request will be whitelisted from any http origin
STEP 2
once the nginx/apache redirection rules are applied, the rails router receives the request and redirects to your controller.
Here you can still add any header you want inside the controller, you can redirect OPTIONS request to a specific controller action, which can add a specific header, be careful to not add twice the same header as that can cause issues.
In this action you can rewrite the Access-Control-Allow-Origin header in the response from the OPTIONS request to whitelist only specific origin domains (you just need to write a routing rule which is applied only to OPTIONS requests)
The origin domain is written in the header of the request
request headers 'Access-Control-Request-Method: POST' and 'Origin: http://snafu.com'
I am trying to lookup a zipcode from usps using curl like so:
curl "https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=11111&zip="
This url does return normal output from browser.
One needs to specify the User-Agent header (using the -A option) so that the request resembles a request from a browser. Also, due to the presence of ! in the URL, one has to use ' instead of ":
curl -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:48.0) Gecko/20100101 Firefox/48.0" \
'https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=Select&urbanCode=&postalCode=10001&zip='
I need post file and JSON object to API (Ruby On Rails) by curl in one request.
My request looks like this:
curl --data #file.pdf -H "Accept: application/json" -H "Content-Type: multipart/form-data" -X POST -d '{"document":{"name":"file name"}}' http://localhost:3000/api/documents
But Rails parse it bad. Params on the server:
Parameters: {"{\"document\":{\"name\":\"file name\"}}"=>nil}
Where is problem?
You should pass multipart/form-data (as you specify), not a JSON string. Try this:
-d 'document[name]=file name'
Result on my machine:
Parameters: {"document"=>{"name"=>"file name"}}
I have a task to send a file using HTTP PUT in ruby.
In Command prompt i am able to get success response using the following command.
curl -X PUT -vN https://myexample.com/blah/myaccount --insecure -H "Connection: keep-alive" -H "Content-Type: audio/wav" -H "Content-Transfer-Encoding: base64" -d #/myfile.wav
While trying to send the request using Curl::Easy in ruby. I get a error response code stating something wrong with audio.
c = Curl::Easy.new("https://myexample.com/blah/myaccount")
c.timeout = 30
c.ssl_verify_host = false
c.http_put ( "Content-Transfer-Encoding=base64;Connection=keep-alive;Content-Type=audio%2fwav;#{File.read(/myfile.wav)}")