I've read other answers on this topic, such as:
Parsing HTTParty response
HTTParty parsing JSON in Rails
However, I still can't figure out how to parse a response I'm receiving.
response.parsed_response = HTTParty.get(url, query: params) returns:
=> #<HTTParty::Response:0x89435d0 parsed_response="http://foo.com", #response=#<Net::HTTPOK 200 OK readbody=true>, #headers={"cache-control"=>["no-cache", "no-store"], "date"=>["Tue, 21 Feb 2017 23:10:47 GMT"], "expires"=>["Thu, 01 Jan 1970 00:00:00 GMT"], "p3p"=>["CP=\"ALL IND DSP COR CUR ADM TAIo PSDo OUR COM INT NAV PUR STA UNI\""], "pragma"=>["no-cache"], "server"=>["Apache-Coyote/1.1"], "set-cookie"=>["bar.Agent.p=c1921b97d1f8a0918621c48bd32ded2b; Domain=.bar.com; Expires=Fri, 19-Feb-2027 23:10:47 GMT; Path=/"], "content-length"=>["366"], "connection"=>["Close"]}>
I need the URL that appears after parsed_response. The other answers seemed to break down hashes that appear after parsed_response, but I'm just looking for the url that appears after parsed_response (and it only appears there in the response).
I tried:
puts response which returns the entire response above.
puts response.parsed_response which returns:
http://foo.com
=> nil
This usually works for me
response = HTTParty.get(url, options)
puts response.body
Related
I am fetching html content directly from my blog as:
response = Net::HTTP.get_response(uri)
respond_to do |format|
format.html { render :text => response.body }
end
Although at the blog engine (WordPress) I am adding header Access-Control-Allow-Origin: * how ever I noticed that its not passed within the response.
However, if I use postman to get the page or view the page into browser directly, I can see that the header is there.
EDIT
I can see other headers passed, ex:
cache-control: no-cache, must-revalidate, max-age=0
content-type: text/html; charset=UTF-8
date: Tue, 24 Jul 2018 06:37:57 GMT
expires: Wed, 11 Jan 1984 05:00:00 GMT
Any idea?
response.body will return you body part not header part. you can convert response to hash and check header like below:
> url = "https://stackoverflow.com/questions/51492025/does-ruby-strip-headers-from-response"
> uri = URI.parse(url)
> response = Net::HTTP.get_response(uri)
#=> #<Net::HTTPOK 200 OK readbody=true>
> response.to_hash
#=> {"cache-control"=>["private"], "content-type"=>["text/html; charset=utf-8"], "last-modified"=>["Tue, 24 Jul 2018 07:04:00 GMT"], "x-frame-options"=>["SAMEORIGIN"], "x-request-guid"=>["22a4b6b6-3039-46e2-b4de-c8af7cad6659"], "strict-transport-security"=>["max-age=15552000"], "content-security-policy"=>["upgrade-insecure-requests"], "accept-ranges"=>["bytes", "bytes"], "age"=>["0", "0"], "content-length"=>["31575"], "date"=>["Tue, 24 Jul 2018 07:04:46 GMT"], "via"=>["1.1 varnish"], "connection"=>["keep-alive"], "x-served-by"=>["cache-bom18221-BOM"], "x-cache"=>["MISS"], "x-cache-hits"=>["0"], "x-timer"=>["S1532415886.990199,VS0,VE280"], "vary"=>["Accept-Encoding,Fastly-SSL"], "x-dns-prefetch-control"=>["off"], "set-cookie"=>["prov=a7dfe911-76a1-f1c1-093b-3fc8fe79af65; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly"]}
You can access specific header as below by passing header name:
> response['Cache-Control']
#=> "private"
for more details read: https://ruby-doc.org/stdlib-2.5.1/libdoc/net/http/rdoc/Net/HTTP.html
If you want to pass through headers that are being served from the host that you're fetching from, you first need to stash the response from your blog in a different variable name. Let's call it blog_response (this is because response is a preexisting special method name in a rails controller instance.).
blog_response = Net::HTTP.get_response(uri)
Then you need to grab the header you care about from the blog_response like this:
header_name, header_value = blog_response.each_header.find do |name, value|
name =~ /pattern-matching-a-header-name-i-care-about/i #case insensitive regex matching recommended for http headers
end
Then you need to set them in your controller before you render the response, e.g.:
response.headers[header_name] = header_value
respond_to do |format|
format.html { render :text => blog_response.body }
end
This example is obviously only for one header, but you can copy multiple headers by just iterating through, matching and setting them in your response like so:
blog_response.each_header.select do |name, value|
if name =~ /pattern-matching-header-names-i-care-about|some-other-pattern-i-care-about/i #case insensitive regex matching recommended for http headers
response.headers[name] = value
end
end
If you want to pass all headers through just do this:
blog_response.each_header do |name, value|
response.headers[name] = value
end
respond_to do |format|
format.html { render :text => blog_response.body }
end
Net::HTTPResponse (that is your response) mixes in Net::HTTPHeader. Thus, you can get an individual header as response['Access-Control-Allow-Origin'], iterate over them with response.each_header, or even get them all as a hash using response.to_hash.
So I was using the splunk rest api to generate the search id thorugh post and then get the search results through a get request.
I am using httpparty, & this is my post and get request as follows:
def sid
self.class.post(
"/services/search/jobs",
headers: {'Content-Type' => 'application/json'},
basic_auth: #auth
).parsed_response
end
The generated sid looks like this:
1503402217.608
I make the necessary get request as follows:
def results(sid)
self.class.get(
"/services/search/jobs/#{sid}/results/",
headers: {'Content-Type' => 'application/json', 'Accept' => 'text/html'},
basic_auth: #auth
)
end
The above request gives a 204 no content.
However if I manually paste the sid in my code and run it, i get results back.
for eg: "/services/search/jobs/1503402217.608/results/"
So my question is how can interpolate the above to get results back.
EDIT: my dyamic get request looks this in the console:
<- "GET /services/search/jobs/1503402217.608/results/ HTTP/1.1\r\nContent-Type: application/json\r\nAccept: text/html\r\nAuthorization: Basic something==\r\nConnection: close\r\nHost: server:8089\r\n\r\n"
-> "HTTP/1.1 204 No Content\r\n"
-> "Date: Tue, 22 Aug 2017 11:43:37 GMT\r\n"
-> "Expires: Thu, 26 Oct 1978 00:00:00 GMT\r\n"
-> "Cache-Control: no-store, no-cache, must-revalidate, max-age=0\r\n"
-> "Content-Length: 0\r\n"
-> "Vary: Cookie, Authorization\r\n"
-> "Connection: Close\r\n"
-> "X-Frame-Options: SAMEORIGIN\r\n"
-> "Server: Splunkd\r\n"
-> "\r\n"
EDIT: Just so you know, this how i am parsing the sid in the first place before sending it as an argument to the GET request:
sid.parsed_response["sid"]
I'd like to send a post request via URI to the eventbrite API in order to receive a user access key. Documented here: https://www.eventbrite.com/developer/v3/reference/authentication/
You must then exchange this access code for an OAuth token. Send a
POST request to:
https://www.eventbrite.com/oauth/token This POST must contain the
following urlencoded data, along with a Content-type:
application/x-www-form-urlencoded header:
code=THE_USERS_AUTH_CODE&client_secret=YOUR_CLIENT_SECRET&client_id=YOUR_API_KEY&grant_type=authorization_code
I try to translate that into rails and sending a post request via URI. The response is expected to be the authorization code:
require "uri"
require "net/http"
params = {'code' => current_user.eventbrite_key, 'client_secret' => 'XXXX', 'client_id' => 'XXXX', 'grant_type' => 'authorization_code' }
response = Net::HTTP.post_form(URI.parse('https://www.eventbrite.com/oauth/token'), params)
This isn't working (http bad request).
I researched that the default content type is already "application/x-www-form-urlencoded" so I would not have to define that in my request. The 'code' should be correct as I simply fetch it before with help of their callback URL. Other credentials should also be correct.
The response is the following:
<Net::HTTPBadRequest:0x007ff8fcc9e4b8>
"{\"server\":[\"nginx\"],\"date\":[\"Wed, 08 Jul 2015 14:48:19
GMT\"],\"content-type\":[\"application/json\"],\"transfer-encoding\":[\"chunked\"],\"connection\":[\"keep-alive\"],\"x-xss-protection\":[\"1;
mode=block\"],\"x-content-type-options\":[\"nosniff\"],\"x-ua-compatible\":[\"IE=edge\"],\"p3p\":[\"CP=\\"NOI
ADM DEV PSAi COM NAV OUR OTRo STP IND
DEM\\"\"],\"x-frame-options\":[\"SAMEORIGIN\"],\"set-cookie\":[\"mgrefby=;
Domain=.eventbrite.com; expires=Thu, 07-Jul-2016 14:48:19 GMT;
httponly; Max-Age=31536000;
Path=/\",\"G=v%3D2%26i%3D54a93968-6bc1-486a-b401-fedab0b33dc4%26a%3D5f9%26s%3D56f73cc9c4519dc0d05f6518a092e66c6c83516c;
Domain=.eventbrite.com; expires=Thu, 07-Jul-2016 14:48:19 GMT;
httponly; Path=/\",\"ebEventToTrack=; expires=Thu, 01-Jan-1970
00:00:00 GMT; Max-Age=0;
Path=/\",\"SS=AE3DLHS8ESSoJhhYoWTny-enqBu_PN4d5A;
Domain=.eventbrite.com; httponly; Path=/;
secure\",\"eblang=lo%3Den_US%26la%3Den-us; Domain=.eventbrite.com;
expires=Thu, 07-Jul-2016 14:48:19 GMT; httponly; Path=/\",\"AN=;
expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0;
Path=/\",\"mgref=typeins; Domain=.eventbrite.com; expires=Thu,
07-Jul-2016 14:48:19 GMT; httponly; Max-Age=31536000;
Path=/\",\"SP=AGQgbblORi0c9X3owNbUIuFSZeUwSlY9HoUdpypGreork-Gf0GI6rzrLrcQDGWvu49mxHIQW9iBqa6JR-1k0eGvBhwnNpaON_Aak96kQ1yu90CaN7P2lnvfddxfskEniVHppbf0rp8YL5PA4vLYzRiaWdSohVy73j8H6HlCakht1OfKyxvwG-FeyR5rwPFEJw0iGB71Azw3oyFOTJcGJcYMWdSSVgS3F6pEbV5QI4ps5WlNMW0C9uL0;
Domain=.eventbrite.com; httponly; Path=/\",\"SERVERID=djc11;
path=/\"]}"
URI.parse('https://www.eventbrite.com/oauth/token') without params returns:
{"scheme":"https","user":null,"password":null,"host":"www.eventbrite.com","port":443,"path":"/oauth/token","query":null,"opaque":null,"fragment":null,"parser":{"regexp":{"SCHEME":"(?-mix:\A[A-Za-z][A-Za-z0-9+\-.]\z)","USERINFO":"(?-mix:\A(?:%\h\h|[!$\u0026-.0-;=A-Z_a-z~])\z)","HOST":"(?-mix:\A(?:(?\u003cIP-literal\u003e\[(?:(?\u003cIPv6address\u003e(?:\h{1,4}:){6}(?\u003cls32\u003e\h{1,4}:\h{1,4}|(?\u003cIPv4address\u003e(?\u003cdec-octet\u003e[1-9]\d|1\d{2}|2[0-4]\d|25[0-5]|\d)\.\g\u003cdec-octet\u003e\.\g\u003cdec-octet\u003e\.\g\u003cdec-octet\u003e))|::(?:\h{1,4}:){5}\g\u003cls32\u003e|\h{,4}::(?:\h{1,4}:){4}\g\u003cls32\u003e|(?:(?:\h{1,4}:)?\h{1,4})?::(?:\h{1,4}:){3}\g\u003cls32\u003e|(?:(?:\h{1,4}:){,2}\h{1,4})?::(?:\h{1,4}:){2}\g\u003cls32\u003e|(?:(?:\h{1,4}:){,3}\h{1,4})?::\h{1,4}:\g\u003cls32\u003e|(?:(?:\h{1,4}:){,4}\h{1,4})?::\g\u003cls32\u003e|(?:(?:\h{1,4}:){,5}\h{1,4})?::\h{1,4}|(?:(?:\h{1,4}:){,6}\h{1,4})?::)|(?\u003cIPvFuture\u003ev\h+\.[!$\u0026-.0-;=A-Z_a-z~]+))\])|\g\u003cIPv4address\u003e|(?\u003creg-name\u003e(?:%\h\h|[!$\u0026-.0-9;=A-Z_a-z~])))\z)","ABS_PATH":"(?-mix:\A\/(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~])(?:\/(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~]))\z)","REL_PATH":"(?-mix:\A(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~])+(?:\/(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~]))\z)","QUERY":"(?-mix:\A(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~\/?])\z)","FRAGMENT":"(?-mix:\A(?:%\h\h|[!$\u0026-.0-;=#-Z_a-z~\/?])\z)","OPAQUE":"(?-mix:\A(?:[^\/].)?\z)","PORT":"(?-mix:\A[\x09\x0a\x0c\x0d
]\d*[\x09\x0a\x0c\x0d ]*\z)"}}}
I am working for importing odesk API into my local project. for that i used omniauth-odesk gem for authentication and httparty gem for call http request for access odesk information.
in my Gemfile.rb
gem 'omniauth-odesk'
gem 'httparty'
in odesk_controller.rb
class OdeskController < ApplicationController
include HTTParty
def user_details
#odesk_user = env["omniauth.auth"]
end
def search_job
#response = HTTParty.get "http://www.odesk.com/api/profiles/v2/search/jobs.json", :headers=>{"Authorization"=>"Token token=\"06ede858bcdf8a5fedfa9119fd7074c4\""}
end
def check_profile
end
end
in view/odesk/search_job.html.erb
<%= #response.inspect %>
inspect information are below :
#<HTTParty::Response:0x3368f40 parsed_response={"server_time"=>1395822483, "error"=>{"status"=>400, "code"=>400, "message"=>"Malformed request: standard autorization is not supported for this API, use an application key"}}, #response=#<Net::HTTPBadRequest 400 Bad Request readbody=true>, #headers={"server"=>["nginx"], "content-type"=>["application/json"], "content-length"=>["155"], "x-odesk-error-code"=>["400"], "x-odesk-error-message"=>["Malformed request: standard autorization is not supported for this API, use an application key"], "expires"=>["Fri, 13 Oct 2000 05:00:00 GMT"], "last-modified"=>["Wed, 26 Mar 2014 08:28:03 GMT"], "cache-control"=>["no-store, no-cache, must-revalidate", "post-check=0, pre-check=0"], "pragma"=>["no-cache"], "vary"=>["Cookie,Accept-Encoding"], "date"=>["Wed, 26 Mar 2014 08:28:03 GMT"], "connection"=>["close"]}>
I am using correct api key and secret key for odesk API. and token is also correct. but where i am wrong please check and let me know.
I am following below link for making http request
http://developers.odesk.com/w/page/12364012/search%20jobs
thanks
I'm stumped!
I have a rake task which is cron'd to run every minute.
It's logs in, it finds the JSON that I'm interested in but can take up to 30 runs of the task before any changes in the JSON are noticed in the rake task. During which time I've missed several changes of certain JSON objects.
Seems like there's some caching going on, I've tried to turn off Mechanize caching as shown, just not sure what else I can try now.
Any pointers?
Thanks in advance.
agent = Mechanize.new # {|a| a.log = Logger.new(STDERR) }
agent.history.clear
agent.max_history = 0
agent.user_agent_alias = 'Mac Safari'
page = agent.get 'http://website.com'
form = page.forms.first
form.email = 'me#home.com'
form.password = 'mypassword'
page = agent.submit form
page = agent.get 'http://website.com/password_protected_page'
jsonDirty = page.search '//script[#type="application/json"]'
Response from server:
{"server"=>"nginx", "date"=>"Thu, 13 Sep 2012 14:16:43 GMT", "content-type"=>"text/html; charset=utf-8", "connection"=>"close", "vary"=>"Cookie", "content-language"=>"plfplen", "set-cookie"=>"csrftoken=pVDg2SJ4KHqONz2OiEkNK7IbKlnJSQQf; expires=Thu, 12-Sep-2013 14:16:43 GMT; Max-Age=31449600; Path=/, affiliate=; expires=Thu, 01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/, one-click-join=; expires=Thu,01-Jan-1970 00:00:00 GMT; Max-Age=0; Path=/", "expires"=>"Thu, 01 Jan 1970 00:00:01 GMT", "cache-control"=>"no-cache", "content-encoding"=>"gzip", "transfer-encoding"=>"chunked"}
You could try appending a random query parameter to the URL. Such as:
page = agent.get "http://website.com/password_protected_page?random=#{Time.now.to_i}"