I am trying to use open-uri with Nokogiri
class Script
require 'nokogiri'
require 'open-uri'
open("http://www.ruby-lang.org/") {|f|
f.each_line {|line| p line}
}
end
It is timing out with the following error
in `initialize': execution expired (Net::OpenTimeout)
Any idea why this is happening?
The page you try to retrieve take time-out. It could be the slow response of the server which can be adjusted like below, or the site down.
url_object = open(url, "ssl_verify_mode"=>0, "allow_redirections"=>:safe, "read_timeout"=>Max_http_timeout/1000)
Refer to the doc for more explanation: https://docs.ruby-lang.org/en/2.0.0/OpenURI/OpenRead.html
Related
I wrote a custom ruby gem that makes simple HTTP requests using the following code:
request = Net::HTTP::Post.new(path, initheader = {'Content-Type' =>'application/json'})
request.body = body
response = Net::HTTP.new(myhost, myport)
response.ssl_version = :TLSv1
response.use_ssl = true
response.verify_mode = ::OpenSSL::SSL::VERIFY_NONE
response.start {|http| #res = http.request(request) }
When I try to call it from my Rails app, it loads/does nothing for about 10 seconds before making the actual POST request. Calling it again after that is almost instant thought, so I assumed the dependencies are loaded in this time.
The gem requires:
require 'net/http'
require 'json'
require 'openssl'
so I simply declared 'json' and 'openssl' as runtime requirements in the gemspec. After running
bundle install
they get installed but the 10 seconds loading time is still there.
I am calling a different method before the one above and that works perfectly fine so the gem is loaded properly.
Does anyone know what might cause this?
I have an API url that is a stream of data with the content type: text/event-stream.
How is it possible to listen to the stream? Like subsribe to each event to print the data? I have tried to use the ruby libary em-eventsource
My test.rb file:
require "em-eventsource"
EM.run do
source = EventMachine::EventSource.new("my_api_url_goes_here")
source.message do |message|
puts "new message #{message}"
end
source.start
end
When I visit my api url I can see the data updated each second. But when I run the ruby file in the terminal it does not print any data/messages.
Set a timer to check source.ready_state, it seems like it does not connect to api for some reason
EDIT: it seems your problem is in https' SNI, which is not supported by current eventmachine release, so upon connecting eventsource tries to connect to default virtual host on api machine, not the one the api is on, thus the CONNECTING state
Try using eventmachine from master branch, it already states to have support for SNI, that is going to be released in 1.2.0:
gem 'eventmachine', github:'eventmachine/eventmachine'
require 'eventmachine'
require 'em-http'
require 'json'
http = EM::HttpRequest.new("api_url", :keepalive => true, :connect_timeout => 0, :inactivity_timeout => 0)
EventMachine.run do
s = http.get({'accept' => 'application/json'})
s.stream do |data|
puts data
end
end
I used EventMachine http libary.
I have a bookmarking site, done in ruby on rails, in which lots of URLs needed to be open and crawl its title and base_uri. Th method used for opening URL is open(url). When I tried to open http://www.mysite.com/ with open URI method, I got 500 Internal server error.
OpenURI::HTTPError in TestsController#test
500 Internal Server Error
I can access this URL through browser.
My code posted below
require 'hpricot'
require 'open-uri'
require 'timeout'
require 'net/http'
url = 'http://www.mysite.com/'
#filep = open(url)
base_uri = #filep.base_uri
I tried the same with hpricot too using the code.
#doc = Nokogiri::HTML(open(url).read) but getting the same error.
Please help me on this.
I had the exact same problem; could open the website in my browser but not through open-uri . Adding a user agent did not fix it, but using the 'restclient' class did
require 'restclient'
url = 'http://www....'
user_info = RestClient.get(url, "User-Agent" => "Ruby")
I'm new with Rubyonrails and Koala gem and I'm sure I'm doing something wrong. I've been tuning my code to the minimun expression but the problem persist. Then I tried to do the same without koala gem, but the problem persisted.
This is the code:
require 'koala'
require 'open-uri'
puts Time.now
#graph = Koala::Facebook::API.new
resp = #graph.graph_call("cocacola", {}, "get", {})
puts resp
puts Time.now
coke_url = "https://graph.facebook.com/cocacola"
response = open coke_url
response = JSON.parse response.read
puts response.inspect
puts Time.now
I have to wait always 21 seconds the Facebook's response. If I put the https://graph.facebook.com/cocacola on my browser, the response is instantaneous ¿is not the same?
thanks
I'm using Koala in my application and did not have such experience. The only difference is that I don't use it anonymously. Instead I created an app and I'm using an access token to access the Facebook API. This might be the root cause, as I've found this post that also seems relates.
Finally, it was a DNS problem (thanks jpgeek).
When I did just a GET request to any website, the response was after 21 seconds, but using the IP of the same website, the response was instantaneously.
I found on google the solution: http://www.mikeperham.com/2010/02/10/asynchronous-dns-resolution/
I have use this personal solution in Gemfile (I'm not sure if it's the best):
group :development do
require 'resolv'
require 'resolv-replace'
end
Now it's working fine!
In PHP I can do this:
$request = "http://www.example.com/someData";
$response = file_get_contents($request);
How would I do the same thing in Ruby (or some Rails method?)
I've been googling for a half an hour and coming up completely short.
The standard library package open-uri is what you're after:
require 'open-uri'
contents = open('http://www.example.com') {|io| io.read}
# or
contents = URI.parse('http://www.example.com').read
require 'net/http'
Net::HTTP.get(URI.parse('http://www.example.com/index.html'))
Not sure why I didn't find this earlier. Unless there's an better way, I'm going with this!
Using the net/http library as shown:
require 'net/http'
response = Net::HTTP.get_response('mysite.com','/api/v1/messages')
p response.body
In your view try
<%= request.inspect %>