Geocoding API not responding fast enough - ruby-on-rails

I am using the Geocoder gem but lately it does not seem to work.
I get this error:
Geocoding API not responding fast enough (use Geocoder.configure(:timeout => ...) to set limit).
My application_controller.rb is:
before_filter :lookup_ip_location
private
def lookup_ip_location
if Rails.env.development?
prr = Geocoder.search(request.remote_ip).first
p "this is #{prr.inspect}"
else
request.location
end
end
This is development.rb:
# test geocoder gem locally
class ActionDispatch::Request
def remote_ip
"71.212.123.5" # ipd home (Denver,CO or Renton,WA)
# "208.87.35.103" # websiteuk.com -- Nassau, Bahamas
# "50.78.167.161" # HOL Seattle, WA
end
end
I am loading an IP addresses from development.rb to check if geocoder works locally, but it does not. I am getting the above error.
Also, when printing prr I get nil.
I also added a geocoder.rb initializer to raise the timeout to 15 seconds but even after 15 seconds of the browser loading the page I'm still getting the same message.
Is it broken? Should I use another gem? If so, do you have any suggestions?

Interesting. I tried your exact methods, and was running into the same problems. I also tried bumping the timeout up to 60 seconds, and same error.
Then I noticed Geocoder uses freegeoip. So I went to see what that was all about. Lo and behold, freegeoip.net is down. Suspicious.
So I checked the Geocoder documentation for any different ip address lookup services they offer. Sure enough, under "Ip Address Services", there are multiple offers. I tried the first one that does not require an API key, which was :ipinfo_io.
[18] pry(main)> Geocoder.configure(ip_lookup: :ipinfo_io)
=> {:timeout=>30,
:lookup=>:google,
:ip_lookup=>:ipinfo_io,
:language=>:en,
:http_headers=>{},
:use_https=>false,
:http_proxy=>nil,
:https_proxy=>nil,
:api_key=>nil,
:cache=>nil,
:cache_prefix=>"geocoder:",
:basic_auth=>{},
:logger=>:kernel,
:kernel_logger_level=>2,
:always_raise=>[],
:units=>:mi,
:distances=>:linear}
[19] pry(main)> Geocoder.search("144.138.175.101")
=> [#<Geocoder::Result::IpinfoIo:0x007fce5da5fe28 #cache_hit=nil, #data={"ip"=>"144.138.175.101", "city"=>"", "region"=>"", "country"=>"AU", "loc"=>"-27.0000,133.0000"}>]
And it works! But the response doesn't have much info. I would recommend looking at the other ip lookup services that Geocoder uses. Find one that is reliable and has enough response info for your needs. Seems that freegeoip is free, but can also be unreliable. Cheers.
EDIT: Found some related information about freegeoip.net here. If you really wish to use freegeoip, looks like you can run your own instance. Hope this helps!

Related

RubyGem 'steam-condenser' - .fetch method only time and fetch_games undefined method `[]'

I am trying to get some basic information from the Steam Community via the steam-condenser gem and so far the Steam.new seems to work just fine with all the players information.
however when I do this (example)
player = SteamId.new("tiger")
stats = player.fetch_games
I get the following error message
Traceback (most recent call last):
1: from lib/assets/ruby/test.rb:15:in `<main>'
/home/zigs/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/steam-condenser-1.3.11/lib/steam/community/steam_id.rb:326:in `fetch_games': undefined method `[]' for nil:NilClass (NoMethodError)
A lot of the information I need seems to be connected to the fetch_games (for example the method total_playtime(id))
Not sure why this is not working. I am lost. Any help or ideas are highly appreciated! Thank you!
TLDR; it looks like this gem no longer works.
the particular module that you're having trouble with is:
def fetch_games
games_data = parse "#{base_url}/games?xml=1"
#games = {}
#recent_playtimes = {}
#total_playtimes = {}
games_data['games']['game'].each do |game_data|
app_id = game_data['appID'].to_i
#games[app_id] = SteamGame.new app_id, game_data
recent = game_data['hoursLast2Weeks'].to_f
total = (game_data['hoursOnRecord'] || '').delete(',').to_f
#recent_playtimes[app_id] = (recent * 60).to_i
#total_playtimes[app_id] = (total * 60).to_i
end
true
end
with the particular problem statement being games_data['games']['game'].each
If we were looking to get information for a particular user, it downloads an XML document about the user from a URL looking like:
http://steamcommunity.com/id/demomenz?xml=1
and this file does not seem to contain any games objects in it.
Having looked at the codebase for the steam-condenser gem; it hasn't really been updated in about 6 years. I can only assume that the XML format has been modified since this time and that the gem will no longer work.
Valve has added more privacy options to Steam Community profiles which are not reflected in the old XML APIs.
Apparently, the profile in question (tiger) has it‘s game details set to “Friends Only” or ”Private” as games are also unavailable in the browser.
The code from the released 1.x versions is no longer guaranteed to work when it comes to Steam Community. Valve deprecated the old XML APIs several years ago. Sadly, the modern Web API hasn‘t gotten much attention from Valve‘s side either. So development of Steam Condenser has mostly come to halt, too.
You might have more luck using the code from the master branch of the GitHub repository which uses Web API for most of the Community features.
You will have to register for a Steam Web API key, though: https://steamcommunity.com/dev/apikey

Threading error when using `ActiveRecord with_connection do` & ActionController::Live

Major edit: Since originally finding this issue I have whittled it down to the below. I think this is now a marginally more precise description of the problem. Comments on the OP may therefore not correlate entirely.
Edit lightly modified version posted in rails/puma projects: https://github.com/rails/rails/issues/21209, https://github.com/puma/puma/issues/758
Edit Now reproduced with OS X and Rainbows
Summary: When using Puma and running long-running connections I am consistently receiving errors related to ActiveRecord connections crossing threads. This manifests itself in message like message type 0x## arrived from server while idle and a locked (crashed) server.
The set up:
Ubuntu 15 / OSX Yosemite
PostgreSQL (9.4) / MySQL (mysqld 5.6.25-0ubuntu0.15.04.1)
Ruby - MRI 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] / Rubinius rbx-2.5.8
Rails (4.2.3, 4.2.1)
Puma (2.12.2, 2.11)
pg (pg-0.18.2) / mysql2
Note, not all combinations of the above versions have been tried. First listed version is what I'm currently testing against.
rails new issue-test
Add a route get 'events' => 'streaming#events'
Add a controller streaming_controller.rb
Set up database stuff (pool: 2, but seen with different pool sizes)
Code:
class StreamingController < ApplicationController
include ActionController::Live
def events
begin
response.headers["Content-Type"] = "text/event-stream"
sse = SSE.new(response.stream)
sse.write( {:data => 'starting'} , {:event => :version_heartbeat})
ActiveRecord::Base.connection_pool.release_connection
while true do
ActiveRecord::Base.connection_pool.with_connection do |conn|
ActiveRecord::Base.connection.query_cache.clear
logger.info 'START'
conn.execute 'SELECT pg_sleep(3)'
logger.info 'FINISH'
sse.write( {:data => 'continuing'}, {:event => :version_heartbeat})
sleep 0.5
end
end
rescue IOError
rescue ClientDisconnected
ensure
logger.info 'Ensuring event stream is closed'
sse.close
end
render nothing: true
end
end
Puma configuration:
workers 1
threads 2, 2
#...
bind "tcp://0.0.0.0:9292"
#...
activate_control_app
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end
Run the server puma -e production -C path/to/puma/config/production.rb
Test script:
#!/bin/bash
timeout 30 curl -vS http://0.0.0.0/events &
timeout 5 curl -vS http://0.0.0.0/events &
timeout 30 curl -vS http://0.0.0.0/events
This reasonably consistently results in a complete lock of the application server (in PostgreSQL, see notes). The scary message comes from libpq:
message type 0x44 arrived from server while idle
message type 0x43 arrived from server while idle
message type 0x5a arrived from server while idle
message type 0x54 arrived from server while idle
In the 'real-world' I have quite a few extra elements and the issue presents itself at random. My research indicates that this message comes from libpq and is subtext for 'communication problem, possibly using connection in different threads'. Finally, while writing this up, I had the server lock up without a single message in any log.
So, the question(s):
Is the pattern I'm following not legal in some way? What have I mis[sed|understood]?
What is the 'standard' for working with database connections here that should avoid these problems?
Can you see a way to reliably reproduce this?
or
What is the underlying issue here and how can I solve it?
MySQL
If running MySQL, the message is a bit different, and the application recovers (though I'm not sure if it is then in some undefined state):
F, [2015-07-30T14:12:07.078215 #15606] FATAL -- :
ActiveRecord::StatementInvalid (Mysql2::Error: This connection is in use by: #<Thread:0x007f563b2faa88#/home/dev/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.3/lib/action_controller/metal/live.rb:269 sleep>: SELECT `tasks`.* FROM `tasks` ORDER BY `tasks`.`id` ASC LIMIT 1):
Warning: read 'answer' as 'seems to make a difference'
I don't see the issue happen if I change the controller block to look like:
begin
#...
while true do
t = Thread.new do #<<<<<<<<<<<<<<<<<
ActiveRecord::Base.connection_pool.with_connection do |conn|
#...
end
end
t.join #<<<<<<<<<<<<<<<<<
end
#...
rescue IOError
#...
But I don't know whether this has actually solved the problem or just made it extremely unlikely. Nor can I really fathom why this would make a difference.
Posting this as a solution in case it helps, but still digging on the issue.

Ruby timeout does not work in Rails?

I'm having an issue trying to get a timeout when connecting via TCPSocket to a remote resource that isn't available. It just hangs indefinitely without timing out. Ideally I'd want it to try reconnect every 2 minutes or so, but the TCPSocket.new call seems to block. I've tried using timeout() but that doesn't do anything either. Trying the same call in an IRB instance works perfectly fine, but when it's in Rails, it fails. Anyone have a work around for this?
My code looks something as follows:
def self.connect!
##connection = TCPSocket.new IP, 4449
end
def self.send(cmd)
puts "send "
unless ##connection
self.connect!
end
loop do
begin
##connection.puts(cmd)
return
rescue IOError
sleep(self.get_reconnect_delay)
self.connect!
end
end
end
Unfortunately, there is currently no way to set timeouts on TCPSocket directly.
See http://bugs.ruby-lang.org/issues/5101 for the feature request. You will have use the basic Socket class and set socket options.

Getting super_exception_notifier to work

I've installed super_exception_notifier by running:
sudo gem install super_exception_notifier
and then I've tried enabling it in my project (which already has mailing working, since it sends emails for other purposes) like this. On environment.rb I added
# Notification configuration
require 'exception_notifier'
ExceptionNotifier.configure_exception_notifier do |config|
config[:exception_recipients] = %w(info#isitsciencefiction.com)
config[:notify_error_codes] = %W( 404 405 500 503 )
end
and on my application_controller.rb I have:
require 'exception_notifiable'
class ApplicationController < ActionController::Base
include ExceptionNotifiable
Am I missing something? because no matter what error I generate. Either a 404, a route error, division by zero in a controller or in the console, in development or production mode, I get no emails and no error messages or anything at all.
Any ideas?
Pablo,
Thanks for pointing out the holes in the documentation. I will setup a blank rails project and then clearly enumerate the steps. I have already updated the Readme in response to the tickets you created on github.
To help with you immediate problem this is how I have it setup, (and it works for me! :)
Not all parts of this are essential to it working, but I'm not editing it (much), so you can see what I have:
I have this in my environment.rb:
config.load_paths += %W( #{RAILS_ROOT}/app/middlewares #{RAILS_ROOT}/app/mixins #{RAILS_ROOT}/app/classes #{RAILS_ROOT}/app/mailers #{RAILS_ROOT}/app/observers )
I have an initializer in config/initializers/super_exception_notification.rb
#The constants ($) are set in the config/environments files.
ExceptionNotifier.configure_exception_notifier do |config|
config[:render_only] = false
config[:skip_local_notification] = false
config[:view_path] = 'app/views/errors'
config[:exception_recipients] = $ERROR_MAIL_RECIPIENTS
config[:send_email_error_codes] = $ERROR_STATUS_SEND_EMAIL
#config[:sender_address] = %("RINO #{(defined?(Rails) ? Rails.env : RAILS_ENV).humanize} Error" )
config[:sender_address] = "errors#swankywebdesign.com"
config[:email_prefix] = "[RINO #{(defined?(Rails) ? Rails.env : RAILS_ENV).capitalize} ERROR] "
end
Then in my application.rb I have this:
include ExceptionNotifiable, CustomEnvironments
alias :rescue_action_locally :rescue_action_in_public if Environments.local_environments.include?(Rails.env)
self.error_layout = 'errors'
self.exception_notifiable_verbose = false
self.exception_notifiable_silent_exceptions = [MethodDisabled]
Then I also have this mixin in my app/mixins directory:
module CustomEnvironments
module Environments
def self.local_environments
%w( development test )
end
def self.deployed_environments
%w( production staging )
end
end
end
One other thing, this plugin does not abolish the rails standard which is that things in public are trump. So if you have 404.html in public, it will always get rendered for 404's.
Peter
Maybe it has has something to do with this:
http://github.com/pboling/exception_notification
Email notifications will only occur when the IP address is determined not to be local. You can specify certain addresses to always be local so that you’ll get a detailed error instead of the generic error page. You do this in your controller (or even per-controller).
consider_local "64.72.18.143", "14.17.21.25"
You can specify subnet masks as well, so that all matching addresses are considered local:
consider_local "64.72.18.143/24"
The address "127.0.0.1" is always considered local. If you want to completely reset the list of all addresses (for instance, if you wanted "127.0.0.1" to NOT be considered local), you can simply do, somewhere in your controller:
local_addresses.clear

ActiveResource timeout not functioning [duplicate]

This question already has an answer here:
Overriding/Modifying Rails Class (ActiveResource)
(1 answer)
Closed 3 years ago.
I'm trying to contact a REST API using ActiveResource on Rails 2.3.2.
I'm attempting to use the timeout functionality so that if the resource I'm contacting is down I can fail quickly - I'm doing this with the following:
class WorkspaceResource < ActiveResource::Base
self.timeout = 5
self.site = "http://mysite.com/restAPI"
end
However, when I try to contact the service when I know it isn't available, the class only times out after the default 60 seconds. I can see from the error stack that the timeout error does indeed come from an ActiveResource class in my gem folder that has the proper functions to allow timeout settings, but my set timeout never seems to work.
Any thoughts?
So apparently the issue is not that timeout is not functioning. I can run a server locally, make it not return a response within the timeout limit, and see that timeout works.
The issue is in fact that if the server does not accept the connection, timeout does not function as I expected it to - it doesn't function at all. It appears as though timeout only works when the server accepts the connection but takes too long to respond.
To me, this seems like an issue - shouldn't timeout also work when the server I'm contacting is down? If not, there should be another mechanism to stop a bunch of requests from hanging...anyone know of a quick way to do this?
The problem
If you're running on Ruby 1.8.x then the problem is its lack of real system threads.
As you can read first hereand then here, there are systemic problems with timeouts in Ruby. An interesting discussion but for you in particular some comments suggest that the timeout is effectively ignored and defaults to 60 seconds - exactly what you are seeing.
Solutions ...
I have a similar issue with our own product when trying to send emails - if the email server is down the thread blocks. For me the solution was to spin the request off on a separate thread and therefore my main request-processing thread doesn't block.
There are non-blocking libraries out there for Ruby but perhaps you could take a look first at this System Timeout Gem.
An option open to anyone using Rails behind a proxy like nginx would be to set the upstream timeout to a lower number - that way you'll get notified if the server is taking too long. I'd only do this if I were really stuck for a solution.
Last but not least, it's possible that running Rails 2.3.2 on top of Ruby 1.9.1 will fix the issue.
Alternatively, you could try to catch these connection errors and retry once (after certain period of time) just to make sure the connection is really out.
retried = false
begin
#businesses = Business.find(:all, :params => { :shop_domain => #shop.domain })
retried = false
rescue ActiveResource::TimeoutError => ex
#raise ex
rescue ActiveResource::ConnectionError, ActiveResource::ServerError, ActiveResource::ClientError => ex
unless retried
sleep(((ex.respond_to?(:response) && ex.response['Retry-After']) || 5).to_i)
retried = true
retry
else
# raise ex
end
end
Inspired by this solution from Shopify for paginating a large number of records. https://ecommerce.shopify.com/c/shopify-apis-and-technology/t/paginate-api-results-113066

Resources