I have a background job being run by Resque and I the process begins correctly but then I see the error:
Error
undefined method `path' for #<Hash:0x007f1900c25298>
/app/app/models/weigh_in.rb:373:in `import_without_check'
/app/app/jobs/uploads.rb:8:in `perform'
and initially this was an error but I've since updated my file and completely removed this line, pushed to heroku, restarted heroku as well as redis, yet still the error persists.
Here's the function being called in app/models/weigh_in:
def self.import_without_check(file, location_id)
error = []
success = []
options = {:key_mapping => #new_hash, :strings_as_keys => true, :keep_original_headers => true, :remove_unmapped_keys => true}
SmarterCSV.process(file, options) do |row|
hashed_row = row[0]
next if hashed_row[:scale_id].blank? || hashed_row[:scale_id].nil?
hashed_row[:unique_scale] = location_id + hashed_row[:scale_id].to_s
hashed_row = hashed_row.to_hash.except!(nil).as_json
p hashed_row
client = Client.select('id', 'name').find_by(unique_scale: hashed_row['unique_scale']) || Client.select('id', 'name').find_by(unique_mb: hashed_row['unique_scale'])
if client.nil?
error << hashed_row
next
end
hashed_row['client_id'] = client.id
program_id = client.programs.last.id
if program_id.nil? || hashed_row['client_id'].nil?
error << hashed_row
next
end
check_in = CheckIn.new(client_id: client.id, type_of_weighin: 'Standard', program_id: program_id)
unless hashed_row['date'].blank? || hashed_row['date'].nil?
p 'date', hashed_row['date']
hashed_row['date'] = Date.strptime(hashed_row["date"], "%m/%d/%y").strftime()
end
hashed_row.except!("unique_scale")
if check_in.save
hashed_row['check_in_id'] = check_in.id
end
if hashed_row['check_in_id'].nil?
error << hashed_row
next
end
weigh_in = WeighIn.new(hashed_row)
p weigh_in.valid?, weigh_in.errors.full_messages, weigh_in.errors
if weigh_in.save
success << hashed_row
end
end
return success, error
end
Is there something I need to do to make this error go away?
I solved this issue by connecting to the redis Heroku add-on via it's cli, then getting its clients, killing all connections, and restarting my heroku dynos. Once this was done, my changes were recorded:
$ heroku redis:cli
21579> client list # Returns clients
21579> client kill 10.159.107.79:43434 # use number from addr property returned with client list command
21579> quit # Exit cli
$ heroku restart
I hope this helps someone else
Related
I am attempting to analyze CSV before importing into my database but am encountering a ActionDispatch::Cookies::CookieOverflow error.
controller
def bulk_upload_weigh_ins
#response = WeighIn.check_file(params[:file])
redirect_to import_weigh_ins_path, notice: #response
end
model
def self.check_file(file)
status = {
name_error: [],
weigh_in_error: []
}
count = 0
CSV.foreach(file.path, headers: #weigh_in_cols) do |row|
hashed_row = row.to_hash
user = User.find_by(scale_id: hashed_row["scale_id"])
if user == nil
status[:name_error] << [hashed_row["name"], hashed_row["scale_id"]]
elsif user.check_ins.with_no_weigh_ins.length != 1
status[:weigh_in_error] << [hashed_row["name"], hashed_row["scale_id"]]
else
count += 1
end
status[:number_of_success] = count
end
return status
end
I've looked at this post but I'm not sure I should be using doing the same fix. Is there a better way to set this up?
I did the following to fix this:
rails generate active_record:session_migration
rake db:migrate
Changed the line in config/initializers/session_store.rb to: Rails.application.config.session_store :active_record_store
Restarted by server
Restarted my browser (wasn't obvious to me)
Hope this helps someone
I am hitting an external service which does some password encryption and returns couple of things.
Now if I want to generate 50 passwords we run this function in a loop 50 times
def self.encrypt_password(password)
retries = 2
uri = URI
params = Base64.encode64(password)
uri.query = URI.encode("Source=#{params}")
begin
retries.times.each do
res = Net::HTTP.get_response(uri)
if res.is_a?(Net::HTTPSuccess)
obj = JSON.parse(res.body)
pw = Base64.decode64(obj["Data"])
ps = Base64.decode64(obj["Key"])
pws = Iconv.iconv('ascii', 'utf-16', pws)
return pwe,pws[0]
end
end
rescue
raise "Error generating pws: #{$!}"
end
end
But the problem, i am encountering is that there are occasions when the service just returns the following error in the middle of a loop and exits:
"Connection reset by Peer error"
My question is how do I rescue from that error and retry a few times without breaking the flow of the program?
Or can someone recommend alternate solutions to my problem?
NOTE: I am using ruby on rails 2 and ruby 1.8.x
Ruby has the retry method, that can be used in the rescue clause.
It just runs the current method again, so you can use a counter to limit the number of retries:
def self.encrypt_password(password)
retries = 2
uri = URI
params = Base64.encode64(password)
uri.query = URI.encode("Source=#{params}")
retries.times.each do
res = Net::HTTP.get_response(uri)
if res.is_a?(Net::HTTPSuccess)
obj = JSON.parse(res.body)
pw = Base64.decode64(obj["Data"])
ps = Base64.decode64(obj["Key"])
pws = Iconv.iconv('ascii', 'utf-16', pws)
return pwe,pws[0]
end
end
rescue SomeExceptionType
if retries > 0
retries -= 1
retry
else
raise "Error generating pws: #{$!}"
end
end
end
Browser error:
NoMethodError
undefined method `connect_timeout=' for #<Mysql2::Client:0x47f7570>
On my browser, an error comes up that connect_timeout is undefined. I'm pretty sure it has something to do with the client.rb file. I'll show you the file. I had to edit some of it to actually get Webrick up and running. When I started the server, an error always appeared on my command line unless I made the changes. I've commented on what I have edited. Sometimes edited random things and some of them worked but they produced different errors on my browser. I am using a windows 8 machine. Thank you for helping.
module Mysql2
class Client
attr_reader :query_options, :read_timeout
##default_query_options = {
:as => :hash, # the type of object you want each row back as; also supports :array (an array of values)
:async => false, # don't wait for a result after sending the query, you'll have to monitor the socket yourself then eventually call Mysql2::Client#async_result
:cast_booleans => false, # cast tinyint(1) fields as true/false in ruby
:symbolize_keys => false, # return field names as symbols instead of strings
:database_timezone => :local, # timezone Mysql2 will assume datetime objects are stored in
:application_timezone => nil, # timezone Mysql2 will convert to before handing the object back to the caller
:cache_rows => true, # tells Mysql2 to use it's internal row cache for results
#:connect_flags => REMEMBER_OPTIONS | LONG_PASSWORD | LONG_FLAG | TRANSACTIONS | PROTOCOL_41 | SECURE_CONNECTION,
#I had to delete the line above because for some reason the command prompt said that each of the constants were undefined were not used in the right place or something
:cast => true,
:default_file => nil,
:default_group => nil
}
def initialize (opts = {})
opts = Mysql2::Util.key_hash_as_symbols( opts )
#read_timeout = nil
#query_options = ##default_query_options.dup
#query_options.merge! opts
#initialize_ext
# the chrome page said that the above variable is undefined :P
# Set default connect_timeout to avoid unlimited retries from signal interruption
opts[:connect_timeout] = 120 unless opts.key?(:connect_timeout)
[:reconnect, :connect_timeout, :local_infile, :read_timeout, :write_timeout, :default_file, :default_group, :secure_auth, :init_command].each do |key|
next unless opts.key?(key)
case key
when :reconnect, :local_infile, :secure_auth
send(:"#{key}=", !!opts[key])
when :connect_timeout, :read_timeout, :write_timeout
send(:"#{key}=", opts[key].to_i)
else
send(:"#{key}=", opts[key])
end
end
# force the encoding to utf8
self.charset_name = opts[:encoding] || 'utf8'
ssl_options = opts.values_at(:sslkey, :sslcert, :sslca, :sslcapath, :sslcipher)
ssl_set(*ssl_options) if ssl_options.any?
if [:user,:pass,:hostname,:dbname,:db,:sock].any?{|k| #query_options.has_key?(k) }
warn "============= WARNING FROM mysql2 ============="
warn "The options :user, :pass, :hostname, :dbname, :db, and :sock will be deprecated at some point in the future."
warn "Instead, please use :username, :password, :host, :port, :database, :socket, :flags for the options."
warn "============= END WARNING FROM mysql2 ========="
end
user = opts[:username] || opts[:user]
pass = opts[:password] || opts[:pass]
host = opts[:host] || opts[:hostname]
port = opts[:port]
database = opts[:database] || opts[:dbname] || opts[:db]
socket = opts[:socket] || opts[:sock]
flags = opts[:flags] ? opts[:flags] | #query_options[:connect_flags] : #query_options[:connect_flags]
# Correct the data types before passing these values down to the C level
user = user.to_s unless user.nil?
pass = pass.to_s unless pass.nil?
host = host.to_s unless host.nil?
port = port.to_i unless port.nil?
database = database.to_s unless database.nil?
socket = socket.to_s unless socket.nil?
connect user, pass, host, port, database, socket, flags
end
def self.default_query_options
##default_query_options
end
def query_info
info = query_info_string
return {} unless info
info_hash = {}
info.split.each_slice(2) { |s| info_hash[s[0].downcase.delete(':').to_sym] = s[1].to_i }
info_hash
end
private
def self.local_offset
::Time.local(2010).utc_offset.to_r / 86400
end
end
end
Mysql2::Client#initialize called connect_timeout= but there isn't such attr_writer in the client.
when :connect_timeout, :read_timeout, :write_timeout
send(:"#{key}=", opts[key].to_i)
else
If this client is written by yourself, add attr_accessor :connect_timeout in Mysql2::Client's definition and make proper use of the attribute. If it is from other library, check your load path. You may have missed some files that opened Mysql2::Client and monkey patched it.
We have an application that makes hundreds of API calls to external services. Sometimes some calls take too much time to respond.
I am using the rake_timeout gem to find time consuming process, so, Timeout::Error will be thrown whenever some request is taking too long to respond. I am rescuing this error and doing a retry on that method:
def new
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
#Below code will be repeated in every method
tries = 0
rescue Timeout::Error => e
tries += 1
retry if tries <= 3
logger.error e.message
end
This lets the method fully re-run it. This is very verbose and I am repeating it every time.
Is there any way to do this so that, if the Timeout:Error occurrs, it will retry that method automatically three times?
I have a little module for that:
# in lib/retryable.rb
module Retryable
# Options:
# * :tries - Number of tries to perform. Defaults to 1. If you want to retry once you must set tries to 2.
# * :on - The Exception on which a retry will be performed. Defaults to Exception, which retries on any Exception.
# * :log - The log level to log the exception. Defaults to nil.
#
# If you work with something like ActiveRecord#find_or_create_by_foo, remember to put that call in a uncached { } block. That
# forces subsequent finds to hit the database again.
#
# Example
# =======
# retryable(:tries => 2, :on => OpenURI::HTTPError) do
# # your code here
# end
#
def retryable(options = {}, &block)
opts = { :tries => 1, :on => Exception }.merge(options)
retry_exception, retries = opts[:on], opts[:tries]
begin
return yield
rescue retry_exception => e
logger.send(opts[:log], e.message) if opts[:log]
retry if (retries -= 1) > 0
end
yield
end
end
and than in your model:
extend Retryable
def new
retryable(:tries => 3, :on => Timeout::Error, :log =>:error) do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
...
end
You could do something like this:
module Foo
def self.retryable(options = {})
retry_times = options[:times] || 10
try_exception = options[:on] || Exception
yield if block_given?
rescue *try_exception => e
retry if (retry_times -= 1) > 0
raise e
end
end
Foo.retryable(on: Timeout::Error, times: 5) do
# your code here
end
You can even pass multiple exceptions to "catch":
Foo.retryable(on: [Timeout::Error, StandardError]) do
# your code here
end
I think what you need is the retryable gem.
With the gem, you can write your method like below
def new
retryable :on => Timeout::Error, :times => 3 do
#make_one_external_service_call = exteral_api_fetch1(params[:id])
#make_second_external_call = exteral_api_fetch1(#make_one_external_service_call)
end
end
Please read the documentation for more information on how to use the gem and the other options it provides
you could just write a helper-method for that:
class TimeoutHelper
def call_and_retry(tries=3)
yield
rescue Timeout::Error => e
tries -= 1
retry if tries > 0
Rails.logger.error e.message
end
end
(completely untested) and call it via
TimeoutHelper.call_and_retry { [your code] }
In daemon, which tracks twitter-stream I have this construction:
client.track(*hashtags) do |status|
if status.coordinates != nil
EventMachine.synchrony
job = Qu.enqueue TweetProcessor, status
puts "Enqueued tweet processing #{job.id}"
end
end
end
For ques library I'm using qu-mongo I have this config
# /config/initializers/qu.rb
Qu.configure do |c|
c.connection = Mongo::Connection.new('127.0.0.1').db("appname_qu")
end
I've tried many options, but it always results with IOError: closed stream.
This problem is related to this question. More on this you can read here. So, in my case I've just reassign rails' logger and qu's loggers to the same file in the beginning of daemon's cycle and close it in the end, and everything is working out just fine:
client.track(*hashtags) do |status|
parser_logger = ActiveSupport::BufferedLogger.new( File.join(Rails.root, "log", "qu.log"))
Rails.logger = parser_logger
Qu.configure do |c|
c.connection = Mongo::Connection.new.db("appname_qu")
c.logger = parser_logger
end
job = Qu.enqueue TweetProcessor, status
Rails.logger.close
end