I keep getting this error (without any backtrace) in my production logs:
!!! ERROR HANDLER THREW AN ERROR !!!
For my understanding that means that a registered error handler is raising an exception.
However my application doesn't have any custom error handler.
How can I debug / solve this issue?
My application looks like this:
class DeliveryJob
include Sidekiq::Worker
sidekiq_options queue: :deliveries, retry: 3
sidekiq_retry_in { |count| 20 * (3 ** count) }
sidekiq_retries_exhausted do |s|
# do something (some methods use Redis) ...
Rails.logger.tagged('delivery') do
Rails.logger.error "Sidekiq retries exhausted for ..."
end
end
def perform(id)
Notification.find(id).deliver
end
end
class Notification
def deliver
# do something that may fail
rescue => e
Rails.logger.tagged('delivery') do
Rails.logger.error "Delivery error for #{endpoint}: #{e.to_s}"
end
raise e
end
end
So in my logs I often see something like:
Delivery error for https://example.com/path: SSL connection error, etc.
Followed by:
!!! ERROR HANDLER THREW AN ERROR !!!
The Delivery error is ok and expected, however I don't understand why I get an error from an error handler.
https://github.com/mperham/sidekiq/blob/d8f11c26518dbe967880f76fd23bb99e9d2411d5/lib/sidekiq/exception_handler.rb#L23
Sidekiq throws this error from its own error handler.
Related
This is sample code which i am trying to handle exception from unicorn.
unicron.rb
worker_processes Integer(ENV['WEB_CONCURRENCY'] || 3)
timeout 15
preload_app true
timeout.rb
Rack::Timeout.timeout = 12
Sample code
def create_store
ActiveRecord::Base.transaction do
#store = Store.new(params[:store])
if #store.save!
sleep 12.3
InformUserWorker.perform_async(store_id)
end
end
rescue => exception
exception.backtrace.each { |trace| puts trace }
rescue Timeout::Error
puts 'That took too long, exiting...1'
rescue Rack::Timeout::RequestTimeoutException
puts 'That took too long, exiting...2'
rescue Rack::Timeout::RequestTimeoutError
puts 'That took too long, exiting...3'
rescue Rack::Timeout::RequestExpiryError
puts 'That took too long, exiting...4'
end
I am getting code=H13 desc="Connection closed without response" with this sleep 12.3 seconds, and the transaction rollback happens, but none of these exceptions are executing. I have added couple of exception here. Anything goes wrong here?.
Rescues are executed in order. So your first rescue captures anything that is being raised and the code never gets past that point.
Try putting it last:
# other rescues in here.
rescue => exception
exception.backtrace.each { |trace| puts trace }
end # this one is the last, only executed if none of the others are
If the problem is that you're not sure which class is being raised, just use something like pry to debug the exception.class
When debugging failing integration tests, I keep running into the same problem where the exceptions raised in my code are suppressed and not shown in the testing output.
For example, for the following controller and test:
class RegistrationController::ApplicationController
def create
# some code that raises an exception
end
end
class RegistrationFlowTest < ActionDispatch::IntegrationTest
test 'user registers successfully' do
post sign_up_path, params: { username: 'arnold', password: '123' }
assert_response :success
end
end
The output is something like
Minitest::Assertion: Expected response to be a <2XX: success>, but was a <500: Internal Server Error>
Is there a way to see the exact raised exception? Instead of just the difference of HTTP response code?
Thanks!
Simon
My recommended approach to this issue would be to actually parse the response provided by Rails (at least by default in test and development environments) which includes the stacktrace for the error and handle that in the case that your test fails. This has the advantage that it won't output the stacktrace when errors are raised that don't result in failing tests (e.g. scenarios where you are intentionally testing how failures are handled).
This little module that I've crafted will allow you to call assert_response_with_errors to assert the response to a call but output the exception message and stack trace in a readable format when the response is not what you expected.
module ActionDispatch
module Assertions
module CustomResponseAssertions
# Use this method when you want to assert a response body but also print the exception
# and full stack trace to the test console.
# Useful when you are getting errors in integration tests but don't know what they are.
#
# Example:
# user_session.post create_gene_path, params: {...}
# user_session.assert_response_with_errors :created
def assert_response_with_errors(type, message = nil)
assert_response(type, message)
rescue Minitest::Assertion => e
message = e.message
message += "\nException message: #{#response.parsed_body[:exception]}"
stack_trace = #response.parsed_body[:traces][:'Application Trace'].map { |line| line[:trace] }.join "\n"
message += "\nException stack trace start"
message += "\n#{stack_trace}"
message += "\nException stack trace end"
raise Minitest::Assertion, message
end
end
end
end
To use this, you need to include it into ActionDispatch::Assertions before Rails has loaded its stack in your test_helper.rb. So just prepend the include into your test_helper.rb, like this:
ActionDispatch::Assertions.include ActionDispatch::Assertions::CustomResponseAssertions
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
...
This will happen because Rails controllers by default handle exceptions and raise the 500 status, making the exceptions invisible to the test suite (which is very helpful if errors are raised in unit tests of a model). Options for disabling this in your test suite, or alternative workarounds, are discussed here.
The key lines of code from that link, which should be added to test/integration/integration_test_helper.rb:
ActionController::Base.class_eval do
def perform_action
perform_action_without_rescue
end
end
Dispatcher.class_eval do
def self.failsafe_response(output, status, exception = nil)
raise exception
end
end
EDIT: I've noticed that that link is quite old now. I'm really familiar with Rack, so whilst the first block looks ok to me, I'm not sure if the second will still be current. You might need to have a look at the relevant current Rails guide if it needs bringing up to date.
My sidekiq worker use get request to get the speech recognition outcome, the response status will be "SUCCESS" or "FAILED" or "RUNNING". when the status is "RUNNING", I want to retry the sidekiq worker in 10 minutes. How can I retry without sleep or raise a exception. Because sleep will consume too much resource and raise exception will leave newrelic error log which I don't want to record.
class GetAsrTextWorker
include Sidekiq::Worker
sidekiq_options :queue => :default, :retry => 5
sidekiq_retry_in do | count|
600 * (count + 1)
end
def perform(task_id):
# get request to get the outcome
if status == "SUCCESSD"
# save the outcome
elsif status == "FAILED"
raise AsrError
elsif status == "RUNNING"
raise "Asr running"
end
rescue AsrError => e
Sidekiq.logger.error e.message
end
end
This method will retry this worker but will lead to ugly error log in newrelic which I don't want to record.
Use the New Relic Ruby agent option error_collector.ignore_errors to ignore the specific exception you are raising:
Define a custom exception you can raise:
# lib/retry_job.rb
class RetryJob < StandardError; end
Define a worker that raises the exception:
# app/workers/foo_worker.rb
class FooWorker
include Sidekiq::Worker
def perform
raise RetryJob
end
end
Ignore that exception with the New Relic agent:
# config/newrelic.yml
common: &default_settings
error_collector:
ignore_errors: "ActionController::RoutingError,Sinatra::NotFound,RetryJob"
In one of my Rails application controllers, I do something like this:
def create
#user = User.create(user_params)
# send welcome email
UserMailer.welcome(#user).deliver_later
end
Now, I've intentionally disabled my Redis server so that I can replicate what would happen in the case that a connection couldn't be made from my app.
Unfortunately, this entire create request fails with a 500 if the deliver_later is unable to connect to Redis.
What I'd like is that the request still succeeds, but the mailer fails silently.
How can I accomplish this?
Additional information:
In config/initializers/action_mailer.rb:
rescue_from(Redis::CannotConnectError) do |exception|
Rails.logger.error "Original record not found: #{#serialized_arguments.join(', ')}"
end
This never gets called though on exception. I tried rescue_from(StandardError) and (Exception), but that was never called either.
I'm using sidekiq as my job queue adapter:
config.active_job.queue_adapter = :sidekiq
The 500 error I get is:
Redis::CannotConnectError (Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)):
My UserMailer is a subclass of ApplicationMailer which is a subclass of ActionMailer::Base.
In order to prevent calls to deliver_later from crashing when Redis is down, we added the following monkey-patch:
# If +raise_delivery_errors+ is false, errors occurring while attempting to
# enqueue the email for delivery will be ignored (for instance, if Redis is
# unreachable). In these cases, the email will never be delivered.
module MessageDeliveryWithSilentQueueingErrors
def deliver_later
super
rescue Redis::CannotConnectError => e
raise if raise_delivery_errors
# Log details of the failed email here
end
def deliver_later!
super
rescue Redis::CannotConnectError => e
raise if raise_delivery_errors
# Log details of the failed email here
end
end
ActionMailer::MessageDelivery.send(:prepend, MessageDeliveryWithSilentQueueingErrors)
I'm trying to figure out the best way to catch a specific error thrown AND the error's message in Ruby on Rails. My use case is that I encounter a timeout error every now and then which is thrown with a general error and I want to treat the timeout error differently than other errors within the same general error. I'm not sure what other type of errors could be thrown in the general error but I assume more of them exist. I have some sample code below of how I'm currently handling it, but I was thinking there might be a better way which I haven't found yet?
tries = 0
begin
tries += 1
<code>
rescue Foo::Bar => e
case e.to_s
when 'More specific timeout error message'
retry unless tries >= 5
else
# Let me see other error messages
log.info("Error: #{e.to_s}")
end
end
You can use multi rescue, to handle different errors.
begin
# DO SOMETHING
rescue Net::Timeout => e # change it to the error your want to catch, check the log.
# handle it
rescue SyntaxError => e # just an example
# handle it
rescue => e # any error that not catch by above rescue go here.
# handle it
end
Read more:
http://phrogz.net/programmingruby/tut_exceptions.html
You can try Rollbar, it help report error on production.
Take a look at retriable gem. It seems like a good fit for what you're proposing. Usually you'd rescue from an specific error type, but retriable also gives you the choice to rescue based on the error message.
begin
Retriable.retriable on: { Foo::Bar => /More specific timeout error message/ }, tries: 3 do
# will retry if an error of type Foo::Bar is raised
# and its message matches /More specific timeout error message/
# code here...
end
rescue => e # rescue for everything else
puts e.message # same as e.to_s
end