All my delayed jobs are failing for no apparent reason in production on the server. I tried restarting, removing all previous jobs and again starting the job worker, but nothing worked.
Suspecting it must be a capistrano issue, I manually ran rake jobs:work via ssh on the server but I got the same error:
ubuntu#ip-172-31-35-0:~/apps/instano-api/current$ rake jobs:work
[Worker(host:ip-172-31-35-0 pid:22174)] Starting job worker
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) RUNNING
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) FAILED (3 prior attempts) with NameError: undefined method `error' for class `Class'
I use ExceptionNotifier gem, from which I got the following stack trace (but none of the lines are from my application code):
A NameError occurred in background at 2014-12-17 02:43:21 +0530 :
undefined method `error' for class `Class'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
-------------------------------
Backtrace:
-------------------------------
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:113:in `hook'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:97:in `rescue in block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:101:in `block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:91:in `invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block (2 levels) in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:82:in `block in timeout'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `catch'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `timeout'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:198:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `block in reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:182:in `block in work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:146:in `block (4 levels) in start'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:145:in `block (3 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:144:in `block (2 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `loop'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `block in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:142:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:124:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:112:in `block in run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/controller.rb:70:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:110:in `run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:91:in `block in daemonize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `daemonize'
bin/delayed_job:5:in `<main>'
-------------------------------
Data:
-------------------------------
* data: {}
This seems to be a feature limitation in Delayed Job. Errors are not reported properly. The trace and error message is completely incorrect.
I debugged mine by looking at the rails production log. It was hard to find because there was no error while adding the task to delayed job i.e. I was getting a [INFO] tag instead of [ERROR] in logs.
I hope someone points out how to get proper and meaningful stack traces in case of failed Delayed Jobs.
As a footnote, my error was a real specific (and lame) one. My smtp settings in /config/environments/production.rb was incompatible my mailer so I was getting this error only in production.
Related
I'm trying to deploy a Ruby on Rails application using a preconfigured image provided by Digital Ocean for RoR applications. However, I'm getting an error as shown below:
Puma caught this error: uninitialized constant ActionText::Engine::ApplicationController (NameError)
/home/rails/.rvm/gems/ruby-2.6.5/gems/actiontext-6.0.2.1/lib/action_text/engine.rb:43:in `block (2 levels) in <class:Engine>'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:429:in `instance_exec'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:201:in `block (2 levels) in halting'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:607:in `block (2 levels) in default_terminator'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:606:in `catch'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:606:in `block in default_terminator'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:202:in `block in halting'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:514:in `block in invoke_before'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:514:in `each'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:514:in `invoke_before'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/callbacks.rb:134:in `run_callbacks'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/execution_wrapper.rb:119:in `complete!'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/execution_wrapper.rb:76:in `ensure in block in run!'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/execution_wrapper.rb:76:in `block in run!'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/execution_wrapper.rb:70:in `tap'
/home/rails/.rvm/gems/ruby-2.6.5/gems/activesupport-6.0.2.1/lib/active_support/execution_wrapper.rb:70:in `run!'
/home/rails/.rvm/gems/ruby-2.6.5/gems/actionpack-6.0.2.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/actionpack-6.0.2.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/actionpack-6.0.2.1/lib/action_dispatch/middleware/host_authorization.rb:83:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/webpacker-4.2.2/lib/webpacker/dev_server_proxy.rb:23:in `perform_request'
/home/rails/.rvm/gems/ruby-2.6.5/gems/rack-proxy-0.6.5/lib/rack/proxy.rb:57:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/railties-6.0.2.1/lib/rails/engine.rb:526:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
/home/rails/.rvm/gems/ruby-2.6.5/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
/home/rails/.rvm/gems/ruby-2.6.5/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
/home/rails/.rvm/gems/ruby-2.6.5/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
According to what I've read, the solution is to make sure that the application_controller.rb file exists. However, it does exist.
I just setup rollbar within my rails app using the rollbar gem, but when running the rake task to test my setup it gives me this error (backtrace appended below).
Started GET "/verify" for at 2017-11-07 01:29:13 +0100
IPAddr::InvalidAddressError: invalid address
Obviously the address is empty and therefore it makes sense, the address is invalid, but i have no clue why this is the case.
The token is correctly set in the environment and within the rollbar dashboard of this project I even see the failed attempt as error.
Further Info: Rails 4.2.4 on ruby 2.2.3
Here is the backtrace, I'd be really happy for any tips!
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/ipaddr.rb:559:in `in6_addr'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/ipaddr.rb:496:in `initialize'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/ipaddr.rb:514:in `new'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/ipaddr.rb:514:in `coerce_other'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/ipaddr.rb:170:in `include?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/whitelist.rb:17:in `block in include?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/whitelist.rb:17:in `any?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/whitelist.rb:17:in `include?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/request.rb:16:in `from_whitelited_ip?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/whiny_request.rb:8:in `from_whitelited_ip?'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/middleware.rb:20:in `block in call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/middleware.rb:18:in `catch'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/web-console-2.3.0/lib/web_console/middleware.rb:18:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/rack/logger.rb:38:in `call_app'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/rack/logger.rb:20:in `block in call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/tagged_logging.rb:68:in `block in tagged'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/tagged_logging.rb:26:in `tagged'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/tagged_logging.rb:68:in `tagged'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/rack/logger.rb:20:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/request_id.rb:21:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rack-1.6.5/lib/rack/methodoverride.rb:22:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rack-1.6.5/lib/rack/runtime.rb:18:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rack-1.6.5/lib/rack/lock.rb:17:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/actionpack-4.2.4/lib/action_dispatch/middleware/static.rb:116:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rack-1.6.5/lib/rack/sendfile.rb:113:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/engine.rb:518:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/railties-4.2.4/lib/rails/application.rb:165:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rollbar-2.15.5/lib/rollbar/rake_tasks.rb:86:in `block (2 levels) in <top (required)>'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:250:in `call'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
from /var/www/project/vendor/bundle/ruby/2.2.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
from /var/www/project/vendor/bundle/ruby/2.2.0/bin/rake:23:in `load'
from /var/www/project/vendor/bundle/ruby/2.2.0/bin/rake:23:in `<top (required)>'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli/exec.rb:74:in `load'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli/exec.rb:27:in `run'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli.rb:335:in `exec'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli.rb:20:in `dispatch'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/cli.rb:11:in `start'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/exe/bundle:32:in `block in <top (required)>'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
from /home/user/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/bundler-1.14.3/exe/bundle:24:in `<top (required)>'
from /home/user/.rbenv/versions/2.2.3/bin/bundle:23:in `load'
from /home/user/.rbenv/versions/2.2.3/bin/bundle:23:in `<main>'
I just ran into this myself. It seems to be a semi-known issue with Rollbar: https://github.com/rollbar/rollbar-gem/issues/541. I can confirm that removing the web-console gem (which I wasn't using anyway) from my Gemfile fixed the problem for me.
I have tried twice to install an installation of RubyMine on a CentOS7 VM. They have now both failed due to a Redis::TimeoutError.
I have received this error when attempting to start the rails server. This happens both through RubyMine and through the Konsole after I start redis through the "service redis start" command.
I can start the rails server fine without starting redis first. And the application works fine (so far) if I start redis after the server starts. However this is only a short term solution as I will be adding automatic start for all services such as redis when my VM starts.
My backtrace
/root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:111:in `rescue in _write_to_socket': Connection timed out (Redis::TimeoutError)
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:104:in `_write_to_socket'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:131:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `loop'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:374:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:271:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:250:in `io'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:269:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:228:in `block (3 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `block (2 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:367:in `ensure_connected'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:221:in `block in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:306:in `logging'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:220:in `process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:120:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:42:in `block in call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/datastores.rb:111:in `wrap'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:41:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:754:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `block in synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:750:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/interface.rb:9:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/ttl.rb:8:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:29:in `_marshal'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `block in write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:60:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `block in instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:58:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:105:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:54:in `store'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:63:in `block (2 levels) in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:62:in `block in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
from /root/RubymineProjects/chop/config/environment.rb:5:in `<top (required)>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/RubymineProjects/chop/config.ru:3:in `block in <main>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `instance_eval'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `initialize'
I hava a same problem when I create redis-cluster.Finally,I find the problem.
you should do that:
1、gem uninstall redis
2、gem install redis --version 3.0.0
I'm using Delayed Job gem to dispatch emails using Amazon SES. However it runs occasionally into execution expired error. Probably some connection issues to SES. I'm wondering how can I catch this exception and re-schedule the email manually? Due to internal reasons Delayed Job is configured to single max attempt
| execution expired
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `initialize'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `open'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `tcp_socket'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:550:in `block in do_start'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:549:in `do_start'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:519:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:112:in `deliver!'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:2129:in `do_delivery'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `block in deliver'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:456:in `block in deliver_mail'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `block in instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:454:in `deliver_mail'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `deliver'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/performable_mailer.rb:6:in `perform'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:105:in `block in invoke_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:102:in `invoke_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block (2 levels) in run'
/usr/local/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block in run'
/usr/local/lib/ruby/2.0.0/benchmark.rb:296:in `realtime'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:205:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `block in reserve_and_run_one_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `reserve_and_run_one_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:189:in `block in work_off'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `times'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `work_off'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:153:in `block (4 levels) in start'
/usr/local/lib/ruby/2.0.0/benchmark.rb:296:in `realtime'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:152:in `block (3 levels) in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:151:in `block (2 levels) in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `loop'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `block in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block in add'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:149:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:104:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:92:in `block in run_process'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:159:in `block (2 levels) in start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:158:in `fork'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:158:in `block in start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:157:in `each'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:157:in `start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/controller.rb:80:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:90:in `run_process'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:83:in `block in daemonize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:81:in `times'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:81:in `daemonize'
/home/ubuntu/www/tf/current/bin/delayed_job:5:in `<main>'
delayed_job provides some hooks, you can use the error hook to rescheduled the task.
def error(job, exception)
# if exception is the one occurring right now for expired execution
# reschedule the job
# be careful to avoid infinite rescheduling
end
I'm trying to push some jobs onto a Sidekiq queue, which up until now, worked just fine. The error only appears in production.
Stack trace:
2013-12-13T20:35:04Z 22616 TID-amwho INFO: Sidekiq client with redis options {}
/home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/pipeline.rb:79:in `finish': EXECABORT Transaction discarded because of previous errors. (Redis::CommandError)
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:121:in `block in call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:245:in `with_reconnect'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:119:in `call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2093:in `block in multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `block in synchronize'
from /home/avishai/.rvm/rubies/ruby-1.9.3-p429/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2085:in `multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:159:in `block in raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:in `with'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq.rb:67:in `redis'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:150:in `raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:50:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:98:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:83:in `client_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:40:in `perform_async'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:79:in `send_to_sidekiq'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:74:in `enqueue'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:235:in `block in parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:24:in `block (2 levels) in each_listing'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/nokogiri-1.6.0/lib/nokogiri/xml/reader.rb:107:in `each'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:22:in `block in each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `open'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:36:in `parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed.rb:153:in `process!'
from run_feed.rb:74:in `block in <main>'
from run_feed.rb:71:in `each'
from run_feed.rb:71:in `<main>'
I also encountered this error when hitting maxmemory on that redis instance.
You can test it on your dev box by setting maxmemory (ie.. in /usr/local/etc/redis.conf) to sthg small e.g.
maxmemory 3MB
Then fill up your redis with a bunch of data, start Sidekiq, and watch the fireworks
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: {"retry"=>true, "queue"=>"default", "backtrace"=>true, "class"=>"MailWorker", "args"=>["Profile::Created", 868], "jid"=>"d7b0589736818c2ffcd58c90", "enqueued_at"=>1391437950.4245608}
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: EXECABORT Transaction discarded because of previous errors.
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: /usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/pipeline.rb:76:in `finish'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:121:in `block in call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:243:in `with_reconnect'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:119:in `call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2077:in `block in multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2069:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:337:in `namespaced_block'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:224:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:93:in `block in stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/connection_pool-1.1.0/lib/connection_pool.rb:49:in `with'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq.rb:67:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/util.rb:25:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:92:in `stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:46:in `block in process'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `public_send'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:67:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/future.rb:15:in `block in new'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `block in create'
I resolved this by doing a FLUSHALL, which cleared out all the data from the redis instance. It appears that somewhere along the way the data got corrupted, and that fixed it.
Another possible reason is a broken master-slave replication. For example with min-slaves-to-write configuration option set:
127.0.0.1:6379> set foo 1
(error) NOREPLICAS Not enough good slaves to write.
To temporarily resolve just execute:
CONFIG SET min-slaves-to-write 0