I'm trying to push some jobs onto a Sidekiq queue, which up until now, worked just fine. The error only appears in production.
Stack trace:
2013-12-13T20:35:04Z 22616 TID-amwho INFO: Sidekiq client with redis options {}
/home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/pipeline.rb:79:in `finish': EXECABORT Transaction discarded because of previous errors. (Redis::CommandError)
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:121:in `block in call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:245:in `with_reconnect'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:119:in `call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2093:in `block in multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `block in synchronize'
from /home/avishai/.rvm/rubies/ruby-1.9.3-p429/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2085:in `multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:159:in `block in raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:in `with'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq.rb:67:in `redis'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:150:in `raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:50:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:98:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:83:in `client_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:40:in `perform_async'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:79:in `send_to_sidekiq'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:74:in `enqueue'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:235:in `block in parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:24:in `block (2 levels) in each_listing'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/nokogiri-1.6.0/lib/nokogiri/xml/reader.rb:107:in `each'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:22:in `block in each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `open'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:36:in `parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed.rb:153:in `process!'
from run_feed.rb:74:in `block in <main>'
from run_feed.rb:71:in `each'
from run_feed.rb:71:in `<main>'
I also encountered this error when hitting maxmemory on that redis instance.
You can test it on your dev box by setting maxmemory (ie.. in /usr/local/etc/redis.conf) to sthg small e.g.
maxmemory 3MB
Then fill up your redis with a bunch of data, start Sidekiq, and watch the fireworks
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: {"retry"=>true, "queue"=>"default", "backtrace"=>true, "class"=>"MailWorker", "args"=>["Profile::Created", 868], "jid"=>"d7b0589736818c2ffcd58c90", "enqueued_at"=>1391437950.4245608}
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: EXECABORT Transaction discarded because of previous errors.
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: /usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/pipeline.rb:76:in `finish'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:121:in `block in call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:243:in `with_reconnect'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:119:in `call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2077:in `block in multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2069:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:337:in `namespaced_block'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:224:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:93:in `block in stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/connection_pool-1.1.0/lib/connection_pool.rb:49:in `with'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq.rb:67:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/util.rb:25:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:92:in `stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:46:in `block in process'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `public_send'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:67:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/future.rb:15:in `block in new'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `block in create'
I resolved this by doing a FLUSHALL, which cleared out all the data from the redis instance. It appears that somewhere along the way the data got corrupted, and that fixed it.
Another possible reason is a broken master-slave replication. For example with min-slaves-to-write configuration option set:
127.0.0.1:6379> set foo 1
(error) NOREPLICAS Not enough good slaves to write.
To temporarily resolve just execute:
CONFIG SET min-slaves-to-write 0
Related
An error occurs every now and then while the Sidekiq is running. What could be the reason? I use the Digital Ocean and:
Ubuntu 18.04.1 LTS
rails - 5.1.7
redis - 4.1.0
sidekiq - 5.2.7
sidekiq: 3 queues, 3 processes
redis_version: 4.0.9
Digital Ocean: CPU Optimized Droplets 8 GB, 4 vCPUs, 50 GB
From sidekiq.log:
E, [2019-07-20T22:53:30.849239 #9392] ERROR -- : Exception occurred: can't alloc thread
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:392:in `new'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:392:in `start_connection_worker'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:298:in `send_command'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:97:in `gauge'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician.rb:75:in `gauge'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician/reporters/redis.rb:20:in `ensure in call_with_metrician_time'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician/reporters/redis.rb:23:in `call_with_metrician_time'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:212:in `block in call_with_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:285:in `with_socket_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:211:in `call_with_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1179:in `block in _bpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:50:in `block in synchronize'
/home/martio/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:50:in `synchronize'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1176:in `_bpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1221:in `brpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:97:in `block in redis'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:94:in `redis'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:89:in `get_one'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:99:in `fetch'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:82:in `process_one'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
Something strange is happening with my application when I boot the jobs server with rake jobs:work. The rails webserver boots fine. I've noticed threads such as this one, however 1) it's not quite the same error message, that one seems more straightforward and 2) the answer did not yield any different results.
Here's my stacktrace -
rake aborted!
Psych::BadAlias: Unknown alias: 1
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:77:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:35:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:35:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:17:in `load_dj'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:85:in `payload_object'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:74:in `name'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:245:in `job_say'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:272:in `handle_failed_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:213:in `block in run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:213:in `rescue in run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:202:in `run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `block in reserve_and_run_one_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `reserve_and_run_one_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:187:in `block in work_off'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `times'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `work_off'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:150:in `block (4 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:149:in `block (3 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:148:in `block (2 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `loop'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `block in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block in add'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:146:in `start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/tasks.rb:9:in `block (2 levels) in <top (required)>'
I got the same issue after ruby upgrade from 2.1.5 to 2.3.8. Thats latest ruby version that could be run with rails 4.1.8.
Upgrading to the latest version of delayed_job did not help. Also tried to specify downgraded version of psych gem in my Gemfile. This did work not as well.
The real issue is that if error appears on job parsing stage then delayed_job does not update last_error column.
So I just went though all the jobs and checked if it could be parsed or not:
failing_jobs = []
Delayed::Job.find_each do |job|
begin
# this code is from dj source https://github.com/collectiveidea/delayed_job/blob/master/lib/delayed/psych_ext.rb#L15
yaml = job.handler
result = Psych.parse(yaml)
result ? Delayed::PsychExt::ToRuby.create.accept(result) : result
rescue Psych::BadAlias => e
failing_jobs << job
end
end
In my case it were 6 jobs out of ~3k that could be deleted safely so I got lucky :)
Refs:
https://github.com/collectiveidea/delayed_job/pull/867
https://github.com/collectiveidea/delayed_job_mongoid/pull/65
I have tried twice to install an installation of RubyMine on a CentOS7 VM. They have now both failed due to a Redis::TimeoutError.
I have received this error when attempting to start the rails server. This happens both through RubyMine and through the Konsole after I start redis through the "service redis start" command.
I can start the rails server fine without starting redis first. And the application works fine (so far) if I start redis after the server starts. However this is only a short term solution as I will be adding automatic start for all services such as redis when my VM starts.
My backtrace
/root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:111:in `rescue in _write_to_socket': Connection timed out (Redis::TimeoutError)
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:104:in `_write_to_socket'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:131:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `loop'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:374:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:271:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:250:in `io'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:269:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:228:in `block (3 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `block (2 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:367:in `ensure_connected'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:221:in `block in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:306:in `logging'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:220:in `process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:120:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:42:in `block in call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/datastores.rb:111:in `wrap'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:41:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:754:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `block in synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:750:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/interface.rb:9:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/ttl.rb:8:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:29:in `_marshal'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `block in write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:60:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `block in instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:58:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:105:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:54:in `store'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:63:in `block (2 levels) in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:62:in `block in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
from /root/RubymineProjects/chop/config/environment.rb:5:in `<top (required)>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/RubymineProjects/chop/config.ru:3:in `block in <main>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `instance_eval'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `initialize'
I hava a same problem when I create redis-cluster.Finally,I find the problem.
you should do that:
1、gem uninstall redis
2、gem install redis --version 3.0.0
I was following this blog, to set up my rails server, and after I ran this command, I am getting the following error
cap production deploy:initial --trace
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host 104.197.138.17: connection closed by remote host
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/runners/parallel.rb:16:in `rescue in block (2 levels) in execute'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Net::SSH::Disconnect: connection closed by remote host
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:50:in `rescue in block (2 levels) in negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:46:in `block (2 levels) in negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `loop'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `block in negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `loop'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:32:in `initialize'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `new'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `block in initialize'
/home/vagrant/.rvm/rubies/ruby-2.1.7/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
/home/vagrant/.rvm/rubies/ruby-2.1.7/lib/ruby/2.1.0/timeout.rb:126:in `timeout'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `initialize'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `new'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `start'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:50:in `call'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:50:in `create_new_entry'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:22:in `checkout'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:187:in `with_ssh'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:137:in `block in _execute'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:133:in `tap'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:133:in `_execute'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:62:in `test'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/capistrano-rvm-0.1.2/lib/capistrano/tasks/rvm.rake:21:in `block (3 levels) in <top (required)>'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:54:in `run'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/runners/parallel.rb:13:in `block (2 levels) in execute'
EOFError: end of file reached
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:47:in `readpartial'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:47:in `block (2 levels) in negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `loop'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:45:in `block in negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `loop'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:43:in `negotiate!'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/server_version.rb:32:in `initialize'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `new'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `block in initialize'
/home/vagrant/.rvm/rubies/ruby-2.1.7/lib/ruby/2.1.0/timeout.rb:75:in `timeout'
/home/vagrant/.rvm/rubies/ruby-2.1.7/lib/ruby/2.1.0/timeout.rb:126:in `timeout'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh/transport/session.rb:84:in `initialize'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `new'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/net-ssh-2.9.2/lib/net/ssh.rb:207:in `start'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:50:in `call'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:50:in `create_new_entry'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/connection_pool.rb:22:in `checkout'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:187:in `with_ssh'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:137:in `block in _execute'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:133:in `tap'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:133:in `_execute'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:62:in `test'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/capistrano-rvm-0.1.2/lib/capistrano/tasks/rvm.rake:21:in `block (3 levels) in <top (required)>'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/backends/netssh.rb:54:in `run'
/home/vagrant/.rvm/gems/ruby-2.1.7/gems/sshkit-1.7.1/lib/sshkit/runners/parallel.rb:13:in `block (2 levels) in execute'
Tasks: TOP => rvm:hook
So in my deploy.rb file which is generated by Capistrano, I commented out
this line. And it started to work.
set :ssh_options,{ forward_agent: true, user: fetch(:user), keys: %w(~/.ssh/id_rsa.pub) }
It appears Capistrano can't connect to the server and is timing out, the most likely culprit is that it's not reading the SSH key correctly. The easiest way to test it would be to try creating an ssh config file:
nano ~/.ssh/config
Add:
Host 104.197.138.17
HostName 104.197.138.17
IdentityFile ~/.ssh/id_rsa
Try:
ssh deploy#104.197.138.17
Make sure it connects. (I'm assuming 104.197.138.17 is the IP of your server...)
If that works, try cap production deploy:check again.
--
What's happening here is the ssh config will feed the extra parameters to any SSH call to that Host (in this case the IP). You could change the Host to 'mysite.com' if you wanted and then you would run ssh deploy#mysite.com if you wanted to create some separation. Digital Ocean has tutorial about it.
All my delayed jobs are failing for no apparent reason in production on the server. I tried restarting, removing all previous jobs and again starting the job worker, but nothing worked.
Suspecting it must be a capistrano issue, I manually ran rake jobs:work via ssh on the server but I got the same error:
ubuntu#ip-172-31-35-0:~/apps/instano-api/current$ rake jobs:work
[Worker(host:ip-172-31-35-0 pid:22174)] Starting job worker
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) RUNNING
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) FAILED (3 prior attempts) with NameError: undefined method `error' for class `Class'
I use ExceptionNotifier gem, from which I got the following stack trace (but none of the lines are from my application code):
A NameError occurred in background at 2014-12-17 02:43:21 +0530 :
undefined method `error' for class `Class'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
-------------------------------
Backtrace:
-------------------------------
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:113:in `hook'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:97:in `rescue in block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:101:in `block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:91:in `invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block (2 levels) in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:82:in `block in timeout'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `catch'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `timeout'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:198:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `block in reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:182:in `block in work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:146:in `block (4 levels) in start'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:145:in `block (3 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:144:in `block (2 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `loop'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `block in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:142:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:124:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:112:in `block in run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/controller.rb:70:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:110:in `run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:91:in `block in daemonize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `daemonize'
bin/delayed_job:5:in `<main>'
-------------------------------
Data:
-------------------------------
* data: {}
This seems to be a feature limitation in Delayed Job. Errors are not reported properly. The trace and error message is completely incorrect.
I debugged mine by looking at the rails production log. It was hard to find because there was no error while adding the task to delayed job i.e. I was getting a [INFO] tag instead of [ERROR] in logs.
I hope someone points out how to get proper and meaningful stack traces in case of failed Delayed Jobs.
As a footnote, my error was a real specific (and lame) one. My smtp settings in /config/environments/production.rb was incompatible my mailer so I was getting this error only in production.