An error occurs every now and then while the Sidekiq is running. What could be the reason? I use the Digital Ocean and:
Ubuntu 18.04.1 LTS
rails - 5.1.7
redis - 4.1.0
sidekiq - 5.2.7
sidekiq: 3 queues, 3 processes
redis_version: 4.0.9
Digital Ocean: CPU Optimized Droplets 8 GB, 4 vCPUs, 50 GB
From sidekiq.log:
E, [2019-07-20T22:53:30.849239 #9392] ERROR -- : Exception occurred: can't alloc thread
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:392:in `new'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:392:in `start_connection_worker'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:298:in `send_command'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/instrumental_agent-2.1.0/lib/instrumental/agent.rb:97:in `gauge'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician.rb:75:in `gauge'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician/reporters/redis.rb:20:in `ensure in call_with_metrician_time'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/metrician-0.1.0/lib/metrician/reporters/redis.rb:23:in `call_with_metrician_time'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:212:in `block in call_with_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:285:in `with_socket_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis/client.rb:211:in `call_with_timeout'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1179:in `block in _bpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:50:in `block in synchronize'
/home/martio/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:50:in `synchronize'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1176:in `_bpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/redis-4.1.0/lib/redis.rb:1221:in `brpop'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:97:in `block in redis'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq.rb:94:in `redis'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:89:in `get_one'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:99:in `fetch'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:82:in `process_one'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/home/xxx/apps/xxx/shared/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
Related
I joined a development team and I have a project in ruby (2.6.5) on rails (6) to which we are adding tests to increase coverage with simplecov. The problem is moving from the Mac CatalinaOS environment to Ubuntu.
We use the asdf version manager we have all the dependencies installed locally, the database also, redis and sidekiq the same. The project can be executed; but nevertheless, when wanting to run the tests with bundle exec rake test Ubuntu gets stuck after running some tests.
We have tested by uninstalling the ruby, node and yarn versions. Reinstalling everything, and even reinstalling PopOS 19.10 and the problem persists. Does anyone know why ruby is hanging in Ubuntu under these conditions??
I am adding the stack trace after CTRL+C with the process hanged for more than 20 min.
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/railtie.rb:28:in `block (4 levels) in <class:Railtie>'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/railtie.rb:27:in `block (3 levels) in <class:Railtie>'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:118:in `instance_exec'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/execution.rb:22:in `execute'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/queue_adapters/test_adapter.rb:50:in `enqueue_or_perform'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/queue_adapters/test_adapter.rb:32:in `enqueue'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/enqueuing.rb:53:in `block in enqueue'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/logging.rb:17:in `block (3 levels) in <module:Logging>'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/logging.rb:46:in `block in tag_logger'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/tagged_logging.rb:28:in `tagged'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/tagged_logging.rb:71:in `tagged'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/logging.rb:46:in `tag_logger'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/logging.rb:16:in `block (2 levels) in <module:Logging>'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:118:in `instance_exec'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activesupport-5.2.4.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/enqueuing.rb:49:in `enqueue'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/enqueuing.rb:20:in `perform_later'
/home/alejandro/EXTRA/REPOS/cartera/test/jobs/expire_installments_job_test.rb:42:in `block in test_it_updates_credit_installments_state_to_more_than_one_expired'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/activejob-5.2.4.1/lib/active_job/test_helper.rb:382:in `perform_enqueued_jobs'
/home/alejandro/EXTRA/REPOS/cartera/test/jobs/expire_installments_job_test.rb:41:in `test_it_updates_credit_installments_state_to_more_than_one_expired'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:98:in `block (3 levels) in run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:195:in `capture_exceptions'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:95:in `block (2 levels) in run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:270:in `time_it'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:94:in `block in run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:365:in `on_signal'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:211:in `with_info_handler'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest/test.rb:93:in `run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:1026:in `run_one_method'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:339:in `run_one_method'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:326:in `block (2 levels) in run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:325:in `each'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:325:in `block in run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:365:in `on_signal'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:352:in `with_info_handler'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:324:in `run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/railties-5.2.4.1/lib/rails/test_unit/line_filtering.rb:10:in `run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:164:in `block in __run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:164:in `map'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:164:in `__run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:141:in `run'
/home/alejandro/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/minitest-5.14.0/lib/minitest.rb:68:in `block in autorun'
As a workaround, you can manually set dns_timeout to any array of numbers. The default should be [5, 10, 15, 20].
Something strange is happening with my application when I boot the jobs server with rake jobs:work. The rails webserver boots fine. I've noticed threads such as this one, however 1) it's not quite the same error message, that one seems more straightforward and 2) the answer did not yield any different results.
Here's my stacktrace -
rake aborted!
Psych::BadAlias: Unknown alias: 1
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:77:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:35:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:35:in `visit_Psych_Nodes_Mapping'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/psych_ext.rb:17:in `load_dj'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:85:in `payload_object'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:74:in `name'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:245:in `job_say'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:272:in `handle_failed_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:213:in `block in run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:213:in `rescue in run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:202:in `run'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `block in reserve_and_run_one_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `reserve_and_run_one_job'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:187:in `block in work_off'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `times'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `work_off'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:150:in `block (4 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:149:in `block (3 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:148:in `block (2 levels) in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `loop'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `block in start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block in add'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/worker.rb:146:in `start'
/Users/ccromar/.rvm/gems/ruby-2.4.0/gems/delayed_job-4.0.6/lib/delayed/tasks.rb:9:in `block (2 levels) in <top (required)>'
I got the same issue after ruby upgrade from 2.1.5 to 2.3.8. Thats latest ruby version that could be run with rails 4.1.8.
Upgrading to the latest version of delayed_job did not help. Also tried to specify downgraded version of psych gem in my Gemfile. This did work not as well.
The real issue is that if error appears on job parsing stage then delayed_job does not update last_error column.
So I just went though all the jobs and checked if it could be parsed or not:
failing_jobs = []
Delayed::Job.find_each do |job|
begin
# this code is from dj source https://github.com/collectiveidea/delayed_job/blob/master/lib/delayed/psych_ext.rb#L15
yaml = job.handler
result = Psych.parse(yaml)
result ? Delayed::PsychExt::ToRuby.create.accept(result) : result
rescue Psych::BadAlias => e
failing_jobs << job
end
end
In my case it were 6 jobs out of ~3k that could be deleted safely so I got lucky :)
Refs:
https://github.com/collectiveidea/delayed_job/pull/867
https://github.com/collectiveidea/delayed_job_mongoid/pull/65
I have tried twice to install an installation of RubyMine on a CentOS7 VM. They have now both failed due to a Redis::TimeoutError.
I have received this error when attempting to start the rails server. This happens both through RubyMine and through the Konsole after I start redis through the "service redis start" command.
I can start the rails server fine without starting redis first. And the application works fine (so far) if I start redis after the server starts. However this is only a short term solution as I will be adding automatic start for all services such as redis when my VM starts.
My backtrace
/root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:111:in `rescue in _write_to_socket': Connection timed out (Redis::TimeoutError)
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:104:in `_write_to_socket'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:131:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `loop'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:130:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/connection/ruby.rb:374:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:271:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:250:in `io'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:269:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:228:in `block (3 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:222:in `block (2 levels) in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:367:in `ensure_connected'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:221:in `block in process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:306:in `logging'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:220:in `process'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis/client.rb:120:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:42:in `block in call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/datastores.rb:111:in `wrap'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/newrelic_rpm-3.16.2.321/lib/new_relic/agent/instrumentation/redis.rb:41:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:754:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `block in synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:58:in `synchronize'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-3.3.2/lib/redis.rb:750:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/interface.rb:9:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/ttl.rb:8:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:29:in `_marshal'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/marshalling.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `block in set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:74:in `namespace'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-store-1.1.7/lib/redis/store/namespace.rb:5:in `set'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `block in write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:212:in `with'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:223:in `write_entry'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:60:in `block in write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `block in instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/notifications.rb:166:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/cache.rb:547:in `instrument'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/redis-activesupport-4.1.5/lib/active_support/cache/redis_store.rb:58:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:105:in `write'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/entity_factory/entity_store.rb:54:in `store'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:63:in `block (2 levels) in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/st_core-1.5.0.2/lib/st_core/engine.rb:62:in `block in <class:Engine>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `instance_exec'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:30:in `run'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:55:in `block in run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `call'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/initializable.rb:54:in `run_initializers'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/railties-4.2.6/lib/rails/application.rb:352:in `initialize!'
from /root/RubymineProjects/chop/config/environment.rb:5:in `<top (required)>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
from /root/RubymineProjects/chop/config.ru:3:in `block in <main>'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `instance_eval'
from /root/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `initialize'
I hava a same problem when I create redis-cluster.Finally,I find the problem.
you should do that:
1、gem uninstall redis
2、gem install redis --version 3.0.0
All my delayed jobs are failing for no apparent reason in production on the server. I tried restarting, removing all previous jobs and again starting the job worker, but nothing worked.
Suspecting it must be a capistrano issue, I manually ran rake jobs:work via ssh on the server but I got the same error:
ubuntu#ip-172-31-35-0:~/apps/instano-api/current$ rake jobs:work
[Worker(host:ip-172-31-35-0 pid:22174)] Starting job worker
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) RUNNING
[Worker(host:ip-172-31-35-0 pid:22174)] Job InstanoMailer.new_quote (id=73) FAILED (3 prior attempts) with NameError: undefined method `error' for class `Class'
I use ExceptionNotifier gem, from which I got the following stack trace (but none of the lines are from my application code):
A NameError occurred in background at 2014-12-17 02:43:21 +0530 :
undefined method `error' for class `Class'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
-------------------------------
Backtrace:
-------------------------------
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/performable_method.rb:7:in `method'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:113:in `hook'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:97:in `rescue in block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:101:in `block in invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/backend/base.rb:91:in `invoke_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block (2 levels) in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:82:in `block in timeout'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `catch'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/timeout.rb:70:in `timeout'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:199:in `block in run'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:198:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `block in reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:275:in `reserve_and_run_one_job'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:182:in `block in work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:181:in `work_off'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:146:in `block (4 levels) in start'
/home/ubuntu/.rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:145:in `block (3 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:144:in `block (2 levels) in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `loop'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:143:in `block in start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:79:in `block in add'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:66:in `execute'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/worker.rb:142:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:124:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:112:in `block in run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/controller.rb:70:in `run'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:110:in `run_process'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:91:in `block in daemonize'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `times'
/home/ubuntu/.rvm/gems/ruby-2.1.0/gems/delayed_job-4.0.4/lib/delayed/command.rb:89:in `daemonize'
bin/delayed_job:5:in `<main>'
-------------------------------
Data:
-------------------------------
* data: {}
This seems to be a feature limitation in Delayed Job. Errors are not reported properly. The trace and error message is completely incorrect.
I debugged mine by looking at the rails production log. It was hard to find because there was no error while adding the task to delayed job i.e. I was getting a [INFO] tag instead of [ERROR] in logs.
I hope someone points out how to get proper and meaningful stack traces in case of failed Delayed Jobs.
As a footnote, my error was a real specific (and lame) one. My smtp settings in /config/environments/production.rb was incompatible my mailer so I was getting this error only in production.
I'm trying to push some jobs onto a Sidekiq queue, which up until now, worked just fine. The error only appears in production.
Stack trace:
2013-12-13T20:35:04Z 22616 TID-amwho INFO: Sidekiq client with redis options {}
/home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/pipeline.rb:79:in `finish': EXECABORT Transaction discarded because of previous errors. (Redis::CommandError)
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:121:in `block in call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:245:in `with_reconnect'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis/client.rb:119:in `call_pipeline'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2093:in `block in multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `block in synchronize'
from /home/avishai/.rvm/rubies/ruby-1.9.3-p429/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:36:in `synchronize'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/redis-3.0.6/lib/redis.rb:2085:in `multi'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:159:in `block in raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:in `with'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq.rb:67:in `redis'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:150:in `raw_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:50:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/client.rb:98:in `push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:83:in `client_push'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/sidekiq-2.17.0/lib/sidekiq/worker.rb:40:in `perform_async'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:79:in `send_to_sidekiq'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed_parser.rb:74:in `enqueue'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:235:in `block in parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:24:in `block (2 levels) in each_listing'
from /home/avishai/apps/XXX/shared/bundle/ruby/1.9.1/gems/nokogiri-1.6.0/lib/nokogiri/xml/reader.rb:107:in `each'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:22:in `block in each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `open'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:21:in `each_listing'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/parsers/listings.rb:36:in `parse'
from /home/avishai/apps/XXX/releases/20131213194843/lib/listings_feed/listings_feed.rb:153:in `process!'
from run_feed.rb:74:in `block in <main>'
from run_feed.rb:71:in `each'
from run_feed.rb:71:in `<main>'
I also encountered this error when hitting maxmemory on that redis instance.
You can test it on your dev box by setting maxmemory (ie.. in /usr/local/etc/redis.conf) to sthg small e.g.
maxmemory 3MB
Then fill up your redis with a bunch of data, start Sidekiq, and watch the fireworks
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: {"retry"=>true, "queue"=>"default", "backtrace"=>true, "class"=>"MailWorker", "args"=>["Profile::Created", 868], "jid"=>"d7b0589736818c2ffcd58c90", "enqueued_at"=>1391437950.4245608}
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: EXECABORT Transaction discarded because of previous errors.
2014-02-03T14:32:30Z 59365 TID-ov2cb18yw WARN: /usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/pipeline.rb:76:in `finish'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:121:in `block in call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:243:in `with_reconnect'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis/client.rb:119:in `call_pipeline'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2077:in `block in multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `block in synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:36:in `synchronize'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-3.0.4/lib/redis.rb:2069:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:337:in `namespaced_block'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/redis-namespace-1.3.1/lib/redis/namespace.rb:224:in `multi'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:93:in `block in stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/connection_pool-1.1.0/lib/connection_pool.rb:49:in `with'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq.rb:67:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/util.rb:25:in `redis'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:92:in `stats'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/sidekiq-2.13.1/lib/sidekiq/processor.rb:46:in `block in process'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `public_send'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:25:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/calls.rb:67:in `dispatch'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/future.rb:15:in `block in new'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `call'
/usr/local/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/celluloid-0.14.1/lib/celluloid/internal_pool.rb:59:in `block in create'
I resolved this by doing a FLUSHALL, which cleared out all the data from the redis instance. It appears that somewhere along the way the data got corrupted, and that fixed it.
Another possible reason is a broken master-slave replication. For example with min-slaves-to-write configuration option set:
127.0.0.1:6379> set foo 1
(error) NOREPLICAS Not enough good slaves to write.
To temporarily resolve just execute:
CONFIG SET min-slaves-to-write 0