I am currently running sidekiq 4.1.2. I've never managed to be able to run more than a handfull of jobs concurrently. Recently it's been looking like I've run into an issue described in the Sidekiq's Troubleshooting WIKI called Too many connections to MongoDB. Apparently, mongoid 3 doesn't properly disconnect workers. However, I am using mongoid 5.1.3.
My issue surfaces when a job, while a few other jobs are running, tries to hit the database with a query:
Timeout::Error: Timed out attempting to dequeue connection after 30 sec.
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `loop'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:62:in `block in dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `synchronize'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:51:in `checkout'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:107:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/context.rb:63:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/operation/executable.rb:34:in `execute'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:80:in `send_initial_query'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:41:in `block in each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `call'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `read_with_retry'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:39:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/query_cache.rb:207:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:121:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual.rb:20:in `map'
/home/me/applications/myapp/releases/20160618143407/app/jobs/myjob.rb:8:in `block in perform'
After one job fails, the other jobs fail soon after. This most often happens after a few handfull of jobs have finished succesfully, which could indicate that those jobs don't disconnect from the database. Looking at top I can't see that mongo load cpu is too much.
At the same time this started to occur, I noticed that my sidetiq 0.7.0 enabled recurring jobs were not scheduled properly. One job has stopped being queued, and others are only queued once after restart.
According to my Sidekiq web interface I have 1 queue, called default, with 25 threads. Max. 12-15 of them get busy at the same time.
Any idea how to troubleshoot this issue?
The default max pool queue size is 5. Bumping max_pool_size up to e.g. 25 will enable more connections to your db.
production:
clients:
default:
options:
max_pool_size: 25
Change adding wait_queue_timeout attribute in mongoid.yml production config:
production:
clients:
default:
uri: mongodb://aaaaa.com:27017/mongo
options:
connect_timeout: 30
wait_queue_timeout: 30
Related
Rails server fails to start in Discourse project either in development or production. Below are the logs when trying to start the server in dev mode. The application was installed and has been working, It's deployed on AWS in production mode and restarting the unicorn loads the application for some time and again the url stops responding with error messages.
Development logs from $rails s
rb t#ip-XXX-XX-XX-XX-app:/var/www/discourse# vi
config/environments/development.r
root#ip-172-31-25-46-app:/var/www/discourse# rails s
=> Booting Puma
=> Rails 5.1.4 application starting in production
=> Run `rails server -h` for more startup options
Exiting
bundler: failed to load command: script/rails (script/rails)
Redis::CommandError: ERR Error running script (call to f_b06356ba4628144e123b652c99605b873107c9be): #user_script:14: #user_script: 14: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis/client.rb:121:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2399:in `block in _eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
/usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2398:in `_eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2450:in `evalsha'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus/backends/redis.rb:380:in `cached_eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus/backends/redis.rb:140:in `publish'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus.rb:248:in `publish'
/var/www/discourse/lib/distributed_cache.rb:72:in `publish'
**Production logs **
/var/www/discourse/lib/demon/base.rb:109:in `ensure_running'
/var/www/discourse/lib/demon/base.rb:34:in `block in ensure_running'
/var/www/discourse/lib/demon/base.rb:33:in `each'
/var/www/discourse/lib/demon/base.rb:33:in `ensure_running'
config/unicorn.conf.rb:145:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:284:in `join'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
E, [2018-01-04T08:43:37.949928 #60] ERROR -- : reaped #<Process::Status: pid 5870 exit 1> worker=unknown
Detected dead worker 5870, restarting...
Loading Sidekiq in process id 5883
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 4 Redis::CommandError (MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.)
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:121:in `call' web-exception
Redis logs
47:M 17 Jan 09:38:01.070 # Can't save in background: fork: Cannot allocate memory
47:M 17 Jan 09:38:07.087 * 10000 changes in 60 seconds. Saving...
The issue has been fixed, Edited this file
/etc/sysctl.conf and added the line at last
vm.overcommit_memory=1
After this restarted sysctl.
$ sudo sysctl -p /etc/sysctl.conf
Redis doesn't need the amount of memory which OS thinks, the status 1 means always overcommit, never check.
More details can be found from Redis Docs.
I have installed the que gem and I would like to run the tasks I'm scheduling with active job, so far I have added in config/application.rb
config.active_job.queue_adapter = :que
and it stores the jobs in the database correctly but when I try to run
rake que:work
It starts but jobs are not working and I have reviewed run_at field, also tried running setting the env to RAIls_ENV=development. Tried with
que ./config/application.rb
and generates and error
I, [2016-06-15T17:30:27.592632 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"worker_count_change","value":"4"}
I, [2016-06-15T17:30:27.592851 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"mode_change","value":"async"}
/home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:109:in `connection_pool': ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished)
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:54:in `checkout_activerecord_adapter'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:7:in `checkout'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/job.rb:85:in `work'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:80:in `block in work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:19:in `block in initialize'
A bit late, but Que requires you to be running a Postgres Server for it work.
I would imagine this is why you are having issues.
My application is running on same server for different environments with sidekiq. I am facing mismatching of sidekiq process in all environments.
So i need to use different database for development/staging/testing environment of same server. How to configure redis to run different database for each environment.
For multiple Redis Database for each environment in sidekiq
add the below line in config/environment/env.rb as
require 'sidekiq'
Sidekiq.configure_client do |config|
config.redis = { :namespace => 'xxx', :url => 'redis://127.0.0.1:6379/2' }
end
Sidekiq.configure_server do |config|
config.redis = { :namespace => 'xxx', :url => 'redis://127.0.0.1:6379/2' }
end
So the /2 will define the redis database. create seperate database for each environment by changing the value /3, /4 and so on
We've used separate sidekiq queues for each environment. It was a bit tricky to get it set up right, but it works.
Then of course you have to run separate workers for each environment, each accessing it's own environment's queue (and it's own environment's database.)
We started doing this because of time we wasted on "integration" tests failing because we had development workers accessing the queue instead of test workers. There may be other ways but this seemed simplest
I tried /2, /3, /4, etc. but sidekiq does not start. If I rollback to /0, it works perfectly.
2022-11-12T02:37:02.601Z pid=8 tid=gx4 INFO: Booting Sidekiq 6.4.2 with redis options {:url=>"redis://xxxxxxxxxxxxxxxxxxxxxxxx.cache.amazonaws.com:6379/2"}
{"severity":"WARN","time":"2022-11-12T02:37:02.608+00:00","message":"Redis::CannotConnectError: Redis client could not connect to any cluster nodes"}
{"severity":"WARN","time":"2022-11-12T02:37:02.608+00:00","message":"/usr/local/bundle/gems/redis-4.6.0/lib/redis/cluster/slot_loader.rb:22:in `load'\n/usr/local/bundle/gems/redis-4.6.0/lib/redis/cluster.rb:116:in `fetch_cluster_info!'\n/usr/local/bundle/gems/redis-4.6.0/lib/redis/cluster.rb:26:in `initialize'\n/usr/local/bundle/gems/redis-4.6.0/lib/redis.rb:84:in `new'\n/usr/local/bundle/gems/redis-4.6.0/lib/redis.rb:84:in `initialize'\n/App/config/initializers/redis_cache.rb:8:in `new'\n/App/config/initializers/redis_cache.rb:8:in `\u003cmain\u003e'\n/usr/local/bundle/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:39:in `load'\n/usr/local/bundle/gems/bootsnap-1.11.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:39:in `load'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/engine.rb:667:in `block in load_config_initializer'\n/usr/local/bundle/gems/activesupport-7.0.3/lib/active_support/notifications.rb:208:in `instrument'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/engine.rb:666:in `load_config_initializer'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/engine.rb:620:in `block (2 levels) in \u003cclass:Engine\u003e'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/engine.rb:619:in `each'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/engine.rb:619:in `block in \u003cclass:Engine\u003e'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:32:in `instance_exec'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:32:in `run'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:61:in `block in run_initializers'\n/usr/local/lib/ruby/3.1.0/tsort.rb:228:in `block in tsort_each'\n/usr/local/lib/ruby/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'\n/usr/local/lib/ruby/3.1.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'\n/usr/local/lib/ruby/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from'\n/usr/local/lib/ruby/3.1.0/tsort.rb:421:in `block in each_strongly_connected_component_from'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:50:in `each'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:50:in `tsort_each_child'\n/usr/local/lib/ruby/3.1.0/tsort.rb:415:in `call'\n/usr/local/lib/ruby/3.1.0/tsort.rb:415:in `each_strongly_connected_component_from'\n/usr/local/lib/ruby/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component'\n/usr/local/lib/ruby/3.1.0/tsort.rb:347:in `each'\n/usr/local/lib/ruby/3.1.0/tsort.rb:347:in `call'\n/usr/local/lib/ruby/3.1.0/tsort.rb:347:in `each_strongly_connected_component'\n/usr/local/lib/ruby/3.1.0/tsort.rb:226:in `tsort_each'\n/usr/local/lib/ruby/3.1.0/tsort.rb:205:in `tsort_each'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/initializable.rb:60:in `run_initializers'\n/usr/local/bundle/gems/railties-7.0.3/lib/rails/application.rb:372:in `initialize!'\n/App/config/environment.rb:5:in `\u003ctop (required)\u003e'\n/usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/cli.rb:273:in `require'\n/usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/cli.rb:273:in `boot_application'\n/usr/local/bundle/gems/sidekiq-6.4.2/lib/sidekiq/cli.rb:37:in `run'\n/usr/local/bundle/gems/sidekiq-6.4.2/bin/sidekiq:31:in `\u003ctop (required)\u003e'\n/usr/local/bundle/bin/sidekiq:25:in `load'\n/usr/local/bundle/bin/sidekiq:25:in `\u003ctop (required)\u003e'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:58:in `load'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:58:in `kernel_load'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli/exec.rb:23:in `run'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli.rb:483:in `exec'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli.rb:31:in `dispatch'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/cli.rb:25:in `start'\n/usr/local/bundle/gems/bundler-2.3.10/exe/bundle:48:in `block in \u003ctop (required)\u003e'\n/usr/local/bundle/gems/bundler-2.3.10/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'\n/usr/local/bundle/gems/bundler-2.3.10/exe/bundle:36:in `\u003ctop (required)\u003e'\n/usr/local/bundle/bin/bundle:25:in `load'\n/usr/local/bundle/bin/bundle:25:in `\u003cmain\u003e'"}
My use case: I want to share the same redis instance between multiple test applications.
I have a really simple job:
class MyJob
#queue = :high
def self.perform(user_id)
user = User.find(user_id)
MyMailer.send_email(user).deliver
end
end
If I run it manually MyJob.perform(some_id)it works perfect. However, when Rescue is executing it, it returns this error:
Exception
Errno::ENOENT Error
No such file or directory - getcwd
shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `expand_path' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `initialize' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `new' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `instances' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:16:in `<class:LookupContext>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:12:in `<module:ActionView>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:5:in `<top (required)>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/abstract_controller/view_paths.rb:45:in `lookup_context' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:456:in `process' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:452:in `initialize' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `new' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `method_missing' releases/1111111111111/app/jobs/my_job.rb:6:in `perform'
Have any ideas why might be happening?
Thanks!
Yes Sky. You are right about it needing to be restarted.
Some people received this error after trying to run from an already deleted directory.
I received this error after switching databases and leaving the server running. The old server info was still showing up but I was getting this error. Restarted my rails server and everything works fine with the new db.
Basically it means that there is a significant state change on the server, and your environment needs to be reset/restarted.
I started having this same issue on my production environment. After some investigation I found that this was caused by my resque workers failing to be properly restarted on each capistrano deployment.
I am running systemu from within a delayed_job process. I had a job fail, and found this error in my delayed_jobs table:
execution expired
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `close'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `popen'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `block (2 levels) in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:168:in `quietly'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:75:in `block in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `call'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `block in tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `loop'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:68:in `systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:10:in `systemu'
It was working fine before, but this is a larger job I'm running. Is there some timeout that I need to set?
Thanks!
OK the problem was that the delayed_job timed out. I had to up the time limit with:
Delayed::Worker.max_run_time = 48.hours #2 days
in app/config/initializers/delayed_job_config.rb