I have multiple workers, each of them seperated based on WorkerClass
Specified queues, concurrency etc. They are triggerin via cron jobs.
nohup bundle exec sidekiq -q worker1 -c 5 -e production
nohup bundle exec sidekiq -q worker2 -c 5 -e production
nohup bundle exec sidekiq -q worker3 -c 5 -e production
nohup bundle exec sidekiq -q worker4 -c 5 -e production
nohup bundle exec sidekiq -q worker5 -c 5 -e production
So, I need to start all of them on EC2 instance, and restart them after next deploy.
Can I use capistrano to do this? or any better way?
Thanks.
You should use something like God, upstart, or runit to keep your sidekiq processes up and running.
Related
Below is an example set up of a Procfile using Sidekiq:
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq
To define specific queues for a worker, the -q flag is used to do the inclusion.
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -q low_priority
My question: is there a way to do an exclusion of a queue for a worker instead?
My use case is the following:
General worker that handles all the the queues except one (lets call it resource_intensive)
A specialized worker that has more resources behind it to handle the resource_intensive queue only.
The set-up using only the inclusion -q flag would be something like the below, where low_priority, high_priority, default, and resource_intensive are the queues which exist:
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -q high_priority -q default -q low_priority
resource_intensive_worker: bundle exec sidekiq -q resource_intensive
A less error prone set-up would just exclude the resource_intensive queue from worker. Does a queue exclusion flag exist?
Two years ago I was running the following command:
bundle exec puma -C config/puma.rb -b unix:/home/user/site/shared/tmp/sockets/user-puma.sock -d
bundle exec sidekiq -d
But now, after some Ubuntu updates, the -d flag is deprecated and I can't start the Puma and Sidekiq processes to run in background.
I also tried running:
bundle exec puma -C config/puma.rb -b unix:/home/user/site/shared/tmp/sockets/user-puma.sock &
bundle exec sidekiq &
This works only while I'm logged on the SSH, when I close the SSH connection, all processes opened with & are closed.
How can I run Puma, Sidekiq and other processes in background as Daemons?
What works for me is setting up my own systemd service for sidekiq
A well-documented example file is in the sidekiq github repository
Along with this, I'd also recommend using monit to watch background processes... here's a recipe from Lugo Labs that I loosely base my deploys off of
We are currently using capistrano-sidekiq, this also handles the automatic restart of sidekiq on each deploy.
However, we also want to manually restart sidekiq each night, without rebooting the whole application.
Which command should be used to automatically restart sidekiq on the production machine using a crontab entry on this machine ?
Find existing sidekiq process and kill it and then start sidekiq again
To run Sidekiq
RAILS_ENV=production bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml
To Kill sidekiq process
RAILS_ENV=production bundle exec sidekiqctl stop tmp/pids/sidekiq.pid
I try to run sidekiq service with supervisord
Here's my config:
[program:my-app-sidekiq-staging]
directory=/srv/www/DOMAIN/current
command=RAILS_ENV=production /usr/local/rvm/bin/rvm ruby-2.2.2#my-app-staging do bundle exec sidekiq -e production -d -C config/sidekiq.yml -L log/sidekiq.log
autostart=true
autorestart=true
redirect_stderr=true
After startup I’m have fatal error:
can't find command 'RAILS_ENV=production'
I’m confused, because my config for rails runs without errors
directory=/srv/www/DOMAIN/current
command=RAILS_ENV=production /usr/local/rvm/bin/rvm ruby-2.2.2#my-app-staging do bundle exec passenger start -S tmp/unicorn/ilp-app-unicorn.sock --environment production --friendly-error-pages
Your environment should not be set in the command but in a separate environment value.
environment=RAILS_ENV=production
See this question.
Supervisor and Environment Variables
In Ubuntu, I can run a rails server in the background as a daemon by passing in the --daemon option;
bundle exec puma -e production -b unix:///var/run/my_app.sock --daemon
However, how do I gracefully shut this daemonized process down? It's not a simple matter of crtl + c anymore :)
It is better to use puma control pumactl, it processes monitor and controller.
and then you can use it like this to stop
bundle exec pumactl -P /var/run/puma.pid stop
OR
pumactl -C unix://var/run/my_app_pumactl.sock [status|restart|halt|stop]