Don't know how to exactly reproduce. It occurs on our production servers but not every time.
After deploy, we will issuse bundle exec pumactl -S pids/puma.state -F config/puma.rb restart (on Ubuntu 14.04 and Ubuntu 16.04). But from time to time, we encounter that one of the puma servers hangs at restarting for very very long time, that we could only kill -9 and start it again. There's no certain pattern of when/which it will stuck.
Like following, after 15 minutes the restart command is issued, I still sees:
$ ps -ef | grep puma
deployer 2535 6533 99 10:11 ? 08:33:16 puma: cluster worker 3: 6533
deployer 2910 6533 99 10:14 ? 08:31:22 puma: cluster worker 2: 6533
deployer 6533 1 0 01:08 ? 00:00:23 puma 3.8.2 (tcp://0.0.0.0:2801)
deployer 9973 9683 0 18:47 pts/0 00:00:00 grep --color=auto puma
And if I run strace -p 6533 -q -f to attach and check inside puma process, I get this: https://gist.github.com/larryzhao/446234a3af91bec917119494f9bc2384
Just don't know where to look into.
I am running on ruby 2.3.4p301 (2017-03-30 revision 58214) [x86_64-linux] with Rails 5.0.5 and Puma 3.8.2
Related
Two years ago I was running the following command:
bundle exec puma -C config/puma.rb -b unix:/home/user/site/shared/tmp/sockets/user-puma.sock -d
bundle exec sidekiq -d
But now, after some Ubuntu updates, the -d flag is deprecated and I can't start the Puma and Sidekiq processes to run in background.
I also tried running:
bundle exec puma -C config/puma.rb -b unix:/home/user/site/shared/tmp/sockets/user-puma.sock &
bundle exec sidekiq &
This works only while I'm logged on the SSH, when I close the SSH connection, all processes opened with & are closed.
How can I run Puma, Sidekiq and other processes in background as Daemons?
What works for me is setting up my own systemd service for sidekiq
A well-documented example file is in the sidekiq github repository
Along with this, I'd also recommend using monit to watch background processes... here's a recipe from Lugo Labs that I loosely base my deploys off of
I am using monit to monitor my WEB service. Everything went OK until monit process was killed by the system. There is a part of monit log file showing the issue
To have monit automatically launched at startup and restarted on failure I added upstart config for monit (I am running Ubuntu 14.04), that looks like this:
# This is an upstart script to keep monit running.
# To install disable the old way of doing things:
#
# /etc/init.d/monit stop && update-rc.d -f monit remove
#
# then put this script here:
#
# /etc/init/monit.conf
#
# and reload upstart configuration:
#
# initctl reload-configuration
#
# You can manually start and stop monit like this:
#
# start monit
# stop monit
#
description "Monit service manager"
limit core unlimited unlimited
start on runlevel [2345]
stop on starting rc RUNLEVEL=[016]
expect daemon
respawn
exec /usr/bin/monit -c /etc/monitrc
pre-stop exec /usr/bin/monit -c /etc/monitrc quit
When I reboot the system monit is not running.
sudo monit status
$ monit: Status not available -- the monit daemon is not running
How can I configure upstart to keep monit running and monitored?
I had the wrong path to the config file in the start monit command. The correct commands are
exec /usr/bin/monit -c /etc/monit/monitrc
pre-stop exec /usr/bin/monit -c /etc/monit/monitrc quit
It starts monit as daemon and restarts when I kill it:
ps aux | grep monit
root 2173 0.0 0.1 104348 1332 ? Sl 04:13 0:00 /usr/bin/monit -c /etc/monit/monitrc
sudo kill -9 2173
ps aux | grep monit
root 2184 0.0 0.1 104348 1380 ? Sl 04:13 0:00 /usr/bin/monit -c /etc/monit/monitrc
I am using sidekiq gem to run API calls in background. I ran sidekiq in Daemon process like:
bundle exec sidekiq -d
Now I made some changes in my method, so I want to restart sidekiq. I tried to kill the sidekiq by using the below command:
kill -9 process_id
but it's not working. I want to know the command to restart sidekiq process. If you have any idea please share with me.
I tried the below command also:
sidekiqctl stop /path/to/pid file/pids/sidekiq.pid
Start:
$ bundle exec sidekiq -d -P tmp/sidekiq.pid -L log/sidekiq.log
where -d demonize, -P pid file, -L log file.
Stop:
$ bundle exec sidekiqctl stop tmp/sidekiq.pid 0
Sidekiq shut down gracefully.
where 0 is number of seconds to wait until Sidekiq exits.
So after you find you PID, you can use the below commands: the first will stop the workers from getting new jobs and will let existing jobs complete:
kill -USR1 [PID]
after that, you can kill the process using:
kill -TERM [PID]
Also, there is a page on sidekiq/wiki about this called Signals.
[edit]
Here is the signal page.
[edit]
Check video
For finding PIDs one can use:
ps aux | grep sidekiq
To keep the daemon running you should definitely have some good error handling in the HardWorker classes, but you can also use the command below to restart the sidekiq runners if they are not found in the system processes.
x=`ps aux | grep sidekiq | grep -v grep | awk '{print $2}'`; [ "$x" == "" ] && cd /path/to/deploy && bundle exec sidekiq -d -L /path/to/deploy/log/sidekiq.log -C /path/to/deploy/config/sidekiq.yml -e production
This basically looks for the PID using ps aux | grep sidekiq | grep -v grep | awk '{print $2}' and then stores it in variable x. Then, if it's empty, it will run a daemonized sidekiq process.
You can stick this guy into a cron job or something. But if your jobs are failing continually, you'll definitely want to figure out why.
EDIT: Added path to deploy from cron.
Type in following command:
ps -ef | grep sidekiq
This gives process_id and other details of running sidekiq process in background.
Copy process_id and use following command:
kill process_id
Use following command to start sidekiq again in background with -d option:
bundle exec sidekiq -d -L log/sidekiq.log
systemctl {start,stop,restart} sidekiq
Use above command on different option to start stop and restart sidekiq on server
e.g. systemctl restart sidekiq
In Ubuntu, I can run a rails server in the background as a daemon by passing in the --daemon option;
bundle exec puma -e production -b unix:///var/run/my_app.sock --daemon
However, how do I gracefully shut this daemonized process down? It's not a simple matter of crtl + c anymore :)
It is better to use puma control pumactl, it processes monitor and controller.
and then you can use it like this to stop
bundle exec pumactl -P /var/run/puma.pid stop
OR
pumactl -C unix://var/run/my_app_pumactl.sock [status|restart|halt|stop]
I am running my rails application using ruby enterprise edition with unicorn as app server. I run this command
bundle exec unicorn -D -c /home/ubuntu/apps/st/config/unicorn.rb
I need to run this command soon after the system reboots or starts. I am running the app on ubuntu 10.04 LTS EC2 instance. I tried couple of examples which are mentioned on this site as well as this site but it’s not working for me. Any heads up
Try it as an Upstart. To do so, you need to create a myapp.conf file into the directory /etc/init/ with the contents below:
description "myapp server"
start on runlevel [23]
stop on shutdown
exec sudo -u myuser sh -c "cd /path/to/my/app && bundle exec unicorn -D -c /home/ubuntu/apps/st/config/unicorn.rb"
respawn
After that, you should be able to start/stop/restart your app with the commands below:
start myapp
stop myapp
restart myapp
Use ps -aux | grep myapp to check if your app is running.
You can use this file as a template, set appropriate paths mentioned in this file, make it executable and symlink into /etc/init.d/my_unicorn_server. Now you can start the server using:
sudo service my_unicorn_server start
Then you can do:
sudo update-rc.d my_unicorn_server defaults
To startup the unicorn server on system reboot automatically.
In my case, I just wanted it quick so I place the startup command in /etc/rc.local like below. Note that i'm using RVM.
# By default this script does nothing.
cd <your project dir>
/usr/local/rvm/gems/ruby-2.2.1/wrappers/bundle exec unicorn -c <your project dir>/config/unicorn.conf -D
test -e /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server
exit 0
Make sure your startup command is above the exit 0. After you reboot, check whether it is running or not by directly hitting the url of your application or use ps -aux | grep unicorn command.
Note* Previously I use Phusion Passenger but I'm having trouble to see its error log, so I move back to unicorn. I also tried #warantesbr without success, which I guess it fails because my whole environment where setup using root access.
If you are using unicorn_init script
You can configure a cron job to start the unicorn server on reboot
crontab -e
and add
#reboot /bin/bash -l -c 'service unicorn_<your service name> start >> /<path to log file>/cron.log 2>&1'