I have a python/Django project (myproject) running on nginx and uwsgi.
I am running uwsgi command via supervisord. This works perfectly, but on restarting supervisord it creates zombie process. what am i doing wrong? What am I overlooking to do this cleanly? any Advise?
Often times supervisor service takes too long. at that point I have found the following in supervisor.log file
INFO waiting for stage2_BB_wsgi, stage3_BB_wsgi, stage4_BB_wsgi to die
Point to Note: I am running multiple staging server in one machine, namely stage2 .. stageN
supervisor.conf file extract
[program:stage2_BB_wsgi]
command=uwsgi --close-on-exec -s /home/black/stage2/shared_locks/uwsgi_bb.sock --touch-reload=/home/black/stage2/shared_locks/reload_uwsgi --listen 10 --chdir /home/black/stage2/myproject/app/ --pp .. -w app.wsgi -C666 -H /home/black/stage2/myproject/venv/
user=black
numprocs=1
stdout_logfile=/home/black/stage2/logs/%(program_name)s.log
stderr_logfile=/home/black/stage2/logs/%(program_name)s.log
autostart=true
autorestart=true
startsecs=10
exitcodes=1
stopwaitsecs=600
killasgroup=true
priority=1000
thanks in advance.
You will want to set your stopsignal to INT or QUIT.
By default supervisord sends out a SIGTERM when restarting a program. This will not kill uwsgi, only reload it and its workers.
Related
Running COS on GCE
Any ideas on how to get COS to do a graceful docker shutdown?
My innermost process is celery, which says he wants a SIGTERM to stop gracefully
http://docs.celeryproject.org/en/latest/userguide/workers.html#stopping-the-worker
My entrypoint is something like
exec celery -A some_app worker -c some_concurrency
On COS I am running my docker a service, something like
write_files:
- path: /etc/systemd/system/servicename.service
permissions: 0644
owner: root
content: |
[Unit]
Description=Some service
[Service]
Environment="HOME=/home/some_home"
RestartSec=10
Restart=always
ExecStartPre=/usr/share/google/dockercfg_update.sh
ExecStart=/usr/bin/docker run -u 2000 --name=somename --restart always some_image param_1 param_2
ExecStopPost=/usr/bin/docker stop servicename
KillMode=processes
KillSignal=SIGTERM
But ultimately when my COS instance it shut down, it just yanks the plug.
Do I need to add a shutdown script to do a docker stop? Do I need to do something more advanced?
What is the expected exit status of your container process when when it receives SIGTERM?
Running systemctl stop <service> then systemctl status -l <service> should show the exit code of the main process. Example:
Main PID: 21799 (code=exited, status=143)
One possibility is that the process does receive SIGTERM and shuts down gracefully, but returns non-zero exit code.
This would make the systemd believe that it didn't shutdown correctly. If that is the case, adding
SuccessExitStatus=143
to your systemd service should help. (Replace 143 with the actual exit code of your main process.)
This is my centos uwsgi service setting:
[Unit]
Description=uWSGI for uwsgi
After=syslog.target
[Service]
Restart=always
ExecStart=/usr/share/nginx/ENV/bin/uwsgi --ini /usr/share/nginx/ENV/config/uwsgi.ini
StandardError=syslog
KillSignal=SIGQUIT
Type=forking
PIDFile=/var/run/uwsgi.pid
[Install]
WantedBy=multi-user.target
And I want to convert to use supervisor to start/stop the uwsgi service
But still not find a solution
Please help me
This is my supervisor.conf :
[program:wiarea-positioning]
command = /usr/share/nginx/ENV/bin/uwsgi --ini /usr/share/nginx/ENV/config/uwsgi.ini
stdout_logfile=/var/log/uwsgi.log
stderr_logfile=/var/log/uwsgi.log
;stopasgroup = true
stopsignal=QUIT
This is my uwsgi.ini
[uwsgi]
chdir = /usr/share/nginx/ENV/mysite
env = DJANGO_SETTINGS_MODULE=mysite.settings
module = mysite.wsgi:application
# the virtualenv
home = /usr/share/nginx/ENV
master = true
thunder-lock=true
processes = 4
pidfile = /var/run/uwsgi.pid
socket = 127.0.0.1:8001
daemonize = /var/log/uwsgi.log
vacuum = true
I think your problem (at least one of them) is this uwsgi.ini line:
daemonize = /var/log/uwsgi.log
Remember that supervisor basically just runs your command= command from the command line, and waits for it to exit. If it exits, supervisor runs the command again.
The uwsgi daemonize option breaks this, because it causes the main uwsgi command to start a background process and immediately exit. Supervisor doesn't know about the background process, so it assumes the command failed and tries to restart it repeatedly. You can confirm this is what's happening by looking at the log files in the /var/log/supervisor/ folder.
So, if you want to run uwsgi with supervisor, you need to remove the daemonize option. After that, you can try just running the command from the command line to confirm that uwsgi starts and stays in the foreground.
This blog has more discussion of daemon processes and supervisor:
I use a supervisor to run uWSGI application. Why uWSGI application does not always stop after stop supervisor?
supervisor config:
[program:test]
autostart = true
user=root
command=uwsgi --master --workers 5 --disable-logging --socket 127.0.0.1:8888
--module web --callable app
priority=1
redirect_stderr=true
stdout_logfile = /data/log
By default supervisor send SIGTERM on stop. SIGTERM in uWSGI means 'brutal reload'.
You have to change it to QUIT or INT:
stopsignal=QUIT
should be enough
Another approach (discouraged) is adding --die-on-term to uWSGI command line to change its default behaviour
project supervisor config file
add stopsignal=INT
project uwsgi config file
remove daemonize=xxx.log to disable daemon mode
If you use "processes = #" into your uwsgi configuration, you also must use "master = true". If not, supervisor only will kill one of workers.
Then:
/etc/supervisor/conf.d/app.conf
stopsignal = QUIT
/etc/uwsgi/app.ini
processes = 4
master = true
If you are running your UWSGI with master and workers you need to add in your /etc/supervisor/conf.d/app.conf file
stopasgroup=false
killasgroup=false
or else no matter what stopping uwsgi will spawn more master and so is workers.
I'm sending a USR2 signal to the master process in order to achieve zero downtime deploy with unicorn. After the old master is dead, I'm getting the following error:
adding listener failed addr=/path/to/unix_socket (in use)
unicorn-4.3.1/lib/unicorn/socket_helper.rb:140:in `initialize':
Address already in use - /path/to/unix_socket (Errno::EADDRINUSE)
The old master is killed in the before_fork block on the unicorn.rb config file. The process is started via upstart without the daemon (-D) option.
Any Ideia on what's going on?
Well, turns out you have to run in daemonized mode (-D) if you want to be able to do zero downtime deployment. I changed a few things in my upstart script and now it works fine:
setuid username
pre-start exec unicorn_rails -E production -c /path/to/app/config/unicorn.rb -D
post-stop exec kill cat `/path/to/app/tmp/pids/unicorn.pid`
respawn
I have a thin cluster set up to start 3 servers:
/etc/thin/myapp.yml
...
wait: 30
servers: 3
daemonize: true
...
and the I use thin restart -C /etc/thin/myapp.yml to restart. However, I would like to restart each server at a time, to reduce downtime.
Is there a way to restart each server by pid number or location for example?
There is something better for you
try option: --onebyone
you may also add the following line to your config file
onebyone: true
afterwards you able to restart you thin cluster without any downtime.
I know the question has been answered, but I'd like to add the -o option to the mix.
So
thin restart -C /etc/thin/myapp.yml -o 3000
Will only start the server running on port 3000. If let's say you have two other servers running on 3001 and 3002, they'll be left untouched.
-o works with start and stop commands too.