I'm trying to use Monit to watch a Thin Rails application server process. Here is my Monit config:
check process thin-3000
with pidfile /var/www/apps/myapp/shared/pids/thin.3000.pid
start program = "/bin/su - deploy -c 'thin start -C /etc/thin/myapp.yml -o 3000'"
stop program = "/bin/su - deploy -c 'thin stop -C /etc/thin/myapp.yml -o 3000'"
if failed port 3000 then restart
group thin
This actually works. If I kill the Thin server process, Monit will faithfully restart it. However, if I watch the Monit log it keeps outputting the following over and over again:
[UTC Jan 14 23:01:04] error : 'thin-3000' process is not running
[UTC Jan 14 23:01:04] info : 'thin-3000' trying to restart
[UTC Jan 14 23:01:04] info : 'thin-3000' start: /bin/su
[UTC Jan 14 23:01:34] error : 'thin-3000' failed to start
It looks like whatever mechanism it's using to check if the process is running isn't working correctly. Do you know what I might be doing wrong?
Related
I configured Monit to watch unicorn and restart it when the memory exceeded or the cpu increased above a certain limit ,
how ever when it happens , monit doesn't restart unicorn , and here is the logs I found in monit log file
[UTC Aug 11 20:15:41] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:16:11] error : 'unicorn_myapp' process is not running
[UTC Aug 11 20:16:11] info : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:16:11] info : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:16:42] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:17:12] error : 'unicorn_myapp' process is not running
[UTC Aug 11 20:17:12] info : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:17:12] info : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:17:42] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:18:12] error : 'unicorn_myapp' process is not running
Here is my monit configuration under /etc/monit/conf.d/
check process unicorn_myapp
with pidfile /home/ubuntu_user/apps/myapp/current/tmp/pids/unicorn.pid
start program = "/etc/init.d/unicorn_myapp start"
stop program = "/etc/init.d/unicorn_myapp stop"
restart program = "/etc/init.d/unicorn_myapp restart"
if not exist then restart
if mem is greater than 300.0 MB for 2 cycles then restart # eating up memory?
if cpu is greater than 50% for 4 cycles then restart # send an email to admin
if cpu is greater than 80% for 30 cycles then restart # hung process?
group unicorn
it should restart unicorn when such error happens which break the app
from unicorn.log file
ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
when I run /etc/init.d/unicorn_myapp restart from terminal it works
Monit mostly uses the restart program to start a program. I do not know why this is, but I also observed this behavior.
Just try to comment out the "restart" line. This should force monit to run the start script, what should not try to kill an existing process.
You might also want to watch your log-file like
CHECK FILE unicorn_log PATH log__file__path___change_me_or_the_world_will_flatten
start program = "/etc/init.d/unicorn_myapp start"
stop program = "/etc/init.d/unicorn_myapp stop"
# No log File entry for one hour?
if timestamp is older than 1 hour then restart
# Allocate Memery error?
if content = "Cannot allocate memory" then restart
I have the following shell script that allows me to start my rails app, let's say it's called start-app.sh:
#!/bin/bash
cd /var/www/project/current
. /home/user/.rvm/environments/ruby-2.3.3
RAILS_SERVE_STATIC_FILES=true RAILS_ENV=production nohup bundle exec rails s -e production -p 4445 > /var/www/project/log/production.log 2>&1 &
the file above have permissions of:
-rwxr-xr-x 1 user user 410 Mar 21 10:00 start-app.sh*
if i want to check the process I do the following:
ps aux | grep -v grep | grep ":4445"
it'd give me the following output:
user 2960 0.0 7.0 975160 144408 ? Sl 10:37 0:07 puma 3.12.0 (tcp://0.0.0.0:4445) [20180809094218]
P.S: the reason i grep ":4445" is because i have few processes running on different ports. (for different projects)
now coming to monit, i used apt-get to install it, and the latest version from repo is 5.16, as i'm running on Ubuntu 16.04, also note that monit is running as root, that's why i specified the gid uid in the following. (because the start script is used to be executed from "user" and not "root")
Here's the configuration for monit:
set daemon 20 # check services at 20 seconds interval
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set eventqueue
basedir /var/lib/monit/events # set the base directory where events will be stored
slots 100 # optionally limit the queue size
set mailserver xx.com port xxx
username "xx#xx.com" password "xxxxxx"
using tlsv12
with timeout 20 seconds
set alert xx#xx.com
set mail-format {
from: xx#xx.com
subject: monit alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
}
set limits {
programOutput: 51200 B
sendExpectBuffer: 25600 B
fileContentBuffer: 51200 B
networktimeout: 10 s
}
check system $HOST
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if cpu usage > 90% for 10 cycles then alert
if memory usage > 85% then alert
if swap usage > 35% then alert
check process nginx with pidfile /var/run/nginx.pid
start program = "/bin/systemctl start nginx"
stop program = "/bin/systemctl stop nginx"
check process redis
matching "redis"
start program = "/bin/systemctl start redis"
stop program = "/bin/systemctl stop redis"
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
include /etc/monit/conf.d/*
include /etc/monit/conf-enabled/*
Now monit, is detecting and alerting me when the process goes down (if i kill it manually) and when it's manually recovered, but it won't start that shell script automatically.. and according to /var/log/monit.log, it's showing the following:
[UTC Aug 13 10:16:41] info : Starting Monit 5.16 daemon
[UTC Aug 13 10:16:41] info : 'production-server' Monit 5.16 started
[UTC Aug 13 10:16:43] error : 'myapp' process is not running
[UTC Aug 13 10:16:46] info : 'myapp' trying to restart
[UTC Aug 13 10:16:46] info : 'myapp' start: /bin/bash
[UTC Aug 13 10:17:17] error : 'myapp' failed to start (exit status 0) -- no output
So far what I see when monit tries to execute the script is that it tries to load it (i can see it for less than 3 seconds using ps aux | grep -v grep | grep ":4445", but this output is different from the above output i showed up, it shows the content of the shell script being executed and specifically this one:
blablalba... nohup bundle exec rails s -e production -p 4445
and then it disappears. then it tries to re-execute the shell.. again and again...
What am I missing, and what is wrong with my configuration? note that I can't change anything in the start-app.sh because it's on production and working 100%. (i just want to monitor it)
Edit: To my understanding and experience, it seems to be a Environment Variable issue or path issue, but i'm not sure how to solve it, it doesn't make any sense to put the env variables inside monit .. what if someone else wanted to edit that shell script or add something new? i hope you get my point
As i expected, it was user-environment issue and i solved it by editing monit configuration as below:
Before (not working)
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
After (working)
check process myapp
matching ":4445"
start program = "/bin/su -s /bin/bash -c '/home/user/start-app.sh' user"
stop program = "/bin/su -s /bin/bash -c '/home/user/stop-app.sh' user"
Explanation: i removed (uid and gid) as "user" from monit because it will only execute the shell script in the name of "user" but it won't get/import/use user's env path, or env variables.
So I have puma_start_staging.sh file. If I execute that file from terminal
. /home/deploy/puma_start_staging.sh
everything is ok. But monit can't execute that file.
Here's my config:
check process puma_staging with pidfile /home/deploy/apps/staging/shared/tmp/pids/puma.pid
start program = "/bin/bash -c '. /home/deploy/puma_start_staging.sh'"
stop program = "/bin/bash -c 'kill `cat /home/deploy/apps/staging/shared/tmp/pids/puma.pid`'"
I get this in monit.log
[EEST Aug 4 10:56:10] info : 'puma_staging' start: /bin/bash
[EEST Aug 4 10:56:40] error : 'puma_staging' failed to start
[EEST Aug 4 10:56:40] info : 'puma_staging' start action done
What am I doing wrong?
There are many things I do not understand, so my question may be silly.
I want to run a puma ror server as a systemd service at centos 7. Use ruby installed using rvm.
My puma_test.service file is:
[Unit]
Description=Puma application server
After=network.target
[Service]
WorkingDirectory=/var/www/test_app
Environment=RAILS_ENV=development
PIDFile=/var/www/shared/pids/puma.pid
ExecStart=/usr/local/rvm/gems/ruby-2.2.1/gems/bundler-1.9.4/bin/bundle exec puma -e development -b unix:///var/www/shared/pids/puma.sock --pidfile /var/www/shared/pids/puma.pid
[Install]
WantedBy=multi-user.target
but when I run it, it does not work. I get error (from journalctl):
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Starting Puma application server...
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Started Puma application server.
kwi 18 22:56:15 vps150852.ovh.net bundle[2072]: /usr/bin/env: ruby: No such file or directory
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: puma_test.service: main process exited, code=exited, status=127/n/a
kwi 18 22:56:15 vps150852.ovh.net systemd[1]: Unit puma_test.service entered failed state.
when I run i /usr/www/test_app
/usr/local/rvm/gems/ruby-2.2.1/gems/bundler-1.9.4/bin/bundle exec puma -e development -b unix:///var/www/shared/pids/puma.sock --pidfile /var/www/shared/pids/puma.pid
everything works fine, but I am probably doing something wrong
Looks like you need to load rvm when you run your task. systemd run in shell, not in bash, your bashrc will not be loaded
I'm sending a USR2 signal to the master process in order to achieve zero downtime deploy with unicorn. After the old master is dead, I'm getting the following error:
adding listener failed addr=/path/to/unix_socket (in use)
unicorn-4.3.1/lib/unicorn/socket_helper.rb:140:in `initialize':
Address already in use - /path/to/unix_socket (Errno::EADDRINUSE)
The old master is killed in the before_fork block on the unicorn.rb config file. The process is started via upstart without the daemon (-D) option.
Any Ideia on what's going on?
Well, turns out you have to run in daemonized mode (-D) if you want to be able to do zero downtime deployment. I changed a few things in my upstart script and now it works fine:
setuid username
pre-start exec unicorn_rails -E production -c /path/to/app/config/unicorn.rb -D
post-stop exec kill cat `/path/to/app/tmp/pids/unicorn.pid`
respawn