So I have puma_start_staging.sh file. If I execute that file from terminal
. /home/deploy/puma_start_staging.sh
everything is ok. But monit can't execute that file.
Here's my config:
check process puma_staging with pidfile /home/deploy/apps/staging/shared/tmp/pids/puma.pid
start program = "/bin/bash -c '. /home/deploy/puma_start_staging.sh'"
stop program = "/bin/bash -c 'kill `cat /home/deploy/apps/staging/shared/tmp/pids/puma.pid`'"
I get this in monit.log
[EEST Aug 4 10:56:10] info : 'puma_staging' start: /bin/bash
[EEST Aug 4 10:56:40] error : 'puma_staging' failed to start
[EEST Aug 4 10:56:40] info : 'puma_staging' start action done
What am I doing wrong?
Related
After starting a SCADA LTS Docker container as suggested on https://github.com/SCADA-LTS/Scada-LTS with the following command:
docker run -it -e DOCKER_HOST_IP=docker-machine ip-p 81:8080 scadalts/scadalts /root/start.sh
...The container works well for some time and then suddenly a "HTTP Status 404" error is shown, like the following:
http://[IP]/ScadaBR/
HTTP Status 404 - /ScadaBR/
type Status report
message /ScadaBR/
description The requested resource is not available.
Apache Tomcat/7.0.85
Where [IP] is the default Docker IP address and port, most of the times is localhost:81.
Any idea how to solve it?
Thank you in advance!
TL;DR
After some time running the MySQLservice dies. Is necessary to restart it manually with this:
docker exec scada service mysql restart
docker exec scada killall tail
DETAILED REPORT
When the error is shown, you can check if all the services are running on the container (in this case named 'scada'):
>docker exec scada ps -A
PID TTY TIME CMD
1 ? 00:00:00 start.sh
790 ? 01:00:22 java
791 ? 00:01:27 tail
858 ? 00:00:00 ps
As can be seen, no MySQL service is running. This explains why Tomcat is running but SCADA-LTS don't.
You can restart MySQL service inside the container with:
docker exec scada service mysql restart
After that SCADA-LTS is still down and you have to restart tomcat which can be done in this way:
docker exec scada killall tail
After a minute or less, all the services are running:
>docker exec scada ps -A
PID TTY TIME CMD
1 ? 00:00:00 start.sh
43 ? 00:00:00 mysqld_safe
398 ? 00:00:00 mysqld
481 ? 00:00:31 java
482 ? 00:00:00 sleep
618 ? 00:00:00 ps
Now SCADA-LTS is running!
I configured Monit to watch unicorn and restart it when the memory exceeded or the cpu increased above a certain limit ,
how ever when it happens , monit doesn't restart unicorn , and here is the logs I found in monit log file
[UTC Aug 11 20:15:41] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:16:11] error : 'unicorn_myapp' process is not running
[UTC Aug 11 20:16:11] info : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:16:11] info : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:16:42] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:17:12] error : 'unicorn_myapp' process is not running
[UTC Aug 11 20:17:12] info : 'unicorn_myapp' trying to restart
[UTC Aug 11 20:17:12] info : 'unicorn_myapp' restart: '/etc/init.d/unicorn_myapp restart'
[UTC Aug 11 20:17:42] error : 'unicorn_myapp' failed to restart (exit status 127) -- '/etc/init.d/unicorn_myapp restart': /etc/init.d/unicorn_myapp: 27: kill: No such process
Couldn't reload, starting 'cd /home/ubuntu_user/apps/myapp/current; bundle exec unicorn -D -c /home/ubuntu_user/apps/myapp/shared/config/unicorn.rb -E pr
[UTC Aug 11 20:18:12] error : 'unicorn_myapp' process is not running
Here is my monit configuration under /etc/monit/conf.d/
check process unicorn_myapp
with pidfile /home/ubuntu_user/apps/myapp/current/tmp/pids/unicorn.pid
start program = "/etc/init.d/unicorn_myapp start"
stop program = "/etc/init.d/unicorn_myapp stop"
restart program = "/etc/init.d/unicorn_myapp restart"
if not exist then restart
if mem is greater than 300.0 MB for 2 cycles then restart # eating up memory?
if cpu is greater than 50% for 4 cycles then restart # send an email to admin
if cpu is greater than 80% for 30 cycles then restart # hung process?
group unicorn
it should restart unicorn when such error happens which break the app
from unicorn.log file
ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
when I run /etc/init.d/unicorn_myapp restart from terminal it works
Monit mostly uses the restart program to start a program. I do not know why this is, but I also observed this behavior.
Just try to comment out the "restart" line. This should force monit to run the start script, what should not try to kill an existing process.
You might also want to watch your log-file like
CHECK FILE unicorn_log PATH log__file__path___change_me_or_the_world_will_flatten
start program = "/etc/init.d/unicorn_myapp start"
stop program = "/etc/init.d/unicorn_myapp stop"
# No log File entry for one hour?
if timestamp is older than 1 hour then restart
# Allocate Memery error?
if content = "Cannot allocate memory" then restart
I have the following shell script that allows me to start my rails app, let's say it's called start-app.sh:
#!/bin/bash
cd /var/www/project/current
. /home/user/.rvm/environments/ruby-2.3.3
RAILS_SERVE_STATIC_FILES=true RAILS_ENV=production nohup bundle exec rails s -e production -p 4445 > /var/www/project/log/production.log 2>&1 &
the file above have permissions of:
-rwxr-xr-x 1 user user 410 Mar 21 10:00 start-app.sh*
if i want to check the process I do the following:
ps aux | grep -v grep | grep ":4445"
it'd give me the following output:
user 2960 0.0 7.0 975160 144408 ? Sl 10:37 0:07 puma 3.12.0 (tcp://0.0.0.0:4445) [20180809094218]
P.S: the reason i grep ":4445" is because i have few processes running on different ports. (for different projects)
now coming to monit, i used apt-get to install it, and the latest version from repo is 5.16, as i'm running on Ubuntu 16.04, also note that monit is running as root, that's why i specified the gid uid in the following. (because the start script is used to be executed from "user" and not "root")
Here's the configuration for monit:
set daemon 20 # check services at 20 seconds interval
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set eventqueue
basedir /var/lib/monit/events # set the base directory where events will be stored
slots 100 # optionally limit the queue size
set mailserver xx.com port xxx
username "xx#xx.com" password "xxxxxx"
using tlsv12
with timeout 20 seconds
set alert xx#xx.com
set mail-format {
from: xx#xx.com
subject: monit alert -- $EVENT $SERVICE
message: $EVENT Service $SERVICE
Date: $DATE
Action: $ACTION
Host: $HOST
Description: $DESCRIPTION
}
set limits {
programOutput: 51200 B
sendExpectBuffer: 25600 B
fileContentBuffer: 51200 B
networktimeout: 10 s
}
check system $HOST
if loadavg (1min) > 4 then alert
if loadavg (5min) > 2 then alert
if cpu usage > 90% for 10 cycles then alert
if memory usage > 85% then alert
if swap usage > 35% then alert
check process nginx with pidfile /var/run/nginx.pid
start program = "/bin/systemctl start nginx"
stop program = "/bin/systemctl stop nginx"
check process redis
matching "redis"
start program = "/bin/systemctl start redis"
stop program = "/bin/systemctl stop redis"
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
include /etc/monit/conf.d/*
include /etc/monit/conf-enabled/*
Now monit, is detecting and alerting me when the process goes down (if i kill it manually) and when it's manually recovered, but it won't start that shell script automatically.. and according to /var/log/monit.log, it's showing the following:
[UTC Aug 13 10:16:41] info : Starting Monit 5.16 daemon
[UTC Aug 13 10:16:41] info : 'production-server' Monit 5.16 started
[UTC Aug 13 10:16:43] error : 'myapp' process is not running
[UTC Aug 13 10:16:46] info : 'myapp' trying to restart
[UTC Aug 13 10:16:46] info : 'myapp' start: /bin/bash
[UTC Aug 13 10:17:17] error : 'myapp' failed to start (exit status 0) -- no output
So far what I see when monit tries to execute the script is that it tries to load it (i can see it for less than 3 seconds using ps aux | grep -v grep | grep ":4445", but this output is different from the above output i showed up, it shows the content of the shell script being executed and specifically this one:
blablalba... nohup bundle exec rails s -e production -p 4445
and then it disappears. then it tries to re-execute the shell.. again and again...
What am I missing, and what is wrong with my configuration? note that I can't change anything in the start-app.sh because it's on production and working 100%. (i just want to monitor it)
Edit: To my understanding and experience, it seems to be a Environment Variable issue or path issue, but i'm not sure how to solve it, it doesn't make any sense to put the env variables inside monit .. what if someone else wanted to edit that shell script or add something new? i hope you get my point
As i expected, it was user-environment issue and i solved it by editing monit configuration as below:
Before (not working)
check process myapp
matching ":4445"
start program = "/bin/bash -c '/home/user/start-app.sh'" as uid "user" and gid "user"
stop program = "/bin/bash -c /home/user/stop-app.sh" as uid "user" and gid "user"
After (working)
check process myapp
matching ":4445"
start program = "/bin/su -s /bin/bash -c '/home/user/start-app.sh' user"
stop program = "/bin/su -s /bin/bash -c '/home/user/stop-app.sh' user"
Explanation: i removed (uid and gid) as "user" from monit because it will only execute the shell script in the name of "user" but it won't get/import/use user's env path, or env variables.
I am able to run HHVM with user hhvm as it is default. But when trying to change the user to nginx I get error when checking it's status.
$ vim /usr/lib/systemd/system/hhvm.service
I add the following
[Unit]
Description=HipHop Virtual Machine (FCGI)
[Service]
ExecStart=/usr/local/bin/hhvm -c /etc/hhvm/server.ini -c /etc/hhvm/php.ini --user nginx --mode daemon
[Install]
WantedBy=multi-user.target
on the config file
$ vim /etc/hhvm/server.ini
this is what I have for config
pid = /var/run/hhvm/pid
hhvm.server.ip = 127.0.0.1
hhvm.server.port = 9000
;hhvm.server.file_socket = /var/run/hhvm/hhvm.sock
hhvm.server.user = nginx
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.source_root = /srv/www/public_html
hhvm.server.always_use_relative_path = false
hhvm.server.thread_count = 32
hhvm.resource_limit.max_socket = 65536
hhvm.jit = true
hhvm.jit_a_size = 67108864
hhvm.jit_a_stubs_size = 22554432
hhvm.jit_global_data_size = 22554432
hhvm.jit_profile_interp_requests = 3
; mysql
hhvm.mysql.socket = /var/lib/mysql/mysql.sock
hhvm.mysql.typed_results = true
; logging
hhvm.log.use_syslog = false
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
after this I run the commands
$ systemctl daemon-reload
$ systemctl start hhvm.service
$ systemctl status hhvm.service
When I check for status I get the following message
hhvm.service - HipHop Virtual Machine (FCGI)
Loaded: loaded (/usr/lib/systemd/system/hhvm.service; enabled)
Active: failed (Result: exit-code) since Fri 2015-04-10 23:43:00 CEST; 2s ago
Process: 2545 ExecStart=/usr/local/bin/hhvm -c /etc/hhvm/server.ini -c /etc/hhvm/php.ini --user nginx --mode daemon (code=exited, status=1/FAILURE)
Main PID: 2545 (code=exited, status=1/FAILURE)
Apr 10 23:43:00 centos7.local systemd[1]: Starting HipHop Virtual Machine (FCGI)...
Apr 10 23:43:00 centos7.local systemd[1]: Started HipHop Virtual Machine (FCGI).
Apr 10 23:43:00 centos7.local systemd[1]: hhvm.service: main process exited, code=exited, status=1/FAILURE
Apr 10 23:43:00 centos7.local systemd[1]: Unit hhvm.service entered failed state.
Also I have to create this folder manually every time I reboot
$ mkdir /var/run/hhvm
Why does it get deleted automatically?
Can you start it in daemon mode normally?
Just execute command you see in the service file, the ExecStart one.
And for the missing hhvm folder maybe you could write a command to make the dir before starting script using ExecStartPre or you could change the location of the pid file to just /var/run folder.
#limilaw i think you can change the user in the service file but be careful as you can mess things up.
I'm trying to use Monit to watch a Thin Rails application server process. Here is my Monit config:
check process thin-3000
with pidfile /var/www/apps/myapp/shared/pids/thin.3000.pid
start program = "/bin/su - deploy -c 'thin start -C /etc/thin/myapp.yml -o 3000'"
stop program = "/bin/su - deploy -c 'thin stop -C /etc/thin/myapp.yml -o 3000'"
if failed port 3000 then restart
group thin
This actually works. If I kill the Thin server process, Monit will faithfully restart it. However, if I watch the Monit log it keeps outputting the following over and over again:
[UTC Jan 14 23:01:04] error : 'thin-3000' process is not running
[UTC Jan 14 23:01:04] info : 'thin-3000' trying to restart
[UTC Jan 14 23:01:04] info : 'thin-3000' start: /bin/su
[UTC Jan 14 23:01:34] error : 'thin-3000' failed to start
It looks like whatever mechanism it's using to check if the process is running isn't working correctly. Do you know what I might be doing wrong?