to start the thin server with monit is just start program = "/etc/init.d/thin start"
but to start private_pub or faye is needed to bundle the rackup.
and how to stop the pid?
someone have a idea?
check process private_pub_myapp
with pidfile "/home/ubuntu/myapp/shared/tmp/pids/private_pub.pid"
start program = "bundle exec rackup /home/ubuntu/myapp/shared/config/private_pub.ru -s thin -E production" with timeout 90 seconds
stop program = "kill -s TERM `cat /home/ubuntu/myapp/shared/config/private_pub.yml`" with timeout 90 seconds
if totalmem is greater than 200 MB for 2 cycles then restart # eating up memory?
group private_pub_myapp
monit spitting out
/conf.d/private_pub.conf:3: Warning: Program does not exist: 'bundle'
/etc/monit/conf.d/private_pub.conf:5: Warning: Program does not exist:
'kill'
You always need to give a full, absolute path when using Monit. For example start program = "/usr/local/bin/bundle exec ..." and similar. That said, I suspect this still won't work. You seem to be trying to cat the config YAML to find the PID to kill which is probably a copy-paste issue and you'll need to make sure your rackup config is actually writing out the PID file.
Related
I'm finding an awful lot of conflicting information about monitoring Unicorn, with people saying other config scripts are wrong, and posting their own. There seems to be no main config that Just Works™
I'm assuming preload_app and zero-downtime deployment are the main culprit. I'd love to have that, but for now I'm more interested in just getting monitoring running, period. So currently I have all those settings turned off.
Right now I'm using capistrano-unicorn which is a really great gem.
It gives me all the capistrano deploy hooks I need to reload unicorn. The app has successfully deployed already with it.
The main thing I want to do now is...
a) Make sure unicorn starts up automatically on server failure/reboot
b) Monitor unicorn to restart workers that die/hang/whatever.
If I'm using this gem, what might be the best approach to complete my goals (keeping in mind I don't necessarily need zero downtime)?
Thanks
Engine yard uses Monit, and it's a pretty little utility that does exactly what you need!
Here is the configuration for unicorn:
check process unicorn
with pidfile /path/to/unicorn.pid
start program = "command to start unicorn"
as uid yourUID and gid yourGID
stop program = "command to stop unicorn"
as uid yourUID and gid yourGID
if mem > 255.0 MB for 2 cycles then restart
if cpu > 100% for 2 cycles then restart
check process unicorn_worker0
with pidfile /path/to/unicorn_worker_0.pid
if mem > 255.0 MB for 2 cycles then exec "/bin/bash -c '/bin/kill -6 `cat /path/to/unicorn_worker_0.pid` && sleep 1'"
...
i'm planning to add some tasks to my capistrano recipes file to give ability for admins to remote start/stop/restart private_pub server. But if for start i can use something like
desc "Start private_pub server"
task :start do
run "cd #{current_path};rackup private_pub.ru -s thin -E production -D"
end
i cannot find any documentation how to stop or restart rackup server. i see option
-P, --pid FILE file to store PID (default: rack.pid)
but maybe use kill command to stop server - not good idea?
I found this while searching for the same solution.
gist.github.com/3197633
basically while starting the process you have it write the pid number to a file in /tmp/pids and then when you go to stop it you read that file and it runs kill -9 ...
Hope it helps.
Your'e correct in that using kill -9 is a bad idea. This can lead to uneccesary loss of data, and as I understand it is recommended to use kill 2 or kill -INT which is equivalent to hitting 'control-c' and should close a normal server for you. I personally have started managing my servers with the God gem by TPW. Here is the script I use for running a local 'geminabox' server, for instance:
God.watch do |w|
w.name = 'gemserver'
w.dir = '/usr/local/gemserver'
w.pid_file = "#{ENV['HOME']}/.god/pids/#{w.name}.pid"
ru = File.expand_path `which rackup`
w.start = "#{ru} -D #{w.dir}/config.ru -P #{w.pid_file}"
# w.stop = lambda { Process.kill(3, `lsof -i :9292`.chomp.to_i) }
w.behavior :clean_pid_file
w.keepalive
end
I am working on a God script to monitor my Unicorns. I started with GitHub's examples script and have been modifying it to match my server configuration. Once God is running, commands such as god stop unicorn and god restart unicorn work just fine.
However, god start unicorn results in WARN: unicorn start command exited with non-zero code = 1. The weird part is that if I copy the start script directly from the config file, it starts right up like a brand new mustang.
This is my start command:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D
I have declared all paths as absolute in the config file. Any ideas what might be preventing this script from working?
I haven't used unicorn as an app server, but I've used god for monitoring before.
If I remember rightly when you start god and give your config file, it automatically starts whatever you've told it to watch. Unicorn is probably already running, which is why it's throwing the error.
Check this by running god status once you've started god. If that's not the case you can check on the command line what the comand's exit status is:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D;
echo $?;
that echo will print the exit status of the last command. If it's zero, the last command reported no errors. Try starting unicorn twice in a row, I expect the second time it'll return 1, because it's already running.
EDIT:
including the actual solution from comments, as this seems to be a popular response:
You can set an explicit user and group if your process requires to be run as a specific user.
God.watch do |w|
w.uid = 'root'
w.gid = 'root'
# remainder of config
end
My problem was that I never bundled as root. Here is what I did:
sudo bash
cd RAILS_ROOT
bundle
You get a warning telling you to never do this:
Don't run Bundler as root. Bundler can ask for sudo if it is needed,
and installing your bundle as root will break this application for all
non-root users on this machine.
But it was the only way I could get resque or unicorn to run with god. This was on an ec2 instance if that helps anyone.
Add the log option has helped me greatly in debugging.
God.watch do |w|
w.log = "#{RAILS_ROOT}/log/god.log"
# remainder of config
end
In the end, my bug turned out to be the start_script in God was executed in development environment. I fixed this by appending the RAILS_ENV to the start script.
start_script = "RAILS_ENV=#{ENV['RACK_ENV']} bundle exec sidekiq -P #{pid_file} -C #{config_file} -L #{log_file} -d"
Are there any examples on the web of how to monitor delayed_job with Monit?
Everything I can find uses God, but I refuse to use God since long running processes in Ruby generally suck. (The most current post in the God mailing list? God Memory Usage Grows Steadily.)
Update: delayed_job now comes with a sample monit config based on this question.
Here is how I got this working.
Use the collectiveidea fork of delayed_job besides being actively maintained, this version has a nice script/delayed_job daemon you can use with monit. Railscasts has a good episode about this version of delayed_job (ASCIICasts version). This script also has some other nice features, like the ability to run multiple workers. I don't cover that here.
Install monit. I installed from source because Ubuntu's version is so ridiculously out of date. I followed these instructions to get the standard init.d scripts that come with the Ubuntu packages. I also needed to configure with ./configure --sysconfdir=/etc/monit so the standard Ubuntu configuration dir was picked up.
Write a monit script. Here's what I came up with:
check process delayed_job with pidfile /var/www/app/shared/pids/delayed_job.pid
start program = "/var/www/app/current/script/delayed_job -e production start"
stop program = "/var/www/app/current/script/delayed_job -e production stop"
I store this in my soucre control system and point monit at it with include /var/www/app/current/config/monit in the /etc/monit/monitrc file.
Configure monit. These instructions are laden with ads but otherwise OK.
Write a task for capistrano to stop and start. monit start delayed_job and monit stop delayed_job is what you want to run. I also reload monit when deploying to pick up any config file changes.
Problems I ran into:
daemons gem must be installed for script/delayed_job to run.
You must pass the Rails environment to script/delayed_job with -e production (for example). This is documented in the README file but not in the script's help output.
I use Ruby Enterprise Edition, so I needed to get monit to start with that copy of Ruby. Because of the way sudo handles the PATH in Ubuntu, I ended up symlinking /usr/bin/ruby and /usr/bin/gem to the REE versions.
When debugging monit, I found it helps to stop the init.d version and run it from the th command line, so you can get error messages. Otherwise it is very difficult to figure out why things are going wrong.
sudo /etc/init.d/monit stop
sudo monit start delayed_job
Hopefully this helps the next person who wants to monitor delayed_job with monit.
For what it's worth, you can always use /usr/bin/env with monit to setup the environment. This is especially important in the current version of delayed_job, 1.8.4, where the environment (-e) option is deprecated.
check process delayed_job with pidfile /var/app/shared/pids/delayed_job.pid
start program = "/usr/bin/env RAILS_ENV=production /var/app/current/script/delayed_job start"
stop program = "/usr/bin/env RAILS_ENV=production /var/app/current/script/delayed_job stop"
In some cases, you may also need to set the PATH with env, too.
I found it was easier to create an init script for delayed job. It is available here: http://gist.github.com/408929
or below:
#! /bin/sh
set_path="cd /home/rails/evatool_staging/current"
case "$1" in
start)
echo -n "Starting delayed_job: "
su - rails -c "$set_path; RAILS_ENV=staging script/delayed_job start" >> /var/log/delayed_job.log 2>&1
echo "done."
;;
stop)
echo -n "Stopping sphinx: "
su - rails -c "$set_path; RAILS_ENV=staging script/delayed_job stop" >> /var/log/delayed_job.log 2>&1
echo "done."
;;
*)
N=/etc/init.d/delayed_job_staging
echo "Usage: $N {start|stop}" >&2
exit 1
;;
esac
exit 0
Then make sure that monit is set to start / restart the app so in your monitrc file:
check process delayed_job with pidfile "/path_to_my_rails_app/shared/pids/delayed_job.pid"
start program = "/etc/init.d/delayed_job start"
stop program = "/etc/init.d/delayed_job stop"
and that works great!
I found a nice way to start delayed_job with cron on boot. I'm using whenever to control cron.
My schedule.rb:
# custom job type to control delayed_job
job_type :delayed_job, 'cd :path;RAILS_ENV=:environment script/delayed_job ":task"'
# delayed job start on boot
every :reboot do
delayed_job "start"
end
Note: I upgraded whenever gem to 0.5.0 version to be able to use job_type
I don't know with Monit, but I've written a couple Munin plugins to monitor Queue Size and Average Job Run Time. The changes I made to delayed_job in that patch might also make it easier for you to write Monit plugins in case you stick with that.
Thanks for the script.
One gotcha -- since monit by definition has a 'spartan path' of
/bin:/usr/bin:/sbin:/usr/sbin
... and for me ruby was installed / linked in /usr/local/bin, I had to thrash around for hours trying to figure out why monit was silently failing when trying to restart delayed_job (even with -v for monit verbose mode).
In the end I had to do this:
check process delayed_job with pidfile /var/www/app/shared/pids/delayed_job.pid
start program = "/usr/bin/env PATH=$PATH:/usr/local/bin /var/www/app/current/script/delayed_job -e production start"
stop program = "/usr/bin/env PATH=$PATH:/usr/local/bin /var/www/app/current/script/delayed_job -e production stop"
I had to combine the solutions on this page with another script made by toby to make it work with monit and starting with the right user.
So my delayed_job.monitrc looks like this:
check process delayed_job
with pidfile /var/app/shared/pids/delayed_job.pid
start program = "/bin/su -c '/usr/bin/env RAILS_ENV=production /var/app/current/script/delayed_job start' - rails"
stop program = "/bin/su -c '/usr/bin/env RAILS_ENV=production /var/app/current/script/delayed_job stop' - rails"
If your monit is running as root and you want to run delayed_job as my_user then do this:
/etc/init.d/delayed_job:
#!/bin/sh
# chmod 755 /etc/init.d/delayed_job
# chown root:root /etc/init.d/delayed_job
case "$1" in
start|stop|restart)
DJ_CMD=$1
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit
esac
su -c "cd /var/www/my_app/current && /usr/bin/env bin/delayed_job $DJ_CMD" - my_user
/var/www/my_app/shared/monit/delayed_job.monitrc:
check process delayed_job with pidfile /var/www/my_app/shared/tmp/pids/delayed_job.pid
start program = "/etc/init.d/delayed_job start"
stop program = "/etc/init.d/delayed_job stop"
if 5 restarts within 5 cycles then timeout
/etc/monit/monitrc:
# add at bottom
include /var/www/my_app/shared/monit/*
Since i didn't want to run as root, I ended up creating a bash init script that monit used for starting and stopping (PROGNAME would be the absolute path to script/delayed_job):
start() {
echo "Starting $PROGNAME"
sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME start
}
stop() {
echo "Stopping $PROGNAME"
sudo -u $USER /usr/bin/env HOME=$HOME RAILS_ENV=$RAILS_ENV $PROGNAME stop
}
I have spent quite a bit of time on this topic. I was fed up with not having a good solution for it so I wrote the delayed_job_tracer plugin that specifically addresses monitoring of delayed_job and its jobs.
Here's is an article I've written about it: http://modernagility.com/articles/5-monitoring-delayed_job-and-its-jobs
This plugin will monitor your delayed job process and send you an e-mail if delayed_job crashes or if one of its jobs fail.
For Rails 3, you may need set HOME env to make compass work properly, and below config works for me:
check process delayed_job
with pidfile /home/user/app/shared/pids/delayed_job.pid
start program = "/bin/sh -c 'cd /home/user/app/current; HOME=/home/user RAILS_ENV=production script/delayed_job start'"
stop program = "/bin/sh -c 'cd /home/user/app/current; HOME=/home/user RAILS_ENV=production script/delayed_job stop'"
I ran into an issue where if the delayed job dies while it still has a job locked, that job will not be freed. I wrote a wrapper script around delayed job that will look at the pid file and free any jobs from the dead worker.
The script is for rubber/capistrano
roles/delayedjob/delayed_job_wrapper:
<% #path = '/etc/monit/monit.d/monit-delayedjob.conf' %>
<% workers = 4 %>
<% workers.times do |i| %>
<% PIDFILE = "/mnt/custora-#{RUBBER_ENV}/shared/pids/delayed_job.#{i}.pid" %>
<%= "check process delayed_job.#{i} with pidfile #{PIDFILE}"%>
group delayed_job-<%= RUBBER_ENV %>
<%= " start program = \"/bin/bash /mnt/#{rubber_env.app_name}-#{RUBBER_ENV}/current/script/delayed_job_wrapper #{i} start\"" %>
<%= " stop program = \"/bin/bash /mnt/#{rubber_env.app_name}-#{RUBBER_ENV}/current/script/delayed_job_wrapper #{i} stop\"" %>
<% end %>
roles/delayedjob/delayed_job_wrapper
#!/bin/bash
<% #path = "/mnt/#{rubber_env.app_name}-#{RUBBER_ENV}/current/script/delayed_job_wrapper" %>
<%= "pid_file=/mnt/#{rubber_env.app_name}-#{RUBBER_ENV}/shared/pids/delayed_job.$1.pid" %>
if [ -e $pid_file ]; then
pid=`cat $pid_file`
if [ $2 == "start" ]; then
ps -e | grep ^$pid
if [ $? -eq 0 ]; then
echo "already running $pid"
exit
fi
rm $pid_file
fi
locked_by="delayed_job.$1 host:`hostname` pid:$pid"
<%=" /usr/bin/mysql -e \"update delayed_jobs set locked_at = null, locked_by = null where locked_by='$locked_by'\" -u#{rubber_env.db_user} -h#{rubber_instances.for_role('db', 'primary' => true).first.full_name} #{rubber_env.db_name} " %>
fi
<%= "cd /mnt/#{rubber_env.app_name}-#{RUBBER_ENV}/current" %>
. /etc/profile
<%= "RAILS_ENV=#{RUBBER_ENV} script/delayed_job -i $1 $2"%>
to see what is going on, run monit in foreground verbose mode: sudo monit -Iv
using rvm installed under user "www1" and group "www1".
in file /etc/monit/monitrc:
#delayed_job
check process delayed_job with pidfile /home/www1/your_app/current/tmp/pids/delayed_job.pid
start program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job start'" as uid www1 and gid www1
stop program "/bin/bash -c 'PATH=$PATH:/home/www1/.rvm/bin;source /home/www1/.rvm/scripts/rvm;cd /home/www1/your_app/current;RAILS_ENV=production bundle exec script/delayed_job stop'" as uid www1 and gid www1
if totalmem is greater than 200 MB for 2 cycles then alert
I am running my rails application using the following
$script/server -d webrick
on my Ubuntu system , above command run the webrick server in background . I could kill the process using kill command
$kill pid
Does rails provide any command to stop the background running daemon server ?
like the one provided by rails to start the server , Thanks .
EDIT When it is appropriate to start the daemon server ? Any real time scenario will help Thanks
if it can be useful, on linux you can find which process is using a port (in this case 3000) you can use:
lsof -i :3000
it'll return the pid too
Like Ryan said:
the pid you want is in tmp/pids/
probably server.pid is the file you want.
You should be able to run kill -9 $(cat tmp/pids/server.pid) to bring down a daemonized server.
How about a rake task?
desc 'stop rails'
task :stop do
pid_file = 'tmp/pids/server.pid'
pid = File.read(pid_file).to_i
Process.kill 9, pid
File.delete pid_file
end
run with rake stop or sudo rake stop
The only proper way to kill the Ruby on Rails default server (which is WEBrick) is:
kill -INT $(cat tmp/pids/server.pid)
If you are running Mongrel, this is sufficient:
kill $(cat tmp/pids/server.pid)
Use kill -9 if your daemon hung. Remember the implications of kill -9 - if the data kept in Active Record caches weren't flushed to disk, you will lose your data. (As I recently did)
The process id of the daemon server is stored in your application directory tmp/pids/. You can use your standard kill process_id with the information you find there.
In your terminal to find out the process id (PID):
$ lsof -wni tcp:3000
Then, use the number in the PID column to kill the process:
$ kill -9 <PID>
pguardiario beat me to it, though his implementation is a bit dangerous since it uses SIGKILL instead of the (recommended) SIGINT. Here's a rake task I tend to import into my development projects:
lib/tasks/stopserver.rake
desc 'stop server'
task :stopserver do
pid_file = 'tmp/pids/server.pid'
if File.file?(pid_file)
print "Shutting down WEBrick\n"
pid = File.read(pid_file).to_i
Process.kill "INT", pid
end
File.file?(pid_file) && File.delete(pid_file)
end
This issues an interrupt to the server if and only if the pidfile exists. It doesn't throw unsightly errors if the server isn't running, and it notifies you if it's actually shutting the server down.
If you notice that the server doesn't want to shut down using this task, add the following line after the Process.kill "INT" line, and try to upgrade to a kernel that has this bug fixed.
Process.kill "CONT", pid
(Hat tip: jackr)
A Ruby ticket, http://bugs.ruby-lang.org/issues/4777, suggests it's a kernel (Linux) bug. They give a work around (essentially equivalent to the Ctrl-C/Ctrl-Z one), for use if you've demonized the server:
kill -INT cat tmp/pids/server.pid
kill -CONT cat tmp/pids/server.pid
This seems to cause the original INT signal to be processed, possibly allowing data flush and so on.
Run this command:
locate tmp/pids/server.pid
output:
Complete path of this file. Check your project directory name to find your concerned file if multiple files are shown in list.
Then run this command:
rm -rf [complete path of tmp/pids/server.pid file]
Here I leave a bash function which, if pasted in you .bashrc or .zshrc will alloy you do things like:
rails start # To start the server in development environment
rails start production # To start the server in production environment
rails stop # To stop the server
rails stop -9 # To stop the server sending -9 kill signal
rails restart # To restart the server in development environment
rails restart production # To restart the server in production environment
rails whatever # Will send the call to original rails command
Here it is the function:
function rails() {
if [ "$1" = "start" ]; then
if [ "$2" = "" ]; then
RENV="development"
else
RENV="$2"
fi
rails server -d -e "$RENV"
return 0
elif [ "$1" = "stop" ]; then
if [ -f tmp/pids/server.pid ]; then
kill $2 $(cat tmp/pids/server.pid)
return 0
else
echo "It seems there is no server running or you are not in a rails project root directory"
return 1
fi
elif [ "$1" = "restart" ]; then
rails stop && rails start $2
else
command rails $#
fi;
}
More information in the blog post I wrote about it.
i don't think it does if you use -d. I'd just kill the process.
In the future, just open up another terminal window instead and use the command without -d, it provides some really useful debugging output.
If this is production, use something like passenger or thin, so that they're easy to stop the processes or restart the servers
one-liner: kill -INT `ps -e | grep ruby | awk '{print $1}'`
ps -e lists every process on the system
grep ruby searches that output for the ruby process
awk passes the first argument of that output
(the pid) to kill -INT.
Try it with echo instead of kill if you just want to see the PID.
if kill process not works, then
delete file server.pid from MyRailsApp/tmp/pids/
I came here because I were trying to (unsuccesfully) stop with a normal kill, and thought I'd being doing something wrong.
A kill -9 is the only sure way to stop a ruby on rails server?
What!? Do you know the implications of this? Can be a disaster...
You can start your server in the background by adding -d to your command. For instance:
puma -d
To stop it, just kill whatever process is running on port 3000:
kill $(cat tmp/pids/server.pid)