Monitoring resque with monit with RVM - ruby-on-rails

I previously had Monit monitoring resque with the following Monit script
check process resque_worker_production_QUEUE
with pidfile /var/tmp/resque_production.pid
start program = "/usr/bin/env HOME=/home/eg RACK_ENV=production PATH=/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c 'cd /apps/eg/production/current; nohup bundle exec rake environment resque:work RAILS_ENV=production QUEUE=mailer VERBOSE=1 PIDFILE=/var/tmp/resque_production.pid & >> log/resque_worker_production_QUEUE.log 2>&1'" as uid eg and gid eg
stop program = "/bin/sh -c 'cd /apps/eg/production/current && kill -9 $(cat
Then I changed some things around, most notably removing the system-wide rvm install and I'm assuming a ruby install at /usr/local/ruby/bin . Around this time monit could no longer start resque. Maybe the cause was something else but I'm thinking it was these uninstalls that caused the breakage.
So I looked throught the script and noticed that /usr/local/ruby/bin doesn't exist, so I tried changing it to what I think it should point to now that the ruby in my rvm is the only one around /home/eg/.rvm/rubies/ruby-1.9.3-p194/bin
But that didn't work. So I google some more and found this suggestion which also didn't work:
check process resque_worker_production_QUEUE
with pidfile /var/tmp/resque_production.pid
start program = "/bin/bash -l -c 'cd /apps/eg/production/current; nohup bundle exec rake environment resque:work RAILS_ENV=production QUEUE=mailer VERBOSE=1 PIDFILE=/var/tmp/resque_production.pid & >> log/resque_worker_production_QUEUE.log 2>&1'" as uid eg and gid eg
All I get in the log is:
[UTC Oct 5 03:06:38] error : 'resque_worker_production_QUEUE' process is not running
[UTC Oct 5 03:06:38] info : 'resque_worker_production_QUEUE' trying to restart
[UTC Oct 5 03:06:38] info : 'resque_worker_production_QUEUE' start: /bin/bash
[UTC Oct 5 03:07:08] error : 'resque_worker_production_QUEUE' failed to start
So I'm not sure how to debug this further. Any suggestions?

To be sure, do you have RVM loading script in bash files?
Something like:
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting
For me it works with:
check process <%= pid_name %>
with pidfile <%= pid %>
start program = "/bin/bash -c -l -i 'cd <%= current_path %> && NEWRELIC_ENABLE=false HOME=/home/<%= resque_user %> RAILS_ENV=<%= rails_env %> QUEUE=<%= queue %> PIDFILE=<%= pid %> BACKGROUND=yes VERBOSE=1 <%= fetch(:bundle_cmd, "bundle") %> exec rake resque:work >> <%= shared_path %>/log/resque.log 2>&1'" as uid <%= resque_user %> and gid <%= resque_group %> with timeout 180 seconds
I see difference in -i flag, not sure if it's required, and in order of env variables
Have you checked resque logs? (log/resque_worker_production_QUEUE.log as I see)
Another thing to try is to run this command after ssh'ing to server as root (or another monit user), sudo su won't be enough as you'll already have rvm loaded, so better try from scratch. It can reveal some errors.
Also check out this page http://rvm.io/integration/cron for tips

Related

Monit failing to restart sidekiq

I'm trying to get monit to restart my sidekiq service on CentOS server. After trying multiple solutions out there, I'm stumped, still failing to start the service.
My sidekiq file from monit.d:
check process sidekiq
with pidfile /var/www/App/tmp/pids/sidekiq.pid
start program = "/bin/bash -l -c 'sudo cd /var/www/App && bundle exec sidekiq --index 0 --pidfile /var/www/App/tmp/pids/sidekiq.pid --environment production --logfile /var/www/App/log/sidekiq.log --daemon'" as uid deploy and gid deploy
stop program = "/bin/bash -l -c 'cd /var/www/App && bundle exec sidekiqctl stop /var/www/App/tmp/pids/sidekiq.pid 10'" as uid deploy and gid deploy
if totalmem is greater than 512 MB for 2 cycles then restart
if 3 restarts within 5 cycles then timeout
If I run start program command manually, it starts the sidekiq fine but the monit doesn't seem to do anything. Just comes up with:
[BST Oct 6 11:51:17] error : 'sidekiq' process is not running
[BST Oct 6 11:51:17] info : 'sidekiq' trying to restart
[BST Oct 6 11:51:17] info : 'sidekiq' start: /bin/bash
[BST Oct 6 11:52:47] error : 'sidekiq' failed to start
So it is including file fine, but somehow doesn't manage to start the service from the script.
What can it be? Some permissions issue of sorts?
You need to update to the latest Monit version (5.14).
Remove your current monit installation and follow these instructions:
https://rtcamp.com/tutorials/monitoring/monit/
Hope it helps!
PS: Found the solution here: https://bitbucket.org/tildeslash/monit/issues/109/failed-to-stop-always-after-60-seconds
according to Debugging monit
I found i need set PATH.
my start program:
/bin/bash -c 'cd /home/vagrant/apps/skylark/current; PATH=/home/vagrant/.rbenv/shims:/home/vagrant/.rbenv/bin:$PATH bundle exec sidekiq -d -e production -C -P /home/vagrant/apps/skylark/shared/tmp/pids/sidekiq.pid -L /home/vagrant/apps/skylark/shared/log/sidekiq.log'
i think the issue is with your user. You need to execute using deploy user.
check process sidekiq
with pidfile /var/www/App/tmp/pids/sidekiq.pid
start program = "/bin/su - deploy -c 'sudo cd /var/www/App && bundle exec sidekiq --index 0 --pidfile /var/www/App/tmp/pids/sidekiq.pid --environment production --logfile /var/www/App/log/sidekiq.log --daemon'" as uid deploy and gid deploy
stop program = "/bin/su - deploy -c 'cd /var/www/App && bundle exec sidekiqctl stop /var/www/App/tmp/pids/sidekiq.pid 10'" as uid deploy and gid deploy
if totalmem is greater than 512 MB for 2 cycles then restart
if 3 restarts within 5 cycles then timeout

How to update ENV variable on Passenger standalone restart

I'm using Capistrano to deploy my application. Application runs on Passenger standalone. When I redeploy the application the Passenger still uses the Gemfile from the the old release because BUNDLE_GEMFILE environment variable has not been updated.
Where I should put the updated path to Gemfile so that Passenger would pick it up on restart?
The server startup command is in monit and I just call monit scripts from Capistrano tasks except for restart where I just touch the restart.txt.
namespace :deploy do
task :stop do
run("sudo /usr/bin/monit stop my_app_#{rails_env}")
end
task :restart do
run("cd #{current_path} && touch tmp/restart.txt")
end
task :start do
run("sudo /usr/bin/monit start my_app_#{rails_env}")
end
The startup command in monit is:
start program = "/bin/su - app_user -l -c 'cd /home/app_user/current && bundle exec passenger start -d -p 8504 -e production --pid-file=/home/app_user/current/tmp/pids/passenger.8504.pid /home/app_user/current'"
I already tried to add the BUNDLE_GEMFILE into the startup command like this:
start program = "/bin/su - app_user -l -c 'cd /home/app_user/current && BUNDLE_GEMFILE=/home/app_user/current/Gemfile bundle exec passenger start -d -p 8504 -e production --pid-file=/home/app_user/current/tmp/pids/passenger.8504.pid /home/app_user/current'"
But it didn't work since the path /home/app_user/current is a symlink to a release path and that release path was picked up instead.
Simple solution.
Define the Gemfile to be used in the server start command. For example:
BUNDLE_GEMFILE=/home/app_user/current/Gemfile bundle exec passenger start -d -p 9999 -e production --pid-file=/home/app_user/current/tmp/pids/passenger.9999.pid /home/app_user/current
The earlier solution (setting the BUNDLE_GEMFILE env variable in .profile) is not good. When you are deploying a new version of your application and there is a new gem in the bundle the migrations etc. will fail because it will still use the Gemfile defined in the env variable.

Start Unicorn with Runit and User's RVM

I'm deploying my Rails App servers with Chef. Have just swapped to RVM from a source install of Ruby (because I was having issues with my deploy user).
Now I have my deploy sorted, assets compiled and bundler's installed all my gems.
The problem I have is supervising Unicorn with Runit..
RVM is not installed as root user - only as my deploy user has it, as follows:
$ rvm list
rvm rubies
=* ruby-2.0.0-p247 [ x86_64 ]
I can manually start Unicorn successfully from my deploy user. However, it won't start as part of runit.
My run file looks like this. I have also tried the solution in this SO question unsuccessfully..
#!/bin/bash
cd /var/www/html/deploy/production/current
exec 2>&1
exec chpst -u deploy:deploy /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -E production -c config/unicorn_production.rb
If I run it manually, I get this error:
/usr/bin/env: ruby_noexec_wrapper: No such file or directory
I created a small script (gist here) which does run as root. However, if I call this from runit, I see the workers start but I get two processes for runit and I can't stop or restart the service:
Output of ps:
1001 29062 1 0 00:08 ? 00:00:00 unicorn master -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
1001 29065 29062 9 00:08 ? 00:00:12 unicorn worker[0] -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
root 29076 920 0 00:08 ? 00:00:00 su - deploy -c cd /var/www/html/deploy/production/current; export GEM_HOME=/home/deploy/.rvm/gems/ruby-2.0.0-p247; /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
1001 29083 29076 0 00:08 ? 00:00:00 -su -c cd /var/www/html/deploy/production/current; export GEM_HOME=/home/deploy/.rvm/gems/ruby-2.0.0-p247; /home/deploy/.rvm/gems/ruby-2.0.0-p247/bin/unicorn -D -E production -c /var/www/html/deploy/production/current/config/unicorn_production.rb
What should I do here? Move back to monit which worked nicely?
your run file is doing it wrong, you are using the binary without setting the environment, for that purpose you should use wrappers:
rvm wrapper ruby-2.0.0-p247 --no-links unicorn
To simplify the script use alias so it does not need to be changed when you decide which ruby should be used:
rvm alias create my_app_unicorn ruby-2.0.0-p247
And change the script to:
#!/bin/bash
cd /var/www/html/deploy/production/current
exec 2>&1
exec chpst -u deploy:deploy /home/deploy/.rvm/wrappers/my_app_unicorn/unicorn -E production -c config/unicorn_production.rb
This will ensure proper environment is used for execution of unicorn and any time you want change ruby used to run it just crate alias to a new ruby.

Manage sidekiq with init.d script using RVM

I'm using the init.d script provided (the init.d script from the sidekiq github repo), but I am on an ubuntu system with RVM installed system wide.
I cannot seem to figure out how to cd into my app directory and issue the command without there being some complaining in the log and nothing actually starting.
Question: What should the startup command for sidekiq look like in my init.d script when I am using RVM? My user is named ubuntu. Currently I have this in my init.d script:
START_CMD="$BUNDLE exec $SIDEKIQ"
# where bundle is /usr/local/rvm/gems/ruby-1.9.3-p385/bin/bundle
# and sidekiq is sidekiq
# I've also tried with the following args: -e $APP_ENV -P $PID_FILE -C $APP_CONFIG/sidekiq.yml -d -L $LOG_FILE"
RETVAL=0
start() {
status
if [ $? -eq 1 ]; then
[ `id -u` == '0' ] || (echo "$SIDEKIQ runs as root only .."; exit 5)
[ -d $APP_DIR ] || (echo "$APP_DIR not found!.. Exiting"; exit 6)
cd $APP_DIR
echo "Starting $SIDEKIQ message processor .. "
echo "in dir `pwd`"
su - ubuntu -c "$START_CMD >> $LOG_FILE 2>&1 &"
RETVAL=$?
#Sleeping for 8 seconds for process to be precisely visible in process table - See status ()
sleep 8
[ $RETVAL -eq 0 ] && touch $LOCK_FILE
return $RETVAL
else
echo "$SIDEKIQ message processor is already running .. "
fi
}
My sidekiq.log gives me this error:Could not locate Gemfile. However, I print the working directory and I am most definitely in my app's current directory, according to the echo pwd, at the time this command is executed.
When I take out the su - ubuntu -c [command here], I get this error:
/usr/bin/env: ruby_noexec_wrapper: No such file or directory
My solution is to just start the process manually. When I manually cd into my app directory and issue this command:
bundle exec sidekiq -d -L log/sidekiq.log -P tmp/pids/sidekiq.pid
things go as planned, and then
sudo /etc/init.d/sidekiq status
tells me things are up and running.
Also, sudo /etc/init.d/sidekiq stop and status work as expected.
I wrote a blog post a few months ago on my experience writing an init.d script for Sidekiq, however I was using rbenv rather than RVM.
https://cdyer.co.uk/blog/init-script-for-sidekiq-with-rbenv/
I think you should be able to use something almost identical except for modifying the username and app dir variables.
use wrappers:
BUNDLER=/usr/local/rvm/wrappers/ruby-1.9.3-p385/bundle
in case the bundler wrapper is not available generate it with:
rvm wrapper ruby-1.9.3-p385 --no-links bundle # OR:
rvm wrapper ruby-1.9.3-p385 --no-links --all
you can use aliases to make it easier:
rvm alias create my_app 1.9.3-p385
and then use it like this:
BUNDLER=/usr/local/rvm/wrappers/my_app/bundle
this way you will not have to change the script when application ruby changes - just update the alias, there is a good description/integration for this in rvm-capistrano => https://github.com/wayneeseguin/rvm-capistrano/#create-application-alias-and-wrappers

Managing Resque workers with Monit on RBenv setup

I'm trying to set up Monit to manage Resque workers, but it fails to start saying /home/deployer/.rbenv/shims/bundle: line 4: exec: rbenv: not found
I've checked that it is running commands as deployer user and if I copy and paste the command directly via SSH everything works fine. Below is my Monit configuration. Thanks!
check process resque_worker_1
with pidfile CURRENT_PATH/tmp/pids/resque_worker_1.pid
start program = "/usr/bin/env HOME=/home/deployer RACK_ENV=production PATH=/home/deployer/.rbenv/shims:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c 'cd CURRENT_PATH; bundle exec rake environment resque:work RAILS_ENV=production QUEUE=high,normal,low VERBOSE=1 PIDFILE=CURRENT_PATH/tmp/pids/resque_worker_1.pid >> CURRENT_PATH/log/resque_worker_.log 2>&1'"
as uid deployer and gid admin
stop program = "/bin/sh -c 'cd CURRENT_PATH && kill -9 $(cat tmp/pids/resque_worker_1.pid) && rm -f tmp/pids/resque_worker_1.pid; exit 0;'"
as uid deployer and gid admin
if totalmem is greater than 300 MB for 10 cycles then restart # eating up memory?
group resque_workers
I'm not sure if this helps, but in my monitrc start line, I have to first su to the user I want to run under. I haven't tried to use the uid and gid flags to know if that works well, so this might be a goose-chase of an answer.
I remember having the same issue as you though... everything worked from the command line, but not when monit would do its thing.
For example, in my monitrc, I am monitoring arsendmail with the following:
# arsendmail_rails3
# daemon that watches and sends mail from the rails app
check process ar_sendmail with pidfile /var/www/rak/log/ar_sendmail.pid
start program "/bin/su - mike && /bin/bash -c 'cd /var/www/rak && ar_sendmail_rails3 -b1000 -d -e production'"
stop program "/bin/ps -ef | /bin/grep ar_sendmail_rails3 | /bin/grep -v grep | /usr/bin/awk '{ /usr/bin/print $2}' | /usr/bin/xargs /bin/kill -9"
I saw that the topic was created in 2012 but I had a similar problem and this thread is top ranked by google.
The problem is that monit launch commands with a restricted env (env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh to simulate).
To use monit with rbenv you must specify the correct path before your bundle exec command.
PATH=/home/[USER]/.rbenv/bin:/home/[USER]/.rbenv/shims:$PATH bundle exec ...
Example with unicorn:
check process unicorn_dev with pidfile /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid
group dev
start program = "/bin/bash -c 'cd /home/wizville/app/dev.wizville.fr/current && PATH=/home/wizville/.rbenv/bin:/home/wizville/.rbenv/shims:$PATH bundle exec unicorn -c config/unicorn.rb -D'" as uid "wizville"
stop program = "/bin/bash -c 'kill -s QUIT `cat /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid`'"
depends on mysql
This worked for me.
check process app_resque_worker with pidfile <%= resque_pid%>
start program = "/usr/bin/env HOME=/home/subcaster RACK_ENV=production PATH=/home/subcaster/.rvm/rubies/ruby-2.0.0-p247/bin/ruby:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c \'cd <%= current_path %>; bundle exec rake environment resque:work RAILS_ENV=production BACKGROUND=yes QUEUE=* PIDFILE=<%= resque_pid %>\'"
stop program = "kill -9 cat <%= resque_pid%> && rm -f <%= resque_pid%>"
if totalmem is greater than 2000 MB for 10 cycles then restart

Resources