I am on CentOS 6.5 and running god to monitor unicorn which is being used with nginx to deploy a ruby on rails application. I am able to start god and unicorn without issue but when I run the command
god log unicorn
I get the "Please wait..." message but nothing happens after that. I've waited for about an hour without anything happening. Does anyone know what would cause this? I tried searching it in google and got absolutely nothing. My files for god configuration are taken directly from http://www.synbioz.com/blog/monitoring_server_processes_with_god. I got to the part that tells you to try god log unicorn. Here is the unicorn.god file I have based on the one in that link. It's slightly different.
rails_root = File.dirname("/home/username/myApp")
God.watch do |w|
pid_file = File.join(rails_root, "myApp/pids/unicorn.pid")
w.name = "unicorn"
w.interval = 60.seconds
w.start = "unicorn -c #{rails_root}/myApp/config/unicorn.rb -D"
w.stop = "kill -s QUIT $(cat #{pid_file})"
w.restart = "kill -s HUP $(cat #{pid_file})"
w.start_grace = 20.seconds
w.restart_grace = 20.seconds
w.pid_file = pid_file
w.behavior(:clean_pid_file)
# When to start?
w.start_if do |start|
start.condition(:process_running) do |c|
# We want to check if deamon is running every ten seconds
# and start it if itsn't running
c.interval = 10.seconds
c.running = false
end
end
# When to restart a running deamon?
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
# Pick five memory usage at different times
# if three of them are above memory limit (100Mb)
# then we restart the deamon
c.above = 100.megabytes
c.times = [3, 5]
end
restart.condition(:cpu_usage) do |c|
# Restart deamon if cpu usage goes
# above 90% at least five times
c.above = 90.percent
c.times = 5
end
end
w.lifecycle do |on|
# Handle edge cases where deamon
# can't start for some reason
on.condition(:flapping) do |c|
c.to_state = [:start, :restart] # If God tries to start or restart
c.times = 5 # five times
c.within = 5.minute # within five minutes
c.transition = :unmonitored # we want to stop monitoring
c.retry_in = 10.minutes # for 10 minutes and monitor again
c.retry_times = 5 # we'll loop over this five times
c.retry_within = 2.hours # and give up if flapping occured five times in two hours
end
end
end
I am using queue_classic for background jobs,
I need to monitor background jobs in production ie start, stop etc.
I found the similar question but it didn't help me
Also I found the god code:
but how would I stop, restart workers?
number_queues.times do |queue_num|
God.watch do |w|
w.name = "QC-#{queue_num}"
w.group = "QC"
w.interval = 5.minutes
w.start = "bundle exec rake queue:work" # This is your rake task to start QC listening
w.gid = 'nginx'
w.uid = 'nginx'
w.dir = rails_root
w.keepalive
w.env = {"RAILS_ENV" => rails_env}
w.log = "#{log_dir}/qc.stdout.log" # Or.... "#{log_dir}//qc-#{queue_num}.stdout.log"
# determine the state on startup
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end
end
end
UPDATE
This code seems doesn't work
namespace :queue_classic do
desc "Start QC worker"
task :start, roles: :web do
run "cd #{release_path} && RAILS_ENV=production bundle exec rake qc:work"
end
after "deploy:restart", "queue_classic:restart"
end
As said in the documentation you can restart your worker by issuing
god restart QC-<worker_number>
where QC-<worker_number> is the name you assign to your worker
Depending on what kind of monitoring you need, you might also look at Toro, which provides a great deal of monitoring, both in a web interface and through the fact that jobs store a great deal of data and can easily be queried using ActiveRecord queries. Toro also supports middleware, which may be useful for your needs.
I'm trying to monitor a redis server with god (ruby gem). I changed the PID file path in /etc/redis/redis.conf to a rails app temp folder of the user deploying my app (using capistrano), and in the redis.god file I added the line "w.pid_file= ..." which points to the same PID path as the one I changed in the redis.conf file. So the redis.god file looks like this at the moment:
rails_env = ENV['RAILS_ENV'] || 'production'
raise "Please specify RAILS_ENV." unless rails_env
rails_root = ENV['RAILS_ROOT'] || File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
# Redis
%w{6379}.each do |port|
God.watch do |w|
w.dir = "#{rails_root}"
w.name = "redis"
w.interval = 30.seconds
w.start = "/etc/init.d/redis-server start /etc/redis/redis.conf"
w.stop = "/etc/init.d/redis-server stop"
w.restart = "/etc/init.d/redis-server restart"
w.start_grace = 10.seconds
w.restart_grace = 10.seconds
w.log = "#{rails_root}/log/redis.log"
w.pid_file = "/home/deployer/myapp/current/tmp/pids/redis-server.pid"
w.behavior(:clean_pid_file)
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
end
end
So the problem that I'm having is that god can get redis started. I looked at its log for the god file and it says the following:
Starting redis-server: touch: cannot touch `/var/run/redis/redis-server.pid': Permission denied
Why is it still trying to look in /var/run/redis/redis-server.pid? I changed the PID path in the redis.conf file to the new one shown above because I was getting Permission denied, but it is still insisting in looking in /var/run/redis/redis-server.pid. FYI, this where I got the idea to changed the PID path: God configuration file to monitor existing processes?
Please make sure you have disabled SElinux
you can disable selinux with this command:
setenforce 0
I've been struggling with this for a while, What is the correct approach to start background processes like resque and resque scheduler? Is using God an overkill?
Currently I'm trying to get God to work, but I'm not sure if the *.god conf files should be in the app's directory or somewhere else.
This is what I use:
config
|- app.god
|- God
|- resque.god
|- resque_scheduler.god
# config/god/resque.god
rails_env = ENV['RAILS_ENV'] || raise(ArgumentError, "RAILS_ENV not defined")
rails_root = ENV['RAILS_ROOT'] || File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
num_workers = rails_env == 'production' ? 5 : 2
num_workers.times do |num|
God.watch do |w|
w.dir = "#{rails_root}"
w.name = "resque-#{num}"
w.group = 'resque'
w.interval = 30.seconds
w.env = {"QUEUE"=>"*", "RAILS_ENV"=>rails_env, "BUNDLE_GEMFILE"=>"#{rails_root}/Gemfile"}
w.start = "/usr/bin/rake -f #{rails_root}/Rakefile environment resque:work"
w.log = "#{rails_root}/log/resque-scheduler.log"
... start/stop methods ...
end
end
There's a root user and a MyApp user. MyApp user has an app located in: /home/myapp/apps/myapp_production/current
The God capistrano recipe I use is:
# config/deploy.rb
after "deploy:restart", "god:restart"
namespace :god do
def try_killing_resque_workers
run "pkill -3 -f resque"
rescue
nil
end
desc "Restart God gracefully"
task "restart", :roles => :app do
god_config_path = File.join(release_path, 'config', 'app.god')
begin
# Throws an exception if god is not running.
run "cd #{release_path}; bundle exec god status && RAILS_ENV=#{rails_env} RAILS_ROOT=#{release_path} bundle exec god load #{god_config_path} && bundle exec god start resque"
# Kill resque processes and have god restart them with the newly loaded config.
try_killing_resque_workers
rescue => ex
# god is dead, workers should be as well, but who knows.
try_killing_resque_workers
# Start god.
run "cd #{release_path}; RAILS_ENV=#{rails_env} bundle exec god -c #{god_config_path}"
end
end
end
When I deploy, I get "The server is not available (or you do not have permissions to access it)"
What's weird is when I even login as a root and run god status it returns nothing but if i run god --version it returns the version.
Anyone knows why?
Did you setup an init.d script ?
Do you have god running ?
/etc/init.d/god status
You can try to launch it by hand
/usr/bin/god -c /etc/god/conf.god -D
and check the logs
I have a working rails app with a resque queue system which works very well. However, I lack a good way of actually demonizing the resque workers.
I can start them just fine by going rake resque:work QUEUE="*" but I guess it's not the point that you should have your workers running in the foreground. For some reason nobody seems to adress this issue. On the official resque github page the claim you can do something like this:
PIDFILE=./resque.pid BACKGROUND=yes QUEUE="*" rake resque:work
well - it doesn't fork into the background here at least.
A +1 for resque-pool - it really rocks. We use it in combination with God to make sure that it is always available.
# Resque
God.watch do |w|
w.dir = RAILS_ROOT
w.name = "resque-pool"
w.interval = 30.seconds
w.start = "cd #{RAILS_ROOT} && sudo -u www-data sh -c 'umask 002 && resque-pool -d -E #{RAILS_ENV}'"
w.start_grace = 20.seconds
w.pid_file = "#{RAILS_ROOT}/tmp/pids/resque-pool.pid"
w.behavior(:clean_pid_file)
# restart if memory gets too high
#w.transition(:up, :restart) do |on|
# on.condition(:memory_usage) do |c|
# c.above = 350.megabytes
# c.times = 2
# end
#end
# determine the state on startup
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end
# determine when process has finished starting
w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) do |c|
c.running = true
c.interval = 5.seconds
end
# failsafe
on.condition(:tries) do |c|
c.times = 5
c.transition = :start
c.interval = 5.seconds
end
end
# start if process is not running
w.transition(:up, :start) do |on|
on.condition(:process_running) do |c|
c.running = false
end
end
end
This then gives you a really elegant way to reload code in your workers without interrupting jobs - simply kill -2 your resque-pool(s) when you deploy. Idle workers will die immediately, busy workers will die when they finish their current jobs, and God will restart resque-pool with workers using your new code.
These are our Resque tasks for Capistrano:
namespace :resque do
desc "Starts resque-pool daemon."
task :start, :roles => :app, :only => { :jobs => true } do
run "cd #{current_path};resque_pool -d -e #{rails_env} start"
end
desc "Sends INT to resque-pool daemon to close master, letting workers finish their jobs."
task :stop, :roles => :app, :only => { :jobs => true } do
pid = "#{current_path}/tmp/pids/resque-pool.pid"
sudo "kill -2 `cat #{pid}`"
end
desc "Restart resque workers - actually uses resque.stop and lets God restart in due course."
task :restart, :roles => :app, :only => { :jobs => true } do
stop # let God restart.
end
desc "List all resque processes."
task :ps, :roles => :app, :only => { :jobs => true } do
run 'ps -ef f | grep -E "[r]esque-(pool|[0-9])"'
end
desc "List all resque pool processes."
task :psm, :roles => :app, :only => { :jobs => true } do
run 'ps -ef f | grep -E "[r]esque-pool"'
end
end
You might need to reconnect any DB connections when resque-pool forks workers - check the docs.
I had the same problem and the following works for me.
PIDFILE=./resque.pid BACKGROUND=yes QUEUE="*" rake resque:work >> worker1.log &
You can also redirect STDERR to the same log file.
To demonize a process you can use nohup:
nohup cmd &
On resque's github there is a config for monit, that shows how to use nohup, it looks something like this:
nohup bundle exec rake resque:work QUEUE=queue_name PIDFILE=tmp/pids/resque_worker_QUEUE.pid & >> log/resque_worker_QUEUE.log 2>&1
Another option you should look into is using the resque pool gem to manage your workers.
You can run resque pool in background by using this command:
resque-pool --daemon --environment production
The BACKGROUND environment variable was added to Resque 1.20; make sure you're not using 1.19 or lower.
One good way is to use God to manage it. It launches a daemonized version of Resque and monitor it. Actually, you can choose between using Resque as a daemon and letting God daemonize Resque. I choose option 2.
A resque.god file example :
rails_env = ENV['RAILS_ENV'] || "production"
rails_root = ENV['RAILS_ROOT'] || "/path/to/my/app/current"
num_workers = rails_env == 'production' ? 5 : 2
num_workers.times do |num|
God.watch do |w|
w.dir = "#{rails_root}"
w.name = "resque-#{num}"
w.group = 'resque'
w.interval = 30.seconds
w.env = {"QUEUE"=>"critical,mailer,high,low", "RAILS_ENV"=>rails_env}
w.start = "bundle exec rake -f #{rails_root}/Rakefile resque:work"
w.stop_signal = 'QUIT'
w.stop_timeout = 20.seconds
w.uid = 'myappuser'
w.gid = 'myappuser'
w.behavior(:clean_pid_file)
# restart if memory gets too high
w.transition(:up, :restart) do |on|
on.condition(:memory_usage) do |c|
c.above = 350.megabytes
c.times = 2
c.notify = {:contacts => ['maxime'], :priority => 9, :category => 'myapp'}
end
end
# determine the state on startup
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) do |c|
c.running = true
end
end
# determine when process has finished starting
w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) do |c|
c.running = true
c.interval = 5.seconds
end
# failsafe
on.condition(:tries) do |c|
c.times = 5
c.transition = :start
c.interval = 5.seconds
end
end
# start if process is not running
w.transition(:up, :start) do |on|
on.condition(:process_running) do |c|
c.running = false
c.notify = {:contacts => ['maxime'], :priority => 1, :category => 'myapp'}
end
end
end
end
I also faced this issue, I start worker in cap task, but I got issue
BACKGROUND causes worker always in starting mode.
nohup process is killed right after finish, we must wait a couple seconds. But unable to append more command after '&'
At last, I must create a shell, let it sleep 5s after nohup... call.
My code
desc 'Start resque'
task :start, :roles => :app do
run("cd #{current_path} ; echo \"nohup bundle exec rake resque:work QUEUE=* RAILS_ENV=#{rails_env} PIDFILE=tmp/pids/resque_worker_1.pid &\nnohup bundle exec rake resque:work QUEUE=* RAILS_ENV=#{rails_env} PIDFILE=tmp/pids/resque_worker_2.pid &\nsleep 5s\" > startworker.sh ")
run("cd #{current_path} ; chmod +x startworker.sh")
run("cd #{current_path} ; ./startworker.sh")
run("cd #{current_path} ; rm startworker.sh")
end
I know this is a situation solution. but it works well in my project
You can manage your workers with this script. Commands available:
rake resque:start_workers
rake resque:stop_workers
rake resque:restart_workers
There is also included resque-scheduler. Comment this lines to disable it:
pid = spawn(env_vars, 'bundle exec rake resque:scheduler', ops_s)
Process.detach(pid)