So i have delayed_job installed on a production app. It runs fine via rake jobs:work. But when i try to start the script via capistrano:
run "if [ -d #{current_path} ]; then cd #{current_path} && RAILS_ENV=#{rails_env} script/delayed_job start -n 2; fi"
It starts without errors. But if i check script/delayed_job status it tells me that no instances are running. Any suggestions?
Edit
looks like there is something running (via sudo ps aux | grep delayed):
1000 7952 0.0 0.1 112312 832 pts/0 S+ 16:17 0:00 grep delayed
Output when i run the script:
/path/to/latest/release/config/initializers/bypass_ssl_verification_for_open_uri.rb:2: warning: already initialized constant VERIFY_PEER
Check the permissions on your shared/tmp/pid folder.
Delayed Job won't run unless the user that capistrano is running has has permission to write the PID file into the folder.
This is how I start the delayed job demon using capistrano, maybe this will also work for you:
require "delayed/recipes"
%w[start stop restart].each do |command|
after "deploy:#{command}", "delayed_job:#{command}"
end
The output of ps aux | grep delayed only shows its own process, so DJ is not running on your machine. Maybe this has something to do with your if-clause. You could try removing that and see if it start correctly using the ps aux | grep commands. The output should be something like this:
username 9989 0.0 0.0 7640 892 pts/0 S+ 10:54 0:00 grep delayed
username 10048 0.0 9.4 288244 99156 ? Sl Jan22 2:16 delayed_job
Related
I've setup monit to monitor my sunspot_solr process, which seems to work at first. If I restart the monit service with sudo service monit restart my sunspot process starts:
ps aux | grep sunspot
root 4086 0.0 0.0 9940 1820 ? Ss 12:41 0:00 bash ./solr start -f -s /ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/solr
root 4137 45.1 4.8 1480560 185632 ? Sl 12:41 0:09 java -server -Xss256k -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:CMSFullGCsBeforeCompaction=1 -XX:CMSTriggerPermRatio=80 -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/server -Dsolr.solr.home=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/solr -Dsolr.install.dir=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr -jar start.jar --module=http
ubuntu 4192 0.0 0.0 10460 936 pts/3 S+ 12:41 0:00 grep --color=auto sunspot
However, I'm also running tail -f /var/logs/monit.log and see this at the same time:
[CST Mar 3 12:42:54] error : 'sunspot_solr' process is not running
[CST Mar 3 12:42:54] info : 'sunspot_solr' trying to restart
[CST Mar 3 12:42:54] info : 'sunspot_solr' start: /usr/bin/sudo
[CST Mar 3 12:43:25] error : 'sunspot_solr' failed to start
Plus, to make sure monit can actually restart the sunspot_solr process, I run sudo kill -9 <the pid> and monit can't restart sunspot_solr:
[CST Mar 3 12:44:25] error : 'sunspot_solr' process is not running
[CST Mar 3 12:44:25] info : 'sunspot_solr' trying to restart
[CST Mar 3 12:44:25] info : 'sunspot_solr' start: /usr/bin/sudo
[CST Mar 3 12:44:55] error : 'sunspot_solr' failed to start
Obviously something is wrong with my monit-solr_sunspot.conf file, but after messing around with it for a few hours now, I'm stumped:
check process sunspot_solr with pidfile /ebs/staging/shared/pids/sunspot-solr.pid
start program = "/usr/bin/sudo -H -u root /bin/bash -l -c 'cd /ebs/staging/releases/20160226191542; bundle exec sunspot-solr start -- -p 8983 -d /ebs/staging/shared/solr/data --pid-dir=/ebs/staging/shared/pids'"
stop program = "/usr/bin/sudo -H -u root /bin/bash -l -c 'cd /ebs/staging/releases/20160226191542; bundle exec sunspot-solr stop -- -p 8983 -d /ebs/staging/shared/solr/data --pid-dir=/ebs/staging/shared/pids'"
I've adapted this monit script to suit my needs: Sample sunspot-solr.monit but am still having no luck!
UPDATE
I've gotten monit to successfully restart sunspot_solr if I kill it, however it still produces the error that it failed to restart in the monit.log file.
I think monit runs as root. You may not want to use sudo both because it prompts for a password and because monit doesn't need it.
Using Rails 3.2.21, whenever gem. This is the list of my crontab:
Begin Whenever generated tasks for: abc
0 * * * * /bin/bash -l -c 'cd /home/deployer/abc/releases/20141201171336 &&
RAILS_ENV=production bundle exec rake backup:perform --silent'
Here's the output when the scheduled job is run:
deployer#localhost:~$ ps aux | grep rake
deployer 25593 0.0 0.0 4448 764 ? Ss 12:00 0:00 /bin/sh -c /bin/bash -l -c
'cd /home/deployer/abc/releases/20141201171336 && RAILS_ENV=production bundle exec rake
backup:perform --silent'
deployer 25594 0.0 0.1 12436 3040 ? S 12:00 0:00 /bin/bash -l -c cd
/home/deployer/abc/releases/20141201171336 && RAILS_ENV=production bundle exec rake
backup:perform --silent
deployer 25631 69.2 4.4 409680 90072 ? Sl 12:00 0:06 ruby /home/deployer/abc/
shared/bundle/ruby/1.9.1/bin/rake backup:perform --silent
deployer 25704 0.0 0.0 11720 2012 pts/0 S+ 12:00 0:00 grep --color=auto rake
Notice the the top 2 processes are actually similar processes. Are they running 2 same jobs concurrently? How do I prevent that?
deployer 25593 0.0 0.0 4448 764 ? Ss 12:00 0:00 /bin/sh -c /bin/bash …
deployer 25594 0.0 0.1 12436 3040 ? S 12:00 0:00 /bin/bash …
Notice the the top 2 processes are actually similar processes. Are they running 2 same jobs concurrently?
No, they aren't. The first is a /bin/sh that started the second, the crontab command /bin/bash …. Most probably /bin/sh is just waiting for termination of /bin/bash and not running again before /bin/bash … has finished execution; you can verify this with e. g. strace -p 25593.
Check your scheduled.rb for a duplicate entry, if you find then remove and deploy.
If there is no duplicate entry in scheduled.rb then you need to remove/comment it from cron tab.
To delete or comment jobs in cron take a look at https://help.1and1.com/hosting-c37630/scripts-and-programming-languages-c85099/cron-jobs-c37727/delete-a-cron-job-a757264.html OR http://www.esrl.noaa.gov/gmd/dv/hats/cats/stations/qnxman/crontab.html
I am using Rails 3.0.7 and the Thinking_sphinx 2.0.13.
Background:
I'm trying to create a deploy script (capistrano) where sphinx will be stoped before capistrano updates the code, and then started again when the update is done.
However, the rake task for thinking_sphinx:start fails.
The commands below are run in this exact order.
$ rake thinking_sphinx:stop
Stopped search daemon (pid 54117).
searchd is not running
$ ps aux | grep searchd
xx 54597 0.0 0.0 2434892 532 s002 R+ 4:04PM 0:00.00 grep searchd
$ rake thinking_sphinx:start
Started successfully (pid 54618).
rake aborted!
searchd is already running.
Tasks: TOP => thinking_sphinx:start
(See full trace by running task with --trace)
$ ps aux | grep searchd
xx 54637 0.8 0.0 2434892 448 s002 R+ 4:06PM 0:00.00 grep searchd
xx 54618 0.0 0.0 2442992 396 s002 S 4:05PM 0:00.02 searchd --pidfile --config /Users/emil/code/wd/config/development.sphinx.conf
So yes, it starts but the rake task fails. When I run these commands in "cap deploy", capistrano will rollback my updated code because the rake task fails.
How do I solve this? I have no ideas left.
I'm trying to set up Monit to manage Resque workers, but it fails to start saying /home/deployer/.rbenv/shims/bundle: line 4: exec: rbenv: not found
I've checked that it is running commands as deployer user and if I copy and paste the command directly via SSH everything works fine. Below is my Monit configuration. Thanks!
check process resque_worker_1
with pidfile CURRENT_PATH/tmp/pids/resque_worker_1.pid
start program = "/usr/bin/env HOME=/home/deployer RACK_ENV=production PATH=/home/deployer/.rbenv/shims:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c 'cd CURRENT_PATH; bundle exec rake environment resque:work RAILS_ENV=production QUEUE=high,normal,low VERBOSE=1 PIDFILE=CURRENT_PATH/tmp/pids/resque_worker_1.pid >> CURRENT_PATH/log/resque_worker_.log 2>&1'"
as uid deployer and gid admin
stop program = "/bin/sh -c 'cd CURRENT_PATH && kill -9 $(cat tmp/pids/resque_worker_1.pid) && rm -f tmp/pids/resque_worker_1.pid; exit 0;'"
as uid deployer and gid admin
if totalmem is greater than 300 MB for 10 cycles then restart # eating up memory?
group resque_workers
I'm not sure if this helps, but in my monitrc start line, I have to first su to the user I want to run under. I haven't tried to use the uid and gid flags to know if that works well, so this might be a goose-chase of an answer.
I remember having the same issue as you though... everything worked from the command line, but not when monit would do its thing.
For example, in my monitrc, I am monitoring arsendmail with the following:
# arsendmail_rails3
# daemon that watches and sends mail from the rails app
check process ar_sendmail with pidfile /var/www/rak/log/ar_sendmail.pid
start program "/bin/su - mike && /bin/bash -c 'cd /var/www/rak && ar_sendmail_rails3 -b1000 -d -e production'"
stop program "/bin/ps -ef | /bin/grep ar_sendmail_rails3 | /bin/grep -v grep | /usr/bin/awk '{ /usr/bin/print $2}' | /usr/bin/xargs /bin/kill -9"
I saw that the topic was created in 2012 but I had a similar problem and this thread is top ranked by google.
The problem is that monit launch commands with a restricted env (env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh to simulate).
To use monit with rbenv you must specify the correct path before your bundle exec command.
PATH=/home/[USER]/.rbenv/bin:/home/[USER]/.rbenv/shims:$PATH bundle exec ...
Example with unicorn:
check process unicorn_dev with pidfile /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid
group dev
start program = "/bin/bash -c 'cd /home/wizville/app/dev.wizville.fr/current && PATH=/home/wizville/.rbenv/bin:/home/wizville/.rbenv/shims:$PATH bundle exec unicorn -c config/unicorn.rb -D'" as uid "wizville"
stop program = "/bin/bash -c 'kill -s QUIT `cat /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid`'"
depends on mysql
This worked for me.
check process app_resque_worker with pidfile <%= resque_pid%>
start program = "/usr/bin/env HOME=/home/subcaster RACK_ENV=production PATH=/home/subcaster/.rvm/rubies/ruby-2.0.0-p247/bin/ruby:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c \'cd <%= current_path %>; bundle exec rake environment resque:work RAILS_ENV=production BACKGROUND=yes QUEUE=* PIDFILE=<%= resque_pid %>\'"
stop program = "kill -9 cat <%= resque_pid%> && rm -f <%= resque_pid%>"
if totalmem is greater than 2000 MB for 10 cycles then restart
How do I stop delayed_job if I'm running it with the -m "monitor" option? The processes keep getting restarted!
The command I start delayed_job with is:
script/delayed_job -n 4 -m start
The -m runs a monitor processes that spawns a new delayed_job process if one dies.
The command I'm using to stop is:
script/delayed_job stop
But that doesn't stop the monitor processes, which in turn start up all the processes again. I would just like them to go away. I can kill them, which I have, but I was hoping there was some command line option to just shut the whole thing down.
In my capistrano deploy script I have this:
desc "Start workers"
task :start_workers do
run "cd #{release_path} && RAILS_ENV=production script/delayed_job -m -n 2 start"
end
desc "Stop workers"
task :stop_workers do
run "ps xu | grep delayed_job | grep monitor | grep -v grep | awk '{print $2}' | xargs -r kill"
run "cd #{current_path} && RAILS_ENV=production script/delayed_job stop"
end
To avoid any errors that may stop your deployment script:
"ps xu" only show processes owned by the current user
"xargs -r kill" only invoke the kill command when there is something to kill
I only kill the delayed_job monitor, and stop the delayed_job deamon the normal way.
I had this same problem. Here's how I solved it:
# ps -ef | grep delay
root 8605 1 0 Jan03 ? 00:00:00 delayed_job_monitor
root 15704 1 0 14:29 ? 00:00:00 dashboard/delayed_job
root 15817 12026 0 14:31 pts/0 00:00:00 grep --color=auto delay
Here you see the delayed_job process and the monitor. Next, I would manually kill these processes and then delete the PIDs. From the application's directory (/usr/share/puppet-dashboard in my case):
# ps -ef | grep delay | grep -v grep | awk '{print $2}' | xargs kill && rm tmp/pids/*
The direct answer is that you have to kill the monitor process first. However AFAIK there isn't an easy way to do this, I don't think the monitor PIDs are stored anywhere and the DJ start and stop script certainly doesn't do anything intelligent there, as you noticed.
I find it odd that the monitor feature was included -- I guess Daemons has it so whomever was writing the DJ script figured they would just pass that option down. But it's not really usable as it is.
I wrote an email to the list about this a while back, didn't get an answer: https://groups.google.com/d/msg/delayed_job/HerSuU97BOc/n4Ps430AI1UJ
You can see more about monitoring with Daemons here: http://daemons.rubyforge.org/classes/Daemons.html#M000004
If you come up with a better answer/solution, add it to the wiki here: https://github.com/collectiveidea/delayed_job/wiki/monitor-process
if you can access server, you can try these commands:
ps -ef | grep delayed_job
kill -9 XXXX #xxxx is process id
OR
cd path-to-app-folder/current
RAILS_ENV=production bin/delay_job stop
RAILS_ENV=production bin/delay_job start
You can also add this script to capistrano3 in config/deploy.rb
namespace :jobworker do
task :start do
on roles(:all) do
within "#{current_path}" do
with rails_env: "#{fetch(:stage)}" do
execute "bin/delayed_job start"
end
end
end
end
task :stop do
on roles(:all) do
within "#{current_path}" do
with rails_env: "#{fetch(:stage)}" do
execute "bin/delayed_job stop"
end
end
end
end
end
then run
cap
cap production jobworker:stop
cap production jobworker:start