I have an application which relies heavily on delayed jobs. So I have setup two servers, one of which servers up the application (m3.medium ec2 instance) while other one runs my delayed jobs(t2.micro ec2 instance). I have created a start and stop script for the delayed jobs. This is where I am facing issues. Delayed jobs run smoothly but the problem is that they stop automatically after some time. So everytime they stop I have to manually start them again. I have no clue whatsoever why they stop in the middle of processing a job.
So basically I have two questions:
What can I do so that the jobs don't stop, or if they do they start automatically immediately/or after some time?
How can I make them start automatically on instance reboot/start?
I have looked at many similar questions, but none seem to help.
Any advice appreciated.
Edit 1:
My start/stop script for the delayed jobs.
set -e
# drop privs if necessary
if [ "$(id -u)" == "0" ]; then
exec su $(stat -c %U $(dirname $(readlink -f $0))/../config/environment.rb) -c "/bin/bash $0 $#"
exit -1;
fi
# switch to app root
cd $(dirname $(readlink -f $0))/..
# set up config
if [ -e "config/GEM_HOME" ]; then
export GEM_HOME=$(cat config/GEM_HOME)
fi
#export GEM_HOME=/path/to/gem/home
export RAILS_ENV=production
# run delayed jobs
exec script/delayed_job $#
# following an article I have tried adding the following code restart on crash.
# restarting the service
respawn
#Give up if restart occurs 10 times in 90 seconds.
respawn limit 10 90
Its seem you might be having memory issue which is killing it. you can try Monit to automatically start job if its killed
ref: http://railscasts.com/episodes/375-monit
Alternative:
You may also use sidekiq instead of delayed job
Related
I started the sidekiq by
bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production
Sometime, Sidekiq was crushed, busy=0 and enqueue > 0.
How can I setup the sidekiq to restart after crushed/stopped?
You can use some automated packages, or write your own bash script to do that.
I personally prefer writing my own scripts so let me explain that:
Write a script that executes sidekiq if not running already
Write a cronjob to execute that script every minute
Note that this method is not instant, meaning that you might have a downtime up to 1 minute, since the cronjob works every minute. So if your project is sensitive on that, you might want to use one of the process management tools such as monit or god.
Your bash script should contain your command,
bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production
But make sure you are using absolute paths if you are going to store your script outside your project directory.
Here is a helpful topic about writing the shell script to check whether the process already exists.
To run the script every minute, do the following:
Go to your terminal
Type crontab -e
Append * * * * * /bin/bash -l -c 'cd /PATH/TO/YOUR/DIR && sh SCRIPTNAME.sh'
Save and exit editor
By doing this, you are telling your computer to execute your script every minute.
Usually people prefer writing only * * * * * /path/to/script.sh to their crontab but this fails in some circumstances.
Hope this helps.
I'm using Ubuntu 14.04.4 LTS
Crontab:
SHELL=/bin/bash
#reboot ~/Projects/MyAPI/startworkers.sh;
startup script:
# /Projects/MyAPI/startworkers.sh
#!/bin/bash
source /home/server-linux/.bashrc
cd ~/Projects/LucyAPI
# Start background workers
bin/delayed_job --pool=tracking:2 --pool=emailverify:6 start
I expect there to be 6 delayed jobs running after reboot. However, none of them start. However, if I manually execute start.sh everything works as expected.
What am I doing wrong?
I think you might need a /bin/bash as part of the crontab and also the absolute path to the user home. Crontab example:
#reboot /bin/bash -l -c '/home/your_user_name/Projects/MyAPI/startworkers.sh'
I would also strongly recommend using the whenever gem to handle your crontab. You can find it here Whenever Gem
Help!
I want to set up a monitoring service on my Debian server, that will monitor and start wen needed the updater for tiny tiny rss. The problem is that it is a php foreground process normally run in a screen on a non-root user.
I can run it as:
php ./update_daemon2.php
or better putting it in the background and in order to run it from a different account
sudo -u tinyrssuser php ./update_deamon2.php -daemon > /dev/null & disown $!
I have installed monit, but cant seem to find a way to have it detect if t is running.
I would prefer to keep with monit but it is not necessary.
Any ideas would be appreciated.
Found the answer at:
http://510x.se/notes/posts/Install_Tiny_Tiny_RSS_on_Debian/
But use this instead under /etc/init.d/
http://mylostnotes.blogspot.co.il/2013/03/tiny-tiny-rss-initd-script.html
make sure to set the user and group
Create an upstart script /etc/init/ttrss.conf:
description "TT-RSS Feed Updater"
author "The Epyon Avenger <epyon_avenger on TT-RSS forums>"
env USER=www-data
env TTRSSDIR=/var/www/ttrss
start on started mysql
stop on stopping mysql
respawn
exec start-stop-daemon --start --make-pidfile --pidfile /var/run/ttrss.pid --chdir $TTRSSDIR --chuid $USER --group $USER --exec /usr/bin/php ./update_daemon2.php >> /var/log/ttrss/ttrss. log 2>&1
Start the script:
sudo start --system ttrss
Add the following lines to your monit conf:
check process ttrss with pidfile /var/run/ttrss.pid
start program = "/sbin/start ttrss"
stop program = "/sbin/stop ttrss"
I have made several jobs that god takes care of in my ruby application. However when the server reboots the job stops. I want to avoid this so I've made this script on my server. It looks like this.
my_app.sh
#!/bin/bash
# god tasks
#
case $1 in
start)
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/god
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/god start
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/god load /usr/local/Linux/apache2/www/hej.se/ruby/config/resque.god
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/god load /usr/local/Linux/apache2/www/hej.se/ruby/config/resque_schedule.god
;;
esac
exit 0
If I log in manually and write
"/etc/init.d/my_app start"
it gives me
Sending 'start' command
No matching task or group
Sending 'load' command with action 'leave'
The following tasks were affected:
resque-0
resque-1
resque-2
resque-3
resque-4
Sending 'load' command with action 'leave'
The following tasks were affected:
resque_scheduler
And everything works, it does what I want it to do, i.e the jobs.
I have tried several ways to start this script on boot (Linux 10.4.4 LTS), rc.local, rc-default and now my latest attempt is crontab.
The script must be run under my user and not root, (it can't find the ruby installation if I run it under root).
Because of this I've configured the crontab under my user account:
#reboot /etc/init.d/my_app start
Sadly this doesn't work... I don't what I'm doing wrong. And this should probably not be necessary. I mean shouldn't you be able to this per auto when booting up the ruby application?
Im using passenger on this server, I don't know if this has something to do with it?
The solution below with the changes I made to the sh:
my_app.sh
bash -c "source /usr/local/rvm/scripts/rvm && /usr/local/rvm/gems/ruby-1.9.3-p194/bin/god"
bash -c "source /usr/local/rvm/scripts/rvm && /usr/local/rvm/gems/ruby-1.9.3-p194/bin/god start"
bash -c "source /usr/local/rvm/scripts/rvm && /usr/local/rvm/gems/ruby-1.9.3-p194/bin/god load /usr/local/Linux/apache2/www/hej.se/ruby/config/resque.god"
bash -c "source /usr/local/rvm/scripts/rvm && /usr/local/rvm/gems/ruby-1.9.3-p194/bin/god load /usr/local/Linux/apache2/www/hej.se/ruby/config/resque_schedule.god"
Forget the cronjob.
Centos/Fedora:
sudo chmod a+x /etc/init.d/my_app
sudo chkconfig --add my_app
sudo chkconfig my_app on
Ubuntu/Debian:
sudo update-rc.d my_app defaults
Both of these symlink the script to /etc/rc1.d, /etc/rc2.d, etc., and make the script available to run on boot for those runlevels.
Hi I have to restart Apache from rails controller I tried to do that with %x{} and system commands but it fails so I decided to do it with cron Is it possible to make cron task that will be executed only once ?
The run once version of cron is called at. See http://en.wikipedia.org/wiki/At_%28Unix%29 for an explanation, and note that specifying "now" as the time causes it to run immediately.
To schedule a cron job to run only once is a bit tricky but can be done by a self deleting script! Schedule your script in the cron for the next minute or for other preferable time,
* * * * * /path/to/self-deleting-script
The self deleting script will be like,
#!/bin/bash
# <your job here>
crontab -l | grep -v $0 | crontab - # to delete your script from the cron
#restart your cron service
rm -f $0 #delete the script now
It solves my problem in an openwrt router where I could not install at command.