I have my rails site deployed under apache. The apache is run as a service. Now I have added delayed_job there and things work fine.
Now I want to start the workers together with apache, e.g, After rebooting the server, my site and workers are up and ready so I don't have to log in and type "sudo RAILS_ENV=production script/delayed_job -n 2 start".
Another issue is that whenever I want to start the delayed_job I have to use "sudo"...
Any idea how to avoid those 2 issues?
Thanks for your help.
Use the whenever gem and its 'every :reboot' functionality. In schedule.rb:
environment = ENV['RAILS_ENV'] || 'production'
every :reboot do
command "cd #{path} && #{environment_variable}=#{environment} bin/delayed_job --pool=queue1:2, --pool=queue2,queue3:1 restart"
end
Could you just create a shell script to execute the commands you need?
#!/bin/sh
# stop delayed job
# restart apache
apachectl restart
# start delayed job
sudo RAILS_ENV=production script/delayed_job -n 2 start
It sounds like you want to have delayed_job automatically start after apache starts when you boot up the hardware. If that's the case you need to write an init script in /etc/init.d or /etc/rc.d/init.d (depending on your system). This page gives a decent primer on this:
http://www.philchen.com/2007/06/04/quick-and-dirty-how-to-write-and-init-script
Related
I am running a Rails 3.2.18 application on AWS. This application is deployed using Capistrano, including starting Resque workers for the application.
My problem is that AWS can occasionally restart instances with little to no warning, or an instance can be restarted from the AWS console. When this happens, Resque is not started as it is during our normal deployment process.
I have tried to create a shell script in /etc/init.d to start Resque on boot, but this script continues to prompt for a password, and I'm not sure what I'm missing. The essence of the start script is:
/bin/su -l deploy-user -c "/www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start"
Obviously the above command works as expected when run as the "deploy" user from the bash prompt, but when run via sudo /etc/init.d/resque start, it prompts for a password upon running the first Capistrano command.
Is there something glaring that I am missing? Or perhaps is there a better way to accomplish this?
You should run su with -c parameter to specify commands, and enclose all commands within double quotes:
/bin/su -l deploy-user -c "/www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start"
Of course, you have other alternatives, like /etc/rc.local.
But if you're going to use an init.d script, I'd suggest to create it propperly (at least start/stop, default runlevels...). Otherwise I'd go with /etc/rc.local or even with a cron job for the deploy-user:
#reboot /www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start
I have some gems in my Rails App, such as resque, sunspot. I run the following command manually when the machines boots:
rake sunspot:solr:start
/usr/local/bin/redis-server /usr/local/etc/redis.conf
rake resque:work QUEUE='*'
Is there a better practice to run these daemon in the background? And is there any side-effect when run these tasks run in the background?
My solution to that is to use a mix of god, capistrano and whenever. A specific problem I have is that I want all app processes to be run as user, so initd scripts are not an option (this could be done, but it's quite a pain of user switching / environment loading).
God
The basic idea is to use god to start / restart / monitor processes. God may be difficult to get start with, but is very powerful :
running god alone will start all your processes (webserver, bg jobs, whatever)
it can detect a process crashed and restart it
you can group processes and batch restart them (staging, production, background, devops, etc)
Whenever
You still have to start god on server restart. A good mean to do so is to use user crontab. Most cron implementation have a special instruction called #reboot, which allows you to run a specific command on server restart :
#reboot /bin/bash -l -c 'cd /home/my_app && SERVER=true god -c production/current/config/app.god"
Whenever is a gem that allows easy management for crontab, including generating reboot command. While it's not absolutely necessary for achieving what I describe, it's really useful for its capistrano integration.
Capistrano
You not only want to start your processes on server restart, you also want to restart them on deploy. If your background jobs code is not up to date, problem will arise.
Capistrano allows to easily handle that, just ask god to restart the whole group (like : god restart production) in a post deploy capistrano task, and it will be handled seamlessly.
Whenever's capistrano integration also ensure your crontab is always up to date, updating it if you changed your config/schedule.rb file.
You can use something like foreman to manage these processes. You can define process types and other things in a Procfile and you can start and do whatever with them.
I'm using delayed job to create job queues such as 'mailer'
For this to work I have to run this:
$ RAILS_ENV=development QUEUE=mailer rake jobs:work
But if the server crashes and is restarted, I need the worker to start running again automatically.
What would be the recommended way to deal with this?
You need to use a third-party service like monit/bluepill/god/upstart to watch the process and restart it. I recommend using the combination of foreman and upstart. See here: http://blog.daviddollar.org/2011/05/06/introducing-foreman.html
Some time ago I wrote a patch for the DelayedJob to reload the classes for every job in development mode. Same patch should work for your requirement also.
betamatt's approach is definitely one way to do it if you have such a monitoring tool in place.
Another way to do it would simply be to add a script to your OS's startup which runs the RAILS_ENV=development QUEUE=mailer rake jobs:work command under a user who has the necessary permissions.
Here's an example of how to do it on Ubuntu using Upstart, but if you lookup similar init.d methods, or whatever is the relevant for your server OS, you'll find other ways. What you're looking for, basically, is "How to run a script on startup [your OS name]", and then wrap your command in an executable script.
I had the same issue with my application am working with. So i wrote a rake task which runs every minute(as a cron job). When delayed job starts it will create a .pid file in the temp folder. I used this to check the existence of a delayed job process. If the file doesn't exist i ran the console command through code.
delayed_job_status = File.file?("./tmp/pids/delayed_job.pid")
This will check the existence of process. If nil response go to next statement
./bundle exec script/delayed_job start production
This will start delayed job
My solution was creating the bash script in user's home "delayed_job_startup.sh"
which contain
#!/bin/bash
cd /home/deploy/project/current/
RAILS_ENV=production bin/delayed_job start
and in file /etc/rc.local I run this script from my user
su -s /bin/bash - deploy /home/deploy/delayed_job_startup.sh
I am working on a God script to monitor my Unicorns. I started with GitHub's examples script and have been modifying it to match my server configuration. Once God is running, commands such as god stop unicorn and god restart unicorn work just fine.
However, god start unicorn results in WARN: unicorn start command exited with non-zero code = 1. The weird part is that if I copy the start script directly from the config file, it starts right up like a brand new mustang.
This is my start command:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D
I have declared all paths as absolute in the config file. Any ideas what might be preventing this script from working?
I haven't used unicorn as an app server, but I've used god for monitoring before.
If I remember rightly when you start god and give your config file, it automatically starts whatever you've told it to watch. Unicorn is probably already running, which is why it's throwing the error.
Check this by running god status once you've started god. If that's not the case you can check on the command line what the comand's exit status is:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D;
echo $?;
that echo will print the exit status of the last command. If it's zero, the last command reported no errors. Try starting unicorn twice in a row, I expect the second time it'll return 1, because it's already running.
EDIT:
including the actual solution from comments, as this seems to be a popular response:
You can set an explicit user and group if your process requires to be run as a specific user.
God.watch do |w|
w.uid = 'root'
w.gid = 'root'
# remainder of config
end
My problem was that I never bundled as root. Here is what I did:
sudo bash
cd RAILS_ROOT
bundle
You get a warning telling you to never do this:
Don't run Bundler as root. Bundler can ask for sudo if it is needed,
and installing your bundle as root will break this application for all
non-root users on this machine.
But it was the only way I could get resque or unicorn to run with god. This was on an ec2 instance if that helps anyone.
Add the log option has helped me greatly in debugging.
God.watch do |w|
w.log = "#{RAILS_ROOT}/log/god.log"
# remainder of config
end
In the end, my bug turned out to be the start_script in God was executed in development environment. I fixed this by appending the RAILS_ENV to the start script.
start_script = "RAILS_ENV=#{ENV['RACK_ENV']} bundle exec sidekiq -P #{pid_file} -C #{config_file} -L #{log_file} -d"
For my RubyOnRails-App I have to start a background job at the end of Capistrano deployment. For this, I tried the following in deploy.rb:
run "nohup #{current_path}/script/runner -e production 'Scheduler.start' &", :pty => true
Sometimes this works, but most of the time it does not start the process (= not listed in ps -aux). And there are no error messages. And there is no nohup.out, not in the home directory and not in the rails app directory.
I tried using trap('SIGHUP', 'IGNORE') in scheduler.rb instead of nohup, but the result is the same.
The only way to get it work is removing the ":pty => true" and do a manual Ctrl-C at the end of "cap deploy". But I don't like this...
Are there any other chances to invoke this Scheduler.start? Or to get some more error messages?
I'm using Rails 2.3.2, Capistrano 2.5.8, Ubuntu Hardy on the Server
With :pty => true, user shell start-up scripts (e.g. bashrc, etc.) are (usually) not loaded. My ruby program exited right after launching because of the lack of dependent environment variables.
Without :pty => true, as you described in the question, capistrano hangs there waiting for the process to exit. You'll need to redirect both stdout and stderr to make it return immediately.
run 'nohup ruby -e "sleep 5" &' # hangs for 5 seconds
run 'nohup ruby -e "sleep 5" > /dev/null &' # hangs for 5 seconds
run 'nohup ruby -e "sleep 5" > /dev/null 2>&1 &' # returns immediately. good.
If your background task still doesn't run. Try redirecting stdout and stderr to a log file so that you can investigate the output.
I'd like to share my solution which also works when executing multiple commands. I tried many other variants found online, including the "sleep N" hack.
run("nohup sh -c 'cd #{release_path} && bundle exec rake task_namespace:task_name RAILS_ENV=production > ~/shared/log/<rakelog>.log &' > /dev/null 2>&1", :pty => true)
This is a dup response launching background process in capistrano task but want to make sure others and myself can google for this solution.
Do you want your Scheduler job to run continually in the background and get restarted when you run Capistrano?
If so, then for that I use runit http://smarden.sunsite.dk/runit/ and DelayedJob http://github.com/Shopify/delayed_job/tree/master
Install runit in the mode of not replacing init
Add your background job as a runit service and add the log monitor for it from runit.
Have Capistrano call sudo sv kill job_name to kill and restart the job.
My backround job is an instance of the Rails plugin DelayedJob which handles background Rails tasks. I kill it with every Capistrano deploy so it will restart with the updated code base.
This has proved to be very reliable.
HTH,
Larry
If this task scheduler has a -d switch it will work. For example passenger standalone has a -d option to start it as a demonized process.
namespace :passenger_standalone do
task :start do
run "cd #{current_path} && passenger start -e #{rails_env} -d"
end
task :stop do
run "cd #{current_path} && RAILS_ENV=#{rails_env} passenger stop"
end
task :restart do
stop
start
end
end