I am running a Rails 3.2.18 application on AWS. This application is deployed using Capistrano, including starting Resque workers for the application.
My problem is that AWS can occasionally restart instances with little to no warning, or an instance can be restarted from the AWS console. When this happens, Resque is not started as it is during our normal deployment process.
I have tried to create a shell script in /etc/init.d to start Resque on boot, but this script continues to prompt for a password, and I'm not sure what I'm missing. The essence of the start script is:
/bin/su -l deploy-user -c "/www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start"
Obviously the above command works as expected when run as the "deploy" user from the bash prompt, but when run via sudo /etc/init.d/resque start, it prompts for a password upon running the first Capistrano command.
Is there something glaring that I am missing? Or perhaps is there a better way to accomplish this?
You should run su with -c parameter to specify commands, and enclose all commands within double quotes:
/bin/su -l deploy-user -c "/www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start"
Of course, you have other alternatives, like /etc/rc.local.
But if you're going to use an init.d script, I'd suggest to create it propperly (at least start/stop, default runlevels...). Otherwise I'd go with /etc/rc.local or even with a cron job for the deploy-user:
#reboot /www/apps/deploy-user/current && bundle exec cap <environment> resque:scheduler:start resque:start
Related
I use the 'whenever' gem for my rails cron file in EC2 and it works great. "Whenever -w" writes it and I never have to worry about it again. The problem is when my instance has a planned reboot. The rails app get passed to a new instance and the whole process is seamless with no downtime, but the new instance does not have my cron file.
How can I make sure that the cron file gets written when I move to a new instance? Is there a way to run it on app start or something like that? Thanks.
Whenever is a command. If you run:
bundle exec whenever --help
You will get a full list of flags. You'll want:
bundle exec whenever -w /path/to/schedule.rb
You can add a simple shell script to /etc/init.d
#!/bin/bash
cd /to/app && /full/path/to/bundle exec whenever -w /full/path/to/schedule.rb
I'm trying to set daily cron job to update my site stats, but it looks like it doesn't work.
Cron entry (for deployer user):
0 0 * * * cd /var/www/my_site/current && rake RAILS_ENV=production stats:update
I'm running ubuntu server, with rbenv.
Any idea what's wrong?
Many times $PATH is defined differently when cron runs compared to when you are working in your own shell. Do "whereis rake" to find the full path to rake and then replace "rake" with its full path. (I am assuming that the "cd" command is working, so I am focusing on whether "rake" is found / running properly.)
Has cron sent you any emails with error messages after you added your command to your crontab?
You might want to run "crontab -l" under the proper user account to make sure that your cron command is actually registered within the crontab, especially if you aren't receiving any emails.
The presence of a Gemfile can also affect the ability to properly run rake. See, for example, Error: "Could not find rake", yet Rake is installed
I have my rails site deployed under apache. The apache is run as a service. Now I have added delayed_job there and things work fine.
Now I want to start the workers together with apache, e.g, After rebooting the server, my site and workers are up and ready so I don't have to log in and type "sudo RAILS_ENV=production script/delayed_job -n 2 start".
Another issue is that whenever I want to start the delayed_job I have to use "sudo"...
Any idea how to avoid those 2 issues?
Thanks for your help.
Use the whenever gem and its 'every :reboot' functionality. In schedule.rb:
environment = ENV['RAILS_ENV'] || 'production'
every :reboot do
command "cd #{path} && #{environment_variable}=#{environment} bin/delayed_job --pool=queue1:2, --pool=queue2,queue3:1 restart"
end
Could you just create a shell script to execute the commands you need?
#!/bin/sh
# stop delayed job
# restart apache
apachectl restart
# start delayed job
sudo RAILS_ENV=production script/delayed_job -n 2 start
It sounds like you want to have delayed_job automatically start after apache starts when you boot up the hardware. If that's the case you need to write an init script in /etc/init.d or /etc/rc.d/init.d (depending on your system). This page gives a decent primer on this:
http://www.philchen.com/2007/06/04/quick-and-dirty-how-to-write-and-init-script
I am working on a God script to monitor my Unicorns. I started with GitHub's examples script and have been modifying it to match my server configuration. Once God is running, commands such as god stop unicorn and god restart unicorn work just fine.
However, god start unicorn results in WARN: unicorn start command exited with non-zero code = 1. The weird part is that if I copy the start script directly from the config file, it starts right up like a brand new mustang.
This is my start command:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D
I have declared all paths as absolute in the config file. Any ideas what might be preventing this script from working?
I haven't used unicorn as an app server, but I've used god for monitoring before.
If I remember rightly when you start god and give your config file, it automatically starts whatever you've told it to watch. Unicorn is probably already running, which is why it's throwing the error.
Check this by running god status once you've started god. If that's not the case you can check on the command line what the comand's exit status is:
/usr/local/bin/unicorn_rails -c /home/my-linux-user/my-rails-app/config/unicorn.rb -E production -D;
echo $?;
that echo will print the exit status of the last command. If it's zero, the last command reported no errors. Try starting unicorn twice in a row, I expect the second time it'll return 1, because it's already running.
EDIT:
including the actual solution from comments, as this seems to be a popular response:
You can set an explicit user and group if your process requires to be run as a specific user.
God.watch do |w|
w.uid = 'root'
w.gid = 'root'
# remainder of config
end
My problem was that I never bundled as root. Here is what I did:
sudo bash
cd RAILS_ROOT
bundle
You get a warning telling you to never do this:
Don't run Bundler as root. Bundler can ask for sudo if it is needed,
and installing your bundle as root will break this application for all
non-root users on this machine.
But it was the only way I could get resque or unicorn to run with god. This was on an ec2 instance if that helps anyone.
Add the log option has helped me greatly in debugging.
God.watch do |w|
w.log = "#{RAILS_ROOT}/log/god.log"
# remainder of config
end
In the end, my bug turned out to be the start_script in God was executed in development environment. I fixed this by appending the RAILS_ENV to the start script.
start_script = "RAILS_ENV=#{ENV['RACK_ENV']} bundle exec sidekiq -P #{pid_file} -C #{config_file} -L #{log_file} -d"
For my RubyOnRails-App I have to start a background job at the end of Capistrano deployment. For this, I tried the following in deploy.rb:
run "nohup #{current_path}/script/runner -e production 'Scheduler.start' &", :pty => true
Sometimes this works, but most of the time it does not start the process (= not listed in ps -aux). And there are no error messages. And there is no nohup.out, not in the home directory and not in the rails app directory.
I tried using trap('SIGHUP', 'IGNORE') in scheduler.rb instead of nohup, but the result is the same.
The only way to get it work is removing the ":pty => true" and do a manual Ctrl-C at the end of "cap deploy". But I don't like this...
Are there any other chances to invoke this Scheduler.start? Or to get some more error messages?
I'm using Rails 2.3.2, Capistrano 2.5.8, Ubuntu Hardy on the Server
With :pty => true, user shell start-up scripts (e.g. bashrc, etc.) are (usually) not loaded. My ruby program exited right after launching because of the lack of dependent environment variables.
Without :pty => true, as you described in the question, capistrano hangs there waiting for the process to exit. You'll need to redirect both stdout and stderr to make it return immediately.
run 'nohup ruby -e "sleep 5" &' # hangs for 5 seconds
run 'nohup ruby -e "sleep 5" > /dev/null &' # hangs for 5 seconds
run 'nohup ruby -e "sleep 5" > /dev/null 2>&1 &' # returns immediately. good.
If your background task still doesn't run. Try redirecting stdout and stderr to a log file so that you can investigate the output.
I'd like to share my solution which also works when executing multiple commands. I tried many other variants found online, including the "sleep N" hack.
run("nohup sh -c 'cd #{release_path} && bundle exec rake task_namespace:task_name RAILS_ENV=production > ~/shared/log/<rakelog>.log &' > /dev/null 2>&1", :pty => true)
This is a dup response launching background process in capistrano task but want to make sure others and myself can google for this solution.
Do you want your Scheduler job to run continually in the background and get restarted when you run Capistrano?
If so, then for that I use runit http://smarden.sunsite.dk/runit/ and DelayedJob http://github.com/Shopify/delayed_job/tree/master
Install runit in the mode of not replacing init
Add your background job as a runit service and add the log monitor for it from runit.
Have Capistrano call sudo sv kill job_name to kill and restart the job.
My backround job is an instance of the Rails plugin DelayedJob which handles background Rails tasks. I kill it with every Capistrano deploy so it will restart with the updated code base.
This has proved to be very reliable.
HTH,
Larry
If this task scheduler has a -d switch it will work. For example passenger standalone has a -d option to start it as a demonized process.
namespace :passenger_standalone do
task :start do
run "cd #{current_path} && passenger start -e #{rails_env} -d"
end
task :stop do
run "cd #{current_path} && RAILS_ENV=#{rails_env} passenger stop"
end
task :restart do
stop
start
end
end