I am deploying my rails application using Capistrano.
Once I have made changes to my app I do "cap deploy" and it seems working properly, but changes don't take effect. I always have to do "cap deploy:stop" and "cap deploy:start" and then everything is fine.
So I guess it has to do with "cap deploy:restart" which is run when deploying changes.
here is my deploy.rb: deploy.rb gist
Hope that someone can help.
Thank you in advance
If you restart unicorn using the USR2 signal it doesn't automatically know the correct environment for bundler. Check out this gist (specially the before_exec block) and adjust your unicorn config accordingly.
https://gist.github.com/534668
Hope that helps.
Just noticed in line (105) in your gist
task :restart, :except => { :no_release => true } do
if File.exist?("/tmp/unicorn.example.pid")
run "kill -s USR2 `cat /tmp/unicorn.example.pid`"
end
end
that you are testing for the pid file existence on your local machine. Instead you should do that on your server. Try changing it to
task :restart, :except => { :no_release => true } do
run "test -f /tmp/unicorn.example.pid && kill -s USR2 `cat /tmp/unicorn.example.pid`"
end
But remember that it still fails silently if the pid file is missing.
Related
I've been using this gem for a while and just took the dive to try deploying an actual staging environment to my staging server, and I ran into issues. Unicorn starts with the command unicorn_rails and -E production despite all the settings being correct afaik.
I noticed in deploy.rb that my unicorn_bin variable was set as unicorn_rails. I took out this setting in my deploy.rb. However unicorn:duplicate still executes the unicorn_rails command, when the default should be unicorn.
My vars are all set to staging in the deploy/staging.rb, as outlined in the multistage setup wiki document, but I noticed -E is still getting set to production.
Relevent info:
Here's my output from my unicorn.log file after a deploy:
executing ["/var/www/apps/myapp/shared/bundle/ruby/2.0.0/bin/unicorn_rails", "-c", "/var/www/apps/bundio/current/config/unicorn.rb", "-E", "production", "-D", {12=>#<Kgio::UNIXServer:/tmp/bundio.socket>, 13=>#<Kgio::TCPServer:fd 13>}] (in /var/www/apps/bundio/current)
Here's the output from cap -T (defaults to staging)
# Environments
rails_env "staging"
unicorn_env "staging"
unicorn_rack_env "staging"
# Execution
unicorn_user nil
unicorn_bundle "/usr/local/rvm/gems/ruby-2.0.0-p247#global/bin/bundle"
unicorn_bin "unicorn"
unicorn_options ""
unicorn_restart_sleep_time 2
# Relative paths
app_subdir ""
unicorn_config_rel_path "config"
unicorn_config_filename "unicorn.rb"
unicorn_config_rel_file_path "config/unicorn.rb"
unicorn_config_stage_rel_file_path "config/unicorn/staging.rb"
# Absolute paths
app_path "/var/www/apps/myapp/current"
unicorn_pid "/var/www/apps/myapp/shared/pids/unicorn.myapp.pid"
bundle_gemfile "/var/www/apps/myapp/current/Gemfile"
unicorn_config_path "/var/www/apps/myapp/current/config"
unicorn_config_file_path "/var/www/apps/myapp/current/config/unicorn.rb"
unicorn_config_stage_file_path
-> "/var/www/apps/myapp/current/config/unicorn/staging.rb"
And another curiousity, the unicorn_rails -E flag should reference the rails environment, whereas the unicorn -E should reference the rack env -- the rack env should only get the values developement and deployment, but it gets set to production, which is a bit strange (see unicorn docs for settings of the RACK_ENV variable.
Any insight into this would be much appreciated. On my staging server, I've also set the RAILS_ENV to staging. I've set up the things for rails for another environment, like adding staging.rb in my environments folder, adding a staging section to database.yml, etc.
Important lines in lib/capistrano-unicorn/config.rb talking about unicorn_rack_env:
_cset(:unicorn_env) { fetch(:rails_env, 'production' ) }
_cset(:unicorn_rack_env) do
# Following recommendations from http://unicorn.bogomips.org/unicorn_1.html
fetch(:rails_env) == 'development' ? 'development' : 'deployment'
end
Thanks in advance.
Ok, after a long time not having the correct environment, I have discovered the issue!
Basically, my init scripts were running BEFORE my capistrano-unicorn bin was doing its thing.
So, make sure that your init.d or upstart scripts to manage Unicorn and its workers are taken into account when capistrano-unicorn is doing the unicorn restart / reload / duplication tasks.
I did not think to look at these scripts when I had to debug the stale pid file / already running / unable to listen on socket errors. But it makes sense, as upstart starts Unicorn when it is not running, and then capistrano-unicorn is also attempting to start Unicorn.
I have now combined these capistrano tasks and hooks with Monit and a Unicorn init script.
Capistrano tasks:
namespace :monit do
desc ' wait 20 seconds '
task :wait_20_seconds do
sleep 20
end
task :monitor_all, :roles => :app do
sudo "monit monitor all"
end
task :unmonitor_all, :roles => :app do
sudo "monit unmonitor all"
end
desc 'monitor unicorn in the monit rc file'
task :monitor_unicorn, :roles => :app do
sudo "monit monitor unicorn"
end
desc 'unmonitor unicorn in the monit rc file'
task :unmonitor_unicorn, :roles => :app do
sudo "monit unmonitor unicorn"
end
end
Capistrano hooks:
after 'deploy:restart', 'unicorn:duplicate' # app preloaded. check https://github.com/sosedoff/capistrano-unicorn section for zero downtime
before 'deploy', "monit:unmonitor_unicorn"
before 'deploy:migrations', "monit:unmonitor_unicorn"
after 'deploy', 'monit:wait_20_seconds'
after "deploy:migrations", "monit:wait_20_seconds"
after 'monit:wait_20_seconds', 'monit:monitor_unicorn'
I use Monit to monitor my unicorn process:
Within /etc/monit/monitrc:
check process unicorn
with pidfile /var/www/apps/my_app/shared/pids/mypid.pid
start program = "/usr/bin/sudo service unicorn start"
stop program = "/usr/bin/sudo service unicorn stop"
Within your init script, you will start the unicorn process with something like:
unicorn_rails -c /var/www/apps/my_app/current/config/unicorn.rb -E staging -D
Make sure the -E flag is set to the correct environment. The capistrano-unicorn gem has directives using :set within deploy.rb which allow you to specify the environment for that unicorn process.
The app deploys fine if I want to tolerate downtime from manually stopping and starting unicorn after a deployment. However, I want to use the zero-downtime unicorn settings, but it isn't working because the new unicorn process that starts up is looking at the old deployment release path. Nothing special, simple cap restart in deploy.rb:
desc "Zero-downtime restart of Unicorn"
task :restart, :except => { :no_release => true } do
run "cd #{current_path}; #{try_sudo} kill -s USR2 `cat /var/www/appname/shared/pids/unicorn.pid`"
end
I know that it's looking at the wrong directory because if the views don't change, and if I set keep_releases to 1 or 2, the unicorn logs will show an error because the directory it's trying to start up in was deleted:
/var/www/appname/shared/bundle/ruby/1.9.1/gems/unicorn-4.4.0/lib/unicorn/http_server.rb:425:in `chdir': No such file or directory - /var/www/appname/releases/20130330104246 (Errno::ENOENT)
I've been trying to debug this on and off for several weeks now. Any help getting this working is greatly appreciated!
Set this environment variable when starting unicorn
BUNDLE_GEMFILE=$APP_PATH/current/Gemfile
Otherwise it will point at a specific release directory, which will cause the behaviour you describe.
eg.
cd $APP_PATH/current && BUNDLE_GEMFILE=$APP_PATH/current/Gemfile bundle exec unicorn_rails -c $APP_PATH/current/config/unicorn.rb -E $RAILS_ENV -D
I've got following settings in deploy.rb to restart my server:
namespace :deploy do
task :restart do
run "if [ -f #{unicorn_pid} ] && [ -e /proc/$(cat #{unicorn_pid}) ]; then kill -USR2 \`cat #{unicorn_pid}\`; else cd #{deploy_to}/current && bundle exec unicorn -c #{unicorn_conf} - E #{rails_env} -D; fi"
end
end
but it doesn't work. I mean that command executes (it asks me the password and gives no errors), but all changes in config files are still ignored (i.e. number of worker processes or database settings).
Maybe this is because of the way unicorn restarts. Not every worker is restarted immediately. This is to make it possible to have zero downtime and loose no requests. If you want to see your changes for sure, try to stop and then start your application instead. I have had to do this some times. Of course you will potentially loose some request.
The following tasks is what I use for restarting, stopping, and starting my unicorn server.
desc "Zero-downtime restart of Unicorn"
task :restart, :except => { :no_release => true } do
run "kill -s USR2 `cat #{shared_path}/pids/unicorn.pid`"
end
desc "Start unicorn"
task :start, :except => { :no_release => true } do
run "cd #{current_path} ; bundle exec unicorn_rails -c config/unicorn.rb -D -E production"
end
desc "Stop unicorn"
task :stop, :except => { :no_release => true } do
run "kill -s QUIT `cat #{shared_path}/pids/unicorn.pid`"
end
Hope this helps you.
Maybe this article is of interest.
see here my baby~
Restarting Unicorn with USR2 doesn't seem to reload production.rb settings
Keep in mind that: your working directory in unicorn.rb should be : /your/cap/directory/current
NOT be: File.expand_path("../..", FILE)
Because the unicorn and linux soft link forking error: soft link can not work well.
You should give capistrano-unicorn a try, that's what I currently use with the default hooks mentioned below.
Setup
Add the library to your Gemfile:
ruby
group :development do
gem 'capistrano-unicorn', :require => false
end
And load it into your deployment script config/deploy.rb:
ruby
require 'capistrano-unicorn'
Add unicorn restart task hook:
ruby
after 'deploy:restart', 'unicorn:reload' # app IS NOT preloaded
after 'deploy:restart', 'unicorn:restart' # app preloaded
after 'deploy:restart', 'unicorn:duplicate' # before_fork hook implemented (zero downtime deployments)
I want to start script/delayed_job start on my production when i will start my rails server.
Is there anyway i can do that?
EDIT::
I have added this line to my config/initializers/delayed_job.rb .
Delayed::Worker.new.start
But my delayed job server is not starting when i am running my rails applicaiton.
Is there any other solution??
I would recommend deploying your app with Capistrano and defining a after:deploy hook to start/restart DJ at every deploy.
I would also recommend using Resque over DelayedJob, as the latter has tendencies to just die without any reason, and usually requires a Monit/God monitoring and restarting it.
namespace :delayed_job do
desc "Start delayed_job process"
task :start, :roles => :app do
run "cd #{current_path}; script/delayed_job start #{rails_env}"
end
desc "Stop delayed_job process"
task :stop, :roles => :app do
run "cd #{current_path}; script/delayed_job stop #{rails_env}"
end
desc "Restart delayed_job process"
task :restart, :roles => :app do
run "cd #{current_path}; script/delayed_job restart #{rails_env}"
end
end
after "deploy:start", "delayed_job:start"
after "deploy:stop", "delayed_job:stop"
after "deploy:restart", "delayed_job:restart"
You can setup an init.d file, but I would recommend either monit or god. God is ruby, so it is familiar, but that also means it leaks a bit. If you are going to run God, I recommend a cron job to restart it. This is a VERY good post on configuring monit on your server.
We went the God route, but if we had it to do over again - we would do monit.
You can do
Delayed::Worker.new.start
in your initializers directory (create a new ".rb" file in there and it'll start with your app)
I have Thinking Sphinx setup and working however I am having a problem getting the Delayed Job rake tasks to start during deployment.
I have the following task in deploy.rb which appears to execute, however the delayed jobs are not processed - they stack up until I run rake ts:dd from the server command line:
namespace :thinkingsphinx do
desc 'Start Delayed Job Sphinx delta indexing'
task :dd do
run "cd #{current_path} && rake ts:dd RAILS_ENV=#{rails_env} &"
end
end
How can I get the delayed jobs to start running from the deployment script?
Thanks
Simon
The link Florian provided has code by Amit Solanki that works!
Here is what I did to get this to work with Capistrano:
Install gems
ts-delayed-delta
daemon_generator
whenever
Create a file called script/delayed_delta with the contents:
#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
dir = File.expand_path(File.join(File.dirname(__FILE__), '..'))
daemon_options = {
:multiple => false,
:dir_mode => :normal,
:dir => File.join(dir, 'tmp', 'pids'),
:backtrace => true
}
puts 'delayed_delta'
Daemons.run_proc('job_runner', daemon_options) do
if ARGV.include?('--')
ARGV.slice! 0..ARGV.index('--')
else
ARGV.clear
end
Dir.chdir dir
RAILS_ENV = ARGV.first || ENV['RAILS_ENV'] || 'development'
require File.join('config', 'environment')
Delayed::Worker.new(
:min_priority => ENV['MIN_PRIORITY'],
:max_priority => ENV['MAX_PRIORITY']
).start
end
Configure Capistrano
Capistrano needs to start Sphinx and job_runner (with our script/delayed_delta).
Add something like this to the deploy.rb:
deploy.task :restart, :roles => :app do
run "export RAILS_ENV=production && cd #{deploy_to}/current && /usr/bin/rake ts:rebuild"
run "export RAILS_ENV=production && cd #{current_path} && /usr/bin/ruby script/delayed_delta start"
end
Configure whenever gem
In your config/schedule.rb add lines to update Sphinx's index and start job_runner if it isn't already running
every 30.minutes do
command "export RAILS_ENV=production && cd /path/to/rails/production && /usr/bin/rake ts:index && /usr/bin/ruby script/delayed_delta start"
end
This gets converted to a crontab that is run every 30 minutes to update sphinx
Final Notes and Lessons Learned
The script/delayed_delta uses the daemon_generator gem to start the job_runner background worker script. This is equivalent to running rake thinking_sphinx:delayed_deltas on the console, but persistent.
Make sure only one job_runner or rake thinking_sphinx:delayed_deltas process is running at one time
Let Capistrano start both Sphinx (rake ts:rebuild) and script/delayed_delta. I had problem when I started sphinx and delayed_deltas from different users or different environments
This link will be useful for anybody who need to restart the delayed_delta rake task from deploy script :
http://amitsolanki.com/2010/04/running-delayed-delta-daemon-in-background-for-thinking-sphinx/#comment-5802
I would put your delayed_job task in a separate script and run it from cron or have it started/monitored by your monitoring tool of choice (e.g., monit). Your deploy script can just kill it to make sure it restarts each time (killall job_runner). Here's the script I use:
#!/usr/bin/env ruby
## this script is for making sure and delayed_jobs get run
## it is used by thinking sphinx
require File.dirname(__FILE__) + '/../config/environment'
# you can also put the definition of this in config/environments/*.rb so it's different for test, production and development
JobRunnerPidFile = "#{RAILS_ROOT}/tmp/pids/job_runner.pid"
if File.exists?(JobRunnerPidFile)
old_pid = File.read(JobRunnerPidFile).to_i
begin
if Process.getpgid(old_pid) > 0
# still running, let's exit silently...
exit(0)
end
rescue
# looks like nothing is running, so let's carry on
end
end
File.open(JobRunnerPidFile, "w") {|f| f.write "#{$$}\n" }
Delayed::Worker.new.start