Capistrano/nginx/Unicorn: Website is randomly up and down - ruby-on-rails

I am using Capistrano to deploy my code to a DigitalOcean server. I added to the deploy.rb file the block for restarting Unicorn after every deploy, but since then I have noticed that when I got to the browser and start refreshing the website, sometimes I get refreshed the website and sometimes only a blank (white) page. It's a total random.
deploy.rb:
# config valid only for current version of Capistrano
lock "3.8.1"
set :application, "project"
set :repo_url, "git#bitbucket.org:username/project.git"
set :branch, "master"
set :tmp_dir, '/home/deployer/tmp'
set :deploy_to, "/home/deployer/apps/project"
set :keep_releases, 5
set(:executable_config_files, %w(
unicorn_init.sh
))
# files which need to be symlinked to other parts of the
# filesystem. For example nginx virtualhosts, log rotation
# init scripts etc.
set(:symlinks, [
{
source: "nginx.conf",
link: "/etc/nginx/sites-enabled/default"
},
{
source: "unicorn_init.sh",
link: "/etc/init.d/unicorn_#{fetch(:application)}"
},
{
source: "log_rotation",
link: "/etc/logrotate.d/#{fetch(:application)}"
},
{
source: "monit",
link: "/etc/monit/conf.d/#{fetch(:application)}.conf"
}
])
namespace :deploy do
desc 'Restart application'
task :restart_unicorn do
on roles(:app) do
execute '/home/deployer/apps/project/current/config/unicorn_init.sh restart'
end
end
after :publishing, :restart_unicorn
desc "Make sure local git is in sync with remote."
task :check_revision do
on roles(:web) do
unless `git rev-parse HEAD` == `git rev-parse origin/master`
puts "WARNING: HEAD is not the same as origin/master"
puts "Run `git push` to sync changes."
exit
end
end
end
before "deploy", "deploy:check_revision"
end
unicorn_init.sh:
set -e
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/deployer/apps/project/current
#PID=$APP_ROOT/tmp/pids/unicorn.pid
PID=/home/deployer/apps/project/shared/pids/unicorn.pid
#CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn/production.rb -E production"
AS_USER=deployer
set -u
OLD_PIN="$PID.oldbin"
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $OLD_PIN && kill -$1 `cat $OLD_PIN`
}
run () {
if [ "$(id -un)" = "$AS_USER" ]; then
eval $1
else
su -c "$1" - $AS_USER
fi
}
case "$1" in
start)
sig 0 && echo >&2 "Already running" && exit 0
run "$CMD"
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
#sig HUP && echo reloaded OK && exit 0
sig USR2 && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
run "$CMD"
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $OLD_PIN && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $OLD_PIN
then
echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
run "$CMD"
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
What I have change in this file - this line:
sig HUP && echo reloaded OK && exit 0
for this one:
sig USR2 && echo reloaded OK && exit 0
I am trying to look to the unicorn log file, but there is nothing relevant to this issue. production.log - same, nothing relevant. When I am looking for an nginx log file, I found on here: /var/log/nginx/error.log, but the fiel is empty (size 0).
Any advice what to do with this issue/what is wrong or where even to start?
EDIT: in the /home/deployer/apps/rentalhistory/shared/pids directory, there are these two files:
unicorn.pid.oldbin and unicorn.pid - should not be one of the removed?
Thanks!

What was the problem in my case - when was reloaded Unicorn, the new unicorn PID was generated (unicorn.pid) - that's correct, but the old one (unicorn.pid.oldbin) was still staying in the system. And this was apparently causing the random downs of the whole website.
FIX: I checked my Unicorn config file (config/production.rb) and this line was wrong - the path:
old_pid = "#{root}/shared/pids/unicorn.pid.oldbin"
The shared directory is on the same level as current, not inside of it.
After this change, the new unicorn PID is created and the old one is properly removed.

Related

can't start unicorn using init.d service – can't find bundle as sudo

I'm trying to set a rails test server, deployed using capistrano.
I know my capistrano scripts are working as it deploys to the production server using the same scripts without a problem.
During deployment, unicorn should be started, to do this
sudo service unicorn_appname start
is called.
This gives the following error:
Job for unicorn_appname.service failed because the control process exited with error code. See "systemctl status unicorn_appname.service" and "journalctl -xe" for details.
When I check sudo journalctl -u unicorn_appname is see
systemd[1]: Starting LSB: starts the unicorn web server...
su[3790]: Successful su for user by root
su[3790]: + ??? root:user
su[3790]: pam_unix(su:session): session opened for user user by (uid=0)
unicorn_appname[3787]: -su: bundle: command not found
systemd[1]: unicorn_appname.service: Control process exited, code=exited status=127
systemd[1]: Failed to start LSB: starts the unicorn web server.
systemd[1]: unicorn_appname.service: Unit entered failed state.
systemd[1]: unicorn_appname.service: Failed with result 'exit-code'.
/etc/init.d/unicorn_appname exists and when in /etc/init.d
./unicorn_appname start works
sudo ./unicorn_appname start however gives -su: bundle: command not found
however which bundle and sudo which bundle both show the same path (/home/user/.rbenv/shims/bundle)
If possible I don't want to change the scripts, as they work on the other server.
So I think there's some setting different or missing on the new server – but I have no more idea where to look.
This is the content of unicorn_appname:
#!/bin/sh
### BEGIN INIT INFO
# Provides: unicorn
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the unicorn web server
# Description: starts unicorn
### END INIT INFO
set -e
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/user/apps/appname/current
PID_DIR=$APP_ROOT/tmp/pids
PID=$PID_DIR/unicorn.pid
CMD="cd $APP_ROOT; bundle exec unicorn -D -c /home/user/apps/appname/shared/config/unicorn.rb -E production"
AS_USER=user
set -u
OLD_PIN="$PID.oldbin"
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $OLD_PIN && kill -$1 `cat $OLD_PIN`
}
workersig () {
workerpid="$APP_ROOT/tmp/pids/unicorn.$2.pid"
test -s "$workerpid" && kill -$1 `cat $workerpid`
}
run () {
if [ "$(id -un)" = "$AS_USER" ]; then
eval $1
else
su -c "$1" - $AS_USER
fi
}
case "$1" in
start)
sig 0 && echo >&2 "Already running" && exit 0
run "$CMD"
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
kill_worker)
workersig QUIT $2 && exit 0
echo >&2 "Worker not running"
;;
restart|reload)
sig USR2 && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
run "$CMD"
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $OLD_PIN && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $OLD_PIN
then
echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
run "$CMD"
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
Any more info you'll need?
EDIT:
user is the user I use to deploy. For root on this system, nothing is installed. Can that be the problem?
I think your problem (may) be:
cd $APP_ROOT; bundle exec unicorn -D -c /home/user/apps/appname/shared/config/unicorn.rb -E production"
Assuming that is the same command you're using on your test server, you'll need to change the environment to test.
I think I found the solution:
On the prod server I had a /etc/profile.d/rbenv.sh that's missing on the new server.
That's its content:
export RBENV_ROOT=/home/user/.rbenv
export PATH=$RBENV_ROOT/shims:$RBENV_ROOT/bin:$PATH

Unicorn Rails looking for separate file/directory for each command?

I'm trying to deploy with Ruby 2.2.2, Rails 4, Capistrano 3, Unicorn and Nginx. When I run cap production deploy I get an error sating that Unicon can't find or access a directory for the "restart" command, stemming (I believe) from this line in my deploy.rb:
execute "/etc/init.d/unicorn_#{fetch(:application)}/unicorn_init.sh/#{command}"
The error:
SSHKit::Runner::ExecuteError: Exception while executing as deploy#0.0.0.0: /etc/init.d/unicorn_app/restart exit status: 126
/etc/init.d/unicorn_app/restart stdout: Nothing written
/etc/init.d/unicorn_app/restart stderr: /usr/bin/env: /etc/init.d/unicorn_app/restart: Permission denied
Deploy.rb
set :application, 'app'
set :repo_url, 'https://github.com/user/app.git'
set :deploy_to, '/opt/www/app'
set :user, 'deploy'
set :linked_dirs, %w{log tmp/pids tmp/cache tmp/sockets}
set :ssh_options, {
forward_agent: true,
port: 7822
}
namespace :deploy do
%w[start stop restart].each do |command|
desc 'Manage Unicorn'
task command do
on roles(:app), in: :sequence, wait: 1 do
execute "/etc/init.d/unicorn_#{fetch(:application)}/unicorn_init.sh/#{command}"
end
end
end
after :publishing, :restart
end
set :rvm_ruby_version, '2.2.2'
set :default_env, { rvm_bin_path: '/usr/local/rvm/bin' }
SSHKit.config.command_map[:rake] = "#{fetch(:default_env)[:rvm_bin_path]}/rvm ruby-#{fetch(:rvm_ruby_version)} do bundle exec rake"
Can anyone help me out here? I'll gladly post any missing info.
EDIT: unicorn_init.sh: I know 'testapp' is referenced in this file, but app is referenced elsewhere in this post. Please know that i'm consistent across all files with the name, I just changed them for the post.
#!/bin/sh
### BEGIN INIT INFO
# Provides: unicorn
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Manage unicorn server
# Description: Start, stop, restart unicorn server for a specific application.
### END INIT INFO
set -e
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/opt/www/testapp/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
AS_USER=deploy
set -u
OLD_PIN="$PID.oldbin"
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $OLD_PIN && kill -$1 `cat $OLD_PIN`
}
run () {
if [ "$(id -un)" = "$AS_USER" ]; then
eval $1
else
su -c "$1" - $AS_USER
fi
}
case "$1" in
start)
sig 0 && echo >&2 "Already running" && exit 0
run "$CMD"
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
sig HUP && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
run "$CMD"
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $OLD_PIN && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $OLD_PIN
then
echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
run "$CMD"
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 "
exit 1
;;
esac

How to restart unicorn and nginx after deployment?

I am pretty new to these new tools - unicorn and nginx - and I'd like to ask you, how to restart them after development. So far, I have this setup:
unicorn_ini.sh
set -e
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/deployer/apps/project/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
AS_USER=deployer
set -u
OLD_PIN="$PID.oldbin"
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $OLD_PIN && kill -$1 `cat $OLD_PIN`
}
run () {
if [ "$(id -un)" = "$AS_USER" ]; then
eval $1
else
su -c "$1" - $AS_USER
fi
}
case "$1" in
start)
sig 0 && echo >&2 "Already running" && exit 0
run "$CMD"
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
sig HUP && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
run "$CMD"
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $OLD_PIN && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $OLD_PIN
then
echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
run "$CMD"
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
unicorn.rb
root = "/home/deployer/apps/project/current"
working_directory root
pid "#{root}/tmp/pids/unicorn.pid"
stderr_path "#{root}/log/unicorn.log"
stdout_path "#{root}/log/unicorn.log"
listen "/tmp/unicorn.project.sock"
worker_processes 4
timeout 30
# Force the bundler gemfile environment variable to
# reference the capistrano "current" symlink
before_exec do |_|
ENV["BUNDLE_GEMFILE"] = File.join(root, 'Gemfile')
end
deploy.rb
lock '3.2.1'
set :application, 'project'
set :scm, :git
set :repo_url, 'git#bitbucket.org:me/project.git'
set :pty, true
set :deploy_to, "/home/deployer/apps/project"
set :keep_releases, 5
namespace :deploy do
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
# Your restart mechanism here, for example:
# execute :touch, release_path.join('tmp/restart.txt')
end
end
after :publishing, :restart
after :restart, :clear_cache do
on roles(:web), in: :groups, limit: 3, wait: 10 do
# Here we can do anything such as:
# within release_path do
# execute :rake, 'cache:clear'
# end
end
end
end
This is the current setup. Will be restarted after successful deployment nginx and unicorn, so I'd see the new code I deployed? Also, will the application be bundled?
Thank you, and sorry once again - I ma doing this the first time
You will consider use something like https://github.com/tablexi/capistrano3-unicorn, it allow you to have zero down time deployment, restart your unicorns, with almost 0 setup needed.

Error with permission in Chef routine

I have the following Chef routine to generate a script to start/stop a unicorn daemon.
However I always get a Permission denied when Capistrano execute this file. What I'm doing wrong with its permissions?
# recipes/default.rb
template "/etc/init.d/unicorn_#{node['pod']['app_name']}" do
owner node['pod']['user']
group node['pod']['user']
mode 777
source "unicorn_init.erb"
variables( :app_name => node['pod']['app_name'], :user => node['pod']['user'] )
end
# templates/default/unicorn_init.rb
#!/bin/sh
set -e
# Feel free to change any of the following variables for your app:
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/<%= #user %>/apps/<%= #app_name %>/current
PID=/var/run/unicorn/unicorn_<%= #app_name %>.pid
CMD="cd $APP_ROOT; bundle exec unicorn -D -c $APP_ROOT/config/unicorn.rb -E production"
AS_USER=<%= #user %>
set -u
OLD_PIN="$PID.oldbin"
sig () {
test -s "$PID" && su -c "kill -$1 `cat $PID`" - $AS_USER
}
oldsig () {
test -s $OLD_PIN && sudo -c "kill -$1 `cat $OLD_PIN`" - $AS_USER
}
run () {
if [ "$(id -un)" = "$AS_USER" ]; then
eval $1
else
su -c "$1" - $AS_USER
fi
}
case "$1" in
start)
sig 0 && echo >&2 "Already running" && exit 0
run "$CMD"
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
sig HUP && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
run "$CMD"
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $OLD_PIN && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $OLD_PIN
then
echo >&2 "$OLD_PIN still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
run "$CMD"
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
As you can see, it doesn't set the correct mode for the file:
[2013-05-27T20:42:00+02:00] INFO: Processing template[/etc/init.d/unicorn_pod] action create (pod::default line 38)
[2013-05-27T20:42:00+02:00] INFO: template[/etc/init.d/unicorn_pod] owner changed to 1001
[2013-05-27T20:42:00+02:00] INFO: template[/etc/init.d/unicorn_pod] group changed to 110
[2013-05-27T20:42:00+02:00] INFO: template[/etc/init.d/unicorn_pod] mode changed to 1411
** [out :: localhost] -r----x--t 1 deployer deployer 1453 2013-05-27 17:48 unicorn_pod
See the docs. Mode should be octal or string:
template "/etc/init.d/unicorn_#{node['pod']['app_name']}" do
[...]
mode 0777 #or mode "777"
[...]
end

Linode Deban Linux + Nginx + unicorn_rails

I'm a new Linode/Linux user running Debian 6. I'm trying to get my Unicorn server to start at boot, but for some reason it is not, and I'm not able to track down any error message. Nginx is starting fine, and I have a multi-user RVM install. My gut feeling is that is has something to do with RVM. This is my unicorn_init.sh file in /rails/todo, and there's a symlink to it at /etc/init.d/unicorn:
# unicorn_init.sh
#!/bin/sh
set -e
TIMEOUT=${TIMEOUT-60}
APP_ROOT=/rails/todo
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="$APP_ROOT/bin/unicorn_rails -D -c $APP_ROOT/config/unicorn.rb -E production"
GEM_HOME="/usr/local/rvm/gems/ruby-1.9.2-p290#global"
action="$1"
set -u
old_id="$PID.oldbin"
cd $APP_ROOT || exit 1
export GEM_HOME=$GEM_HOME
sig () {
test -s "$PID" && kill -$1 `cat $PID`
}
oldsig () {
test -s $old_pid && kill -$1 `cat $old_pid`
}
case $action in
start)
sig 0 && echo >&2 "Already running" && exit 0
su -c "$CMD" - root
;;
stop)
sig QUIT && exit 0
echo >&2 "Not running"
;;
force-stop)
sig TERM && exit 0
echo >&2 "Not running"
;;
restart|reload)
sig HUP && echo reloaded OK && exit 0
echo >&2 "Couldn't reload, starting '$CMD' instead"
su -c "$CMD" - root
;;
upgrade)
if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
then
n=$TIMEOUT
while test -s $old_pid && test $n -ge 0
do
printf '.' && sleep 1 && n=$(( $n - 1 ))
done
echo
if test $n -lt 0 && test -s $old_pid
then
echo >&2 "$old_pid still exists after $TIMEOUT seconds"
exit 1
fi
exit 0
fi
echo >&2 "Couldn't upgrade, starting '$CMD' instead"
su -c "$CMD" - root
;;
reopen-logs)
sig USR1
;;
*)
echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
exit 1
;;
esac
I'm 99% of the way to getting my setup working—any advice would be much appreciated.
Here is the output of $ update-rc.d unicorn defaults:
update-rc.d: using dependency based boot sequencing
insserv: warning: script 'unicorn' missing LSB tags and overrides
insserv: There is a loop between service nginx and unicorn if stopped
insserv: loop involving service unicorn at depth 2
insserv: loop involving service nginx at depth 1
insserv: Stopping unicorn depends on nginx and therefore on system facility `$all' which can not be true!
insserv: exiting now without changing boot order!
update-rc.d: error: insserv rejected the script header
update-rc.d: error: insserv rejected the script header
The start of your file looks like:
# unicorn_init.sh
#!/bin/sh
The shebang line (#!/bin/sh) MUST be the very first line of the file.
I can't comment on the loop-detection messages as I haven't ever seen 'em before. It's possible something you say in /etc/init.d/nginx is expressing dependency on unicorn, but I don't see anything in the unicorn init which expresses a dependency on nginx, so the loop isn't clear.
insserv: warning: script 'unicorn' missing LSB tags and overrides
You should add LSB info to your init script http://wiki.debian.org/LSBInitScripts

Resources