in attempt to deploy rails app to server I faced problem that 'thin' does't stars when I try do star it with cap production deploy:start. What is realy strange, than it hasn't any errors.
After this I try do it on deplyment server
env RAILS_ENV=production bundle exec thin start -C config/thin.yml
Starting server on /home/deployer/app/current/tmp/sockets/thin.0.sock ...
Starting server on /home/deployer/app/current/tmp/sockets/thin.1.sock ...
ls /home/deployer/app/current/tmp/sockets/
ps -aux | grep thin
root 16769 0.0 0.1 15468 908 pts/0 S 11:34 0:00 grep --color=auto thin
thin.yml
chdir: /home/deployer/app/current
environment: production
timeout: 30
log: /home/deployer/app/current/log/thin.log
pid: /home/deployer/app/current/tmp/pids/thin.pid
socket: /home/deployer/app/current/tmp/sockets/thin.sock
max_conns: 1024
max_persistent_conns: 10
require: []
wait: 30
servers: 2
daemonize: true
What is gone wrong?
In production.log only migrations
bundle exec thin start -C config/thin.yml &
returns
Starting server on /home/deployer/app/current/tmp/sockets/thin.0.sock ...
Starting server on /home/deployer/app/current/tmp/sockets/thin.1.sock ...
'bundle exec thin start -C confi…' has ended
Answer
Okey, answer was log/thin.0.log there are some errors in code
You need to demonize thin for running it in production by adding &. Try this:
RAILS_ENV=production bundle exec thin start -C config/thin.yml &
Related
I'm struggling with starting sidekiq remotely with a custom v2 capistrano task:
namespace :sidekiq do
desc "Start sidekiq"
task :start do
run "cd #{current_path} && bundle exec sidekiq --version"
run "cd #{current_path} && bundle exec sidekiq --environment production --daemon --config config/sidekiq.yml && echo OK"
end
end
Output:
* 2018-01-05 11:40:51 executing `sidekiq:start'
* executing "cd /home/deploy/applications/xxx/current && bundle exec sidekiq --version"
servers: ["198.58.110.211"]
[198.58.110.211] executing command
** [out :: 198.58.110.211] Sidekiq 5.0.5
** [out :: 198.58.110.211]
command finished in 1424ms
* executing "cd /home/deploy/applications/xxx/current && bundle exec sidekiq --environment production --daemon --config config/sidekiq.yml && echo OK"
servers: ["198.58.110.211"]
[198.58.110.211] executing command
** [out :: 198.58.110.211] OK
command finished in 1128ms
I can confirm I'm getting the environment (rbenv & bundler correctly) as the first run cmd shows. But unexpectedly the sidekiq task starts and dispersal into obliviom: 1) tmp/pids/sidekiq.pid gets initialized but the process not exists and 2) logs/sidekiq.log gets created but only with the header:
# Logfile created on 2018-01-05 11:34:09 -0300 by logger.rb/56438
If I remove the --daemon switch I get the process running perfectly, but of course the capistrano deploy task never ends and when I do CTRL+C sidekiq closes.
If I just ssh into the remote and execute the command (replacing current_path obviously) it works perfectly.
I've tried almost everything I can imagine: not using a config.file, using RAILS_ENV instead of --environment, etc.
As the "&& echo OK" shows, the command is not returning an error.
Capistrano is using "/bin/bash --login -c 'cd /home/deploy/applications/microgestion/current && bundle exec sidekiq --environment production --daemon --config config/sidekiq.yml'" as far as I can tell to run the command.
Ruby v2.3.3, Capistrano 2.15.5, Sidekiq 5.0.5, Rails 4.0.12
Solved it by adding && sleep 1 at the end as explained here: http://blog.hartshorne.net/2013/03/capistrano-nohup-and-sleep.html.
desc "Start sidekiq"
task :start do
run "cd #{current_path} && bundle exec sidekiq --environment production --daemon --config config/sidekiq.yml && sleep 1"
end
Thanks #user3309314 for pointing me in the correct direction.
If you use plain Capistrano to daemonize Sidekiq, any crash will lead to downtime. Don't do this. You need to use a process monitor that will restart the Sidekiq process if it dies. Use systemd, upstart and/or Foreman as explained in the docs.
https://github.com/mperham/sidekiq/wiki/Deployment#running-your-own-process
I've setup monit to monitor my sunspot_solr process, which seems to work at first. If I restart the monit service with sudo service monit restart my sunspot process starts:
ps aux | grep sunspot
root 4086 0.0 0.0 9940 1820 ? Ss 12:41 0:00 bash ./solr start -f -s /ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/solr
root 4137 45.1 4.8 1480560 185632 ? Sl 12:41 0:09 java -server -Xss256k -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:CMSFullGCsBeforeCompaction=1 -XX:CMSTriggerPermRatio=80 -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/server/logs/solr_gc.log -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/server -Dsolr.solr.home=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr/solr -Dsolr.install.dir=/ebs/staging/shared/bundle/ruby/2.3.0/gems/sunspot_solr-2.2.4/solr -jar start.jar --module=http
ubuntu 4192 0.0 0.0 10460 936 pts/3 S+ 12:41 0:00 grep --color=auto sunspot
However, I'm also running tail -f /var/logs/monit.log and see this at the same time:
[CST Mar 3 12:42:54] error : 'sunspot_solr' process is not running
[CST Mar 3 12:42:54] info : 'sunspot_solr' trying to restart
[CST Mar 3 12:42:54] info : 'sunspot_solr' start: /usr/bin/sudo
[CST Mar 3 12:43:25] error : 'sunspot_solr' failed to start
Plus, to make sure monit can actually restart the sunspot_solr process, I run sudo kill -9 <the pid> and monit can't restart sunspot_solr:
[CST Mar 3 12:44:25] error : 'sunspot_solr' process is not running
[CST Mar 3 12:44:25] info : 'sunspot_solr' trying to restart
[CST Mar 3 12:44:25] info : 'sunspot_solr' start: /usr/bin/sudo
[CST Mar 3 12:44:55] error : 'sunspot_solr' failed to start
Obviously something is wrong with my monit-solr_sunspot.conf file, but after messing around with it for a few hours now, I'm stumped:
check process sunspot_solr with pidfile /ebs/staging/shared/pids/sunspot-solr.pid
start program = "/usr/bin/sudo -H -u root /bin/bash -l -c 'cd /ebs/staging/releases/20160226191542; bundle exec sunspot-solr start -- -p 8983 -d /ebs/staging/shared/solr/data --pid-dir=/ebs/staging/shared/pids'"
stop program = "/usr/bin/sudo -H -u root /bin/bash -l -c 'cd /ebs/staging/releases/20160226191542; bundle exec sunspot-solr stop -- -p 8983 -d /ebs/staging/shared/solr/data --pid-dir=/ebs/staging/shared/pids'"
I've adapted this monit script to suit my needs: Sample sunspot-solr.monit but am still having no luck!
UPDATE
I've gotten monit to successfully restart sunspot_solr if I kill it, however it still produces the error that it failed to restart in the monit.log file.
I think monit runs as root. You may not want to use sudo both because it prompts for a password and because monit doesn't need it.
I'm trying to get monit to restart my sidekiq service on CentOS server. After trying multiple solutions out there, I'm stumped, still failing to start the service.
My sidekiq file from monit.d:
check process sidekiq
with pidfile /var/www/App/tmp/pids/sidekiq.pid
start program = "/bin/bash -l -c 'sudo cd /var/www/App && bundle exec sidekiq --index 0 --pidfile /var/www/App/tmp/pids/sidekiq.pid --environment production --logfile /var/www/App/log/sidekiq.log --daemon'" as uid deploy and gid deploy
stop program = "/bin/bash -l -c 'cd /var/www/App && bundle exec sidekiqctl stop /var/www/App/tmp/pids/sidekiq.pid 10'" as uid deploy and gid deploy
if totalmem is greater than 512 MB for 2 cycles then restart
if 3 restarts within 5 cycles then timeout
If I run start program command manually, it starts the sidekiq fine but the monit doesn't seem to do anything. Just comes up with:
[BST Oct 6 11:51:17] error : 'sidekiq' process is not running
[BST Oct 6 11:51:17] info : 'sidekiq' trying to restart
[BST Oct 6 11:51:17] info : 'sidekiq' start: /bin/bash
[BST Oct 6 11:52:47] error : 'sidekiq' failed to start
So it is including file fine, but somehow doesn't manage to start the service from the script.
What can it be? Some permissions issue of sorts?
You need to update to the latest Monit version (5.14).
Remove your current monit installation and follow these instructions:
https://rtcamp.com/tutorials/monitoring/monit/
Hope it helps!
PS: Found the solution here: https://bitbucket.org/tildeslash/monit/issues/109/failed-to-stop-always-after-60-seconds
according to Debugging monit
I found i need set PATH.
my start program:
/bin/bash -c 'cd /home/vagrant/apps/skylark/current; PATH=/home/vagrant/.rbenv/shims:/home/vagrant/.rbenv/bin:$PATH bundle exec sidekiq -d -e production -C -P /home/vagrant/apps/skylark/shared/tmp/pids/sidekiq.pid -L /home/vagrant/apps/skylark/shared/log/sidekiq.log'
i think the issue is with your user. You need to execute using deploy user.
check process sidekiq
with pidfile /var/www/App/tmp/pids/sidekiq.pid
start program = "/bin/su - deploy -c 'sudo cd /var/www/App && bundle exec sidekiq --index 0 --pidfile /var/www/App/tmp/pids/sidekiq.pid --environment production --logfile /var/www/App/log/sidekiq.log --daemon'" as uid deploy and gid deploy
stop program = "/bin/su - deploy -c 'cd /var/www/App && bundle exec sidekiqctl stop /var/www/App/tmp/pids/sidekiq.pid 10'" as uid deploy and gid deploy
if totalmem is greater than 512 MB for 2 cycles then restart
if 3 restarts within 5 cycles then timeout
So i have delayed_job installed on a production app. It runs fine via rake jobs:work. But when i try to start the script via capistrano:
run "if [ -d #{current_path} ]; then cd #{current_path} && RAILS_ENV=#{rails_env} script/delayed_job start -n 2; fi"
It starts without errors. But if i check script/delayed_job status it tells me that no instances are running. Any suggestions?
Edit
looks like there is something running (via sudo ps aux | grep delayed):
1000 7952 0.0 0.1 112312 832 pts/0 S+ 16:17 0:00 grep delayed
Output when i run the script:
/path/to/latest/release/config/initializers/bypass_ssl_verification_for_open_uri.rb:2: warning: already initialized constant VERIFY_PEER
Check the permissions on your shared/tmp/pid folder.
Delayed Job won't run unless the user that capistrano is running has has permission to write the PID file into the folder.
This is how I start the delayed job demon using capistrano, maybe this will also work for you:
require "delayed/recipes"
%w[start stop restart].each do |command|
after "deploy:#{command}", "delayed_job:#{command}"
end
The output of ps aux | grep delayed only shows its own process, so DJ is not running on your machine. Maybe this has something to do with your if-clause. You could try removing that and see if it start correctly using the ps aux | grep commands. The output should be something like this:
username 9989 0.0 0.0 7640 892 pts/0 S+ 10:54 0:00 grep delayed
username 10048 0.0 9.4 288244 99156 ? Sl Jan22 2:16 delayed_job
I'm trying to set up Monit to manage Resque workers, but it fails to start saying /home/deployer/.rbenv/shims/bundle: line 4: exec: rbenv: not found
I've checked that it is running commands as deployer user and if I copy and paste the command directly via SSH everything works fine. Below is my Monit configuration. Thanks!
check process resque_worker_1
with pidfile CURRENT_PATH/tmp/pids/resque_worker_1.pid
start program = "/usr/bin/env HOME=/home/deployer RACK_ENV=production PATH=/home/deployer/.rbenv/shims:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c 'cd CURRENT_PATH; bundle exec rake environment resque:work RAILS_ENV=production QUEUE=high,normal,low VERBOSE=1 PIDFILE=CURRENT_PATH/tmp/pids/resque_worker_1.pid >> CURRENT_PATH/log/resque_worker_.log 2>&1'"
as uid deployer and gid admin
stop program = "/bin/sh -c 'cd CURRENT_PATH && kill -9 $(cat tmp/pids/resque_worker_1.pid) && rm -f tmp/pids/resque_worker_1.pid; exit 0;'"
as uid deployer and gid admin
if totalmem is greater than 300 MB for 10 cycles then restart # eating up memory?
group resque_workers
I'm not sure if this helps, but in my monitrc start line, I have to first su to the user I want to run under. I haven't tried to use the uid and gid flags to know if that works well, so this might be a goose-chase of an answer.
I remember having the same issue as you though... everything worked from the command line, but not when monit would do its thing.
For example, in my monitrc, I am monitoring arsendmail with the following:
# arsendmail_rails3
# daemon that watches and sends mail from the rails app
check process ar_sendmail with pidfile /var/www/rak/log/ar_sendmail.pid
start program "/bin/su - mike && /bin/bash -c 'cd /var/www/rak && ar_sendmail_rails3 -b1000 -d -e production'"
stop program "/bin/ps -ef | /bin/grep ar_sendmail_rails3 | /bin/grep -v grep | /usr/bin/awk '{ /usr/bin/print $2}' | /usr/bin/xargs /bin/kill -9"
I saw that the topic was created in 2012 but I had a similar problem and this thread is top ranked by google.
The problem is that monit launch commands with a restricted env (env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh to simulate).
To use monit with rbenv you must specify the correct path before your bundle exec command.
PATH=/home/[USER]/.rbenv/bin:/home/[USER]/.rbenv/shims:$PATH bundle exec ...
Example with unicorn:
check process unicorn_dev with pidfile /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid
group dev
start program = "/bin/bash -c 'cd /home/wizville/app/dev.wizville.fr/current && PATH=/home/wizville/.rbenv/bin:/home/wizville/.rbenv/shims:$PATH bundle exec unicorn -c config/unicorn.rb -D'" as uid "wizville"
stop program = "/bin/bash -c 'kill -s QUIT `cat /home/wizville/app/dev.wizville.fr/shared/pids/unicorn.pid`'"
depends on mysql
This worked for me.
check process app_resque_worker with pidfile <%= resque_pid%>
start program = "/usr/bin/env HOME=/home/subcaster RACK_ENV=production PATH=/home/subcaster/.rvm/rubies/ruby-2.0.0-p247/bin/ruby:/usr/local/bin:/usr/local/ruby/bin:/usr/bin:/bin:$PATH /bin/sh -l -c \'cd <%= current_path %>; bundle exec rake environment resque:work RAILS_ENV=production BACKGROUND=yes QUEUE=* PIDFILE=<%= resque_pid %>\'"
stop program = "kill -9 cat <%= resque_pid%> && rm -f <%= resque_pid%>"
if totalmem is greater than 2000 MB for 10 cycles then restart