Once upon a time, I had one app - Cashyy. It used sidekiq. I deployed it and used this upstart script to manage sidekiq (start/restart).
I decide to deploy another app to the same server. The app (let's call it Giimee) also uses sidekiq.
And here is the issue. Sometimes I need to restart sidekiq for Cashyy, but not for Giimee. Now, as I understand I will need to hack something using index thing (in upstart script and managing sidekiqs: sudo restart sidekiq index=1) (if I understood it correctly).
BUT!
I have zero desire to dabble with these indexes (nightmare to support? Like you need to know how many apps are using sidekiq and be sure to assign unique index to each sidekiq. And to know assigned index if you want to restart specific sidekiq).
So here is the question: how can I isolate each sidekiq (so I would not need to maintain the index) and still get the stability and usability of upstart (starting the process, restarting, etc.)?
Or maybe I don't understand something and the thing with index is state of art?
You create two services:
cp sidekiq.conf /etc/init/cashyy.conf
cp sidekiq.conf /etc/init/glimee.conf
Edit each as necessary. sudo start cashyy, sudo stop glimee, etc. Now you'll have two completely separate Sidekiq processes running.
As an alternative to an upstart script, you can use Capistrano and Capistrano-Sidekiq to manage those Sidekiqs.
We have Sidekiq running on 3 machines and have had a good experience with these two libraries/tools.
Note: we currently use an older version of Capistrano (2.15.5)
In our architecture, the three machines are customized slightly on deploy. This led us to break up our capistrano deploy scripts by machine so that we could customize some classes, manage Sidekiq, etc. Our capistrano files are structured something like this:
- config/
- deploy.rb
- deploy/
- gandalf.rb
- gollum.rb
- legolas.rb
With capistrano-sidekiq, we are able to control, well, Sidekiq :) at any time (during a deploy or otherwise). We set up the Sidekiq aspects of our deploy scripts in the following way:
# config/deploy.rb
# global sidekiq settings
set :sidekiq_default_hooks, false
set :sidekiq_cmd, "#{fetch(:bundle_cmd, 'bundle')} exec sidekiq"
set :sidekiqctl_cmd, "#{fetch(:bundle_cmd, 'bundle')} exec sidekiqctl"
set :sidekiq_role, :app
set :sidekiq_pid, "#{current_path}/tmp/pids/sidekiq.pid"
set :sidekiq_env, fetch(:rack_env, fetch(:rails_env, fetch(:default_stage)))
set :sidekiq_log, File.join(shared_path, 'log', 'sidekiq.log')
# config/deploy/gandalf.rb
# Custom Sidekiq settings
set :sidekiq_timeout, 30
set :sidekiq_processes, 1
namespace :sidekiq do
# .. code omitted from methods and tasks for brevity
def for_each_process(&block)
end
desc 'Quiet sidekiq (stop accepting new work)'
task :quiet, :roles => lambda { fetch(:sidekiq_role) }, :on_no_matching_servers => :continue do
end
desc 'Stop sidekiq'
task :stop, :roles => lambda { fetch(:sidekiq_role) }, :on_no_matching_servers => :continue do
end
desc 'Start sidekiq'
task :start, :roles => lambda { fetch(:sidekiq_role) }, :on_no_matching_servers => :continue do
end
desc 'Restart sidekiq'
task :restart, :roles => lambda { fetch(:sidekiq_role) }, :on_no_matching_servers => :continue do
end
end
When I need to restart one of my Sidekiq instances, I can just go to my terminal and execute the following:
$ bundle exec cap gandalf sidekiq:restart
$ bundle exec cap gollum sidekiq:stop
It's made Sidekiq management quite painless for our team and thought it would be worth sharing in the event something similar could help you out.
Related
I am using capistrano for our RAILS deployment. We want to deploy to one server first, and after deployment is finished on the first server, then we want to start deployment on second server. We do not want restarting in sequence with delay. We want to have complete deployment one at a time. So far I have this:
namespace :deploy do
task :sequence do
on roles(:app), in: :sequence do |host|
invoke 'deploy'
end
end
end
The problem is with invoke 'deploy'
It calls deploy for all the app servers which in turns deploy in parallel.
Finally How Do I invoke deploy task for a specific host?
Following should help you to run the deploy task in sequential mode:
task :my_task, roles: :web do
find_servers_for_task(current_task).each do |server|
run "YOUR_COMMAND", hosts: server.host
end
end
If I had that requirement, I'd probably script it. You can run Capistrano with the --hosts parameter to define which of the servers you described in your stage file (config/deploy/dev|stage|prod|somethingelse.rb) you actually want to run the command against. This can take two forms. Let's say I have three servers, test1, test2, and prod1. I can run it with a list, like cap prod --hosts=test1,test2 deploy and only test1 and test2 will receive the deployment. You can also use a regular expression to achieve the same thing, like cap prod --hosts=^test deploy.
This is documented here: http://capistranorb.com/documentation/advanced-features/host-filtering/
With this in mind, I'd probably write a script (or Makefile) which runs capistrano N time, for a different server each time.
I have build some capistrano tasks which I need to run on within the defined :app roles. This is what I have so far:
desc "Stop unicorn"
task :stop, :roles => :app do
logger.info "Stopping unicorn server(s).."
run "touch #{unicorn_pid}"
pid = capture("cat #{unicorn_pid}").to_i
run "kill -s QUIT #{pid}" if pid > 0
end
As far as I know, this should run the given commands on the servers given in the :app role, right? But the fact of the matter is that it's running the commands on the servers in the :db role.
Can anyone give some insight into this problem? Or is there a way to force Capistrano to adhere to the :roles flag?
Thanks in advance
// Emil
Using Capture will cause the task to be run only on the first server listed.
From the documentation:
The capture helper will execute the given command on the first matching server, and will return the output of the command as a string.
https://github.com/capistrano/capistrano/wiki/2.x-DSL-Action-Inspection-Capture
Unfortunately I am facing a similar issue, the find_servers solution may work, but it's hacky, and runs N x N times, where N in the number of servers you have.
Kinda new to Rails, so please cope with me. What i'm doing now is background processing some Ruby code use Resque. To get the Rescque rake task started, I've been using (on heroku), I have a resque.rake file with that recommended code to attach into heroku's magical(or strange) threading architecture:
require "resque/tasks"
require 'resque_scheduler/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
Since I need access to the Rails code, I reference :environment. If I set at least 1 worker dyno in the background on heroku, my Resque does great, gets cleared, everything is happy. Until i try to automate stuff...
So I wanted to evolve the code and automatically fill the queue with relevant tasks every minute or so. Do that (without using cron, because heroku is not adequate with cron), I declare an initializer named task_scheduler.rb that uses Rufus scheduler to run tasks:
scheduler = Rufus::Scheduler.start_new
scheduler.in '5s' do
autoprocessor_method
end
scheduler.every '1m' do
autoprocessor_method
end
Things appear to work awesome for a while....then the rake process just stops picking up from the queue unexplainably. The queue just gets larger and larger. Even if i have multiple worker dynos running, they all eventually get tired and stop processing the queue. I'm not sure what I am doing wrong, but I suspect the referencing of the Rails environment in my rake task is causing the task_scheduler.rb code to run again, causing duplicate scheduling. I'm wondering how to solve that problem if someone knows, and I'm also curious if that is the reason for the rake task to stop working.
Thank you
You should not be booting the scheduler in an initializer, you should have a daemon process running the scheduler and filling up your queue. It would be something like this ("script/scheduler"):
#!/usr/bin/env ruby
root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
Dir.chdir(root)
require 'rubygems'
gem 'daemons'
require 'daemons'
options = {
:dir_mode => :normal,
:dir => File.join(root, 'log'),
:log_output => true,
:backtrace => true,
:multiple => false
}
Daemons.run_proc("scheduler", options) do
Dir.chdir(root)
require(File.join(root, 'config', 'environment'))
scheduler = Rufus::Scheduler.start_new
scheduler.in '5s' do
autoprocessor_method
end
scheduler.every '1m' do
autoprocessor_method
end
end
And you can call this script as a usual daemon from your app:
script/scheduler start
This is going to make sure you have only one process sending work for the resque workers instead of one for each mongrel that you're running.
First of all, if you are not running on Heroku, i would not recommend this approach. I'd look at Mauricio's answer, or consider using a classic cron job or using Whenever to schedule the cron job.
But if you are in the pain of running on heroku and trying to do this, here is how i got this to work.
I kept the same original Resque.rake code in place, as i pasted in the original question. In addition, i created another rake task that i attached to the jobs:work rake process, just like the first case:
desc "Scheduler processor"
task :scheduler => :environment do
autoprocess_method
scheduler = Rufus::Scheduler.start_new
scheduler.every '1m' do
twitter_autoprocess
end
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "scheduler"
Couple of notes:
This will be imperfect once you use more than one worker dyno because the scheduler will run in more than one spot. you can solve that by saving state somewhere, but its not as clean as I would like.
I found the original reason why the process would hang. It was this line of code:
scheduler.in '5s' do
autoprocessor_method
end
I'm not sure why, but when I removed that, it never hung again.
For the life of me I can't figure out how to make this work properly.
The problem is similar to what others have, such as: How to do a rolling restart of a cluster of mongrels
We, however, are using Nginx/Passenger instead of Mongrel.
The issue is that on a deploy if we use this standard :restart task:
task :restart, :roles => [:app], :except => {:no_release => true} do
run "cd #{deploy_to}/current && touch tmp/restart.txt"
end
It touches the restart.txt file across every web server, but any passenger instances currently serving requests need to finish before the new ones are spawned it seems. This creates a serious delay and causes our app to be unavailable for up to 2 minutes while everything is coming back up.
In order to get around that the plan is to do the following:
deploy code
go to server 1, remove it from the load balancer
restart nginx-passenger on server 1
wait 60 seconds
add server 1 back to load balancer
go to server 2 (repeat steps 3 - 5)
To accomplish this, I attempted this:
(lb.txt is the file that the load balancer looks for)
task :restart, :roles => [:app], :except => {:no_release => true} do
servers = find_servers_for_task(current_task)
servers.map do |s|
run "cd #{deploy_to}/current && echo '' > public/lb.txt", :host => s.host
run %Q{rvmsudo /etc/init.d/nginx-passenger restart > /dev/null}, :host => s.host
sleep 60
run "cd #{deploy_to}/current && echo 'ok' > public/lb.txt", :host => s.host
end
end
This almost works, however, during the deploy it seemed to run the loop through the servers once per servers listed in the :app role. We currently have 6 app servers, so the loop runs 6 times, restarting nginx-passenger 6 times per server.
I just need this loop to run through one time.
I know it seems that eventually passenger will get rolling restarts, but they do not seem to exist yet.
If it helps, we are using Capistrano 2.x and Rails 3
Any help would be great.
Thanks.
run "cd #{deploy_to}/current && echo 'ok' > public/lb.txt", :host => s.host
should actually be:
run "cd #{deploy_to}/current && echo 'ok' > public/lb.txt", :hosts => s.host
I ran across this gem capify-ec2 which has a rolling restart feature. capify-ec2 on github.
I am about to install it and try it out.
Here's the description from the readme that describes what their rolling restart feature does:
"This feature allows you to deploy your code to instances one at a time, rather than simultaneously. This becomes useful for more complex applications that may take longer to startup after a deployment. Capistrano will perform a full deploy (including any custom hooks) against a single instance, optionally perform a HTTP healthcheck against the instance, then proceed to the next instance if deployment was successful.
After deployment a status report is displayed, indicating on which instances deployment succeeded, failed or did not begin. With some failures, further action may need to be taken manually; for example if an instance is removed from an ELB (see the 'Usage with Elastic Load Balancers' section below) and the deployment fails, the instance will not be reregistered with the ELB, for safety reasons."
I've inherited an existing Rails 2 application and am currently trying to deploy it on production servers.
As a rails/unix novice, what's the best way to find out what webserver the rails application is running on and how can I restart the server. (since from what I've read, rails will cache everything on production servers)
The previous developer used Capistrano, but unfortunately I don't have access to the GIT repository.
I noticed /configuration/deploy.rb has the following lines:
desc "Custom restart task for mongrel cluster"
task :restart, :roles => :app, :except => { :no_release => true } do
deploy.mongrel.restart
end
desc "Custom start task for mongrel cluster"
task :start, :roles => :app do
deploy.mongrel.start
end
desc "Custom stop task for mongrel cluster"
task :stop, :roles => :app do
deploy.mongrel.stop
end
Does this imply mongrel_rails is being used?
If so what's the best way to restart the application to pick up my changes?
Many thanks.
Does this imply mongrel_rails is being
used?
Yes.
If so what's the best way to restart
the application to pick up my changes?
It depends on which Application server you currently use. Assuming the current recipe is ok, simply call the Capistrano restart task.
$ cap deploy:restart