Starting multiple DelayedJob workers w/ specific queues via Capistrano tasks - ruby-on-rails

Question

I'm looking into using queues with delayed_job. I've found this page which outlines various ways of starting workers, however I'd like to keep my currently Capistrano method:
set :delayed_job_args, "-n 2 -p ecv2.production"
after "deploy:start", "delayed_job:start"
...
I was wondering how I could modify the delayed_job_args to handle spawning 1 worker with a specific queue, and 1 worker for every other job. So far, all I have is overriding each task like so:
namespace :delayed_job do
task :restart, :roles => :app do
run "cd #{current_path}; RAILS_ENV=#{rails_env} script/delayed_job -p ecv2.production --queue=export restart"
run "cd #{current_path}; RAILS_ENV=#{rails_env} script/delayed_job -p ecv2.production restart"
end
end
... But that's no fun. Any suggestions?

Answer 1

I split my jobs into two queues with one worker isolated to each queue with this setup in my deploy.rb file:
namespace :delayed_job do
task :start, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one start"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two start"
end
task :stop, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one stop"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two stop"
end
task :restart, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_one --queue=one restart"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i queue_two --queue=two restart"
end
end
The -i name part of the command is very important. That's the part that allows multiple delayed_job instances to run.
If you want to add workers to specific queues, then you would expand them out like this (where I have two workers exclusively on queue one, and one worker exclusively on queue two):
namespace :delayed_job do
task :start, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 --queue=one start"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 --queue=one start"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two --queue=two start"
end
task :stop, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 stop"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 stop"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two stop"
end
task :restart, roles: :app do
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one1 --queue=one restart"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i one2 --queue=one restart"
run "cd #{current_path}; #{rails_env} bundle exec script/delayed_job -i two --queue=two restart"
end
end

Answer 2

After a bit of messing around, the trick I found was to revert to the 'set :delayed_job_args' and use --queues= (plural) instead of --queue= (singular). Hope this helps anyone else who runs into the same issue.
set :delayed_job_args, "-n 2 -p ecv2.production --queues=cache,export"
UPDATE: What I'm using now...
after "deploy:stop", "delayed_job:stop"
after "deploy:start", "delayed_job:start"
after "deploy:restart", "delayed_job:restart"
namespace :delayed_job do
# See 'man nice' for details, default priority is 10 and 15 is a bit lower
task :start, :roles => :app do
run "cd #{current_path}; #{rails_env} nice -n 15 ruby script/delayed_job -n 1 -p yourapp.#{application} start"
end
task :restart, :roles => :app do
stop
start
end
end