How do you schedule Resque jobs on Heroku? - ruby-on-rails

The rescue-scheduler gem installation says to create a separate Rake task for scheduling:
https://github.com/resque/resque-scheduler#rake-integration
namespace :resque do
task :setup_schedule => :setup do
...
task :scheduler => :setup_schedule
$ rake resque:scheduler
However my Heroku Procfile already has a worker process to run Resque:
resque: env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work COUNT=1
I don't want to create yet another worker process just to run scheduled jobs. This old blog posts says you can set it up without using Rake tasks.
https://www.perfectline.co/blog/2011/07/cron-tasks-for-your-rails-application-with-resque-2/
How do you schedule Resque jobs on Heroku without use 2 worker dynos?
I tried to add multiple dependencies but it gave me a circular dependency error.
namespace :resque do
task :setup => [:environment, :scheduler]
Console
>env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work COUNT=1
rake aborted!
Circular dependency detected: TOP => resque:work => resque:preload => resque:setup => resque:scheduler => resque:setup_schedule => resque:setup
Tasks: TOP => resque:work => resque:preload => resque:setup => resque:scheduler => resque:setup_schedule

This works
https://grosser.it/2012/04/14/resque-scheduler-on-heroku-without-extra-workers/
require 'resque/tasks'
require 'resque/scheduler/tasks'
namespace :resque do
desc "schedule and work, so we only need 1 dyno"
task :schedule_and_work do
if Process.respond_to? :fork
if Process.fork
sh "rake environment resque:work"
else
sh "rake resque:scheduler"
Process.wait
end
else # windows
pid = Process.spawn "rake environment resque:work"
Rake::Task["resque:scheduler"].invoke
Process.wait pid
end
end
task :setup => :environment
task :setup_schedule => :setup do
require 'resque-scheduler'
# The schedule doesn't need to be stored in a YAML, it just needs to
# be a hash. YAML is usually the easiest.
# Resque.schedule = YAML.load_file('your_resque_schedule.yml')
Resque.schedule = {merit_rules: {every: '1m', class: 'Merit::ReputationChangeObserver', queue: :badge_queue,
description: 'This job runs Merit\'s rules to award badges.'}}
end
task :scheduler => :setup_schedule
end
And to launch it
env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:schedule_and_work COUNT=1

Related

Rails 2 + Delayed_job 2.0.7: Best way to automate the worker to start jobs?

I'm researching and finding it hard to find a mainstream way to start the worker for the delayed_job gem. So far, I have one solution to use Capistrano:
# add this to config/deploy.rb
namespace :delayed_job do
desc "Start delayed_job process"
task :start, :roles => :app do
run "cd #{current_path}; script/delayed_job start #{rails_env}"
end
desc "Stop delayed_job process"
task :stop, :roles => :app do
run "cd #{current_path}; script/delayed_job stop #{rails_env}"
end
desc "Restart delayed_job process"
task :restart, :roles => :app do
run "cd #{current_path}; script/delayed_job restart #{rails_env}"
end
end
after "deploy:start", "delayed_job:start"
after "deploy:stop", "delayed_job:stop"
after "deploy:restart", "delayed_job:restart"
but apparently there can be some cases where the Apache user is diferent from the Rails app user which will cause issues? Is there a better way to automate the worker process so that I don't have to sit at my comp and type
$ RAILS_ENV=production script/delayed_job start
$ RAILS_ENV=production script/delayed_job stop
all day in the console?
I created a rake task to run " rake:jobs workoff " once daily. You can set as often as you need. I used heroku scheduler.

Load ActiveRecord's data to Capistrano's deploy.rb

I'am using Capistrano for deployment and Sidekiq for queues. I need to load user ids to set :sidekiq_queues variable in deploy.rb file to perform sidekiq queues. Now I use following code
set :sidekiq_queues, ::User.all.map {|u| "-q parsing_user_#{u.id}"}.join(" ") + " -q parsing_user_0"
but it throws following error
./config/deploy.rb:29:in `load': uninitialized constant User (NameError)
I tried to require 'rubygems' and 'active_record' into deploy.rb, but it didn't help.
In result I should have
sidekiq_queues == "-q parsing_user_1 -q parsing_user_2 -q parsing_user_3 -q parsing_user_4 -q parsing_user_5 -q parsing_user_0".
Hardcoding queue names isn't a solution.
Capistrano executes deploy.rb locally, and it doesn't load your Rails environment in deploy.rb.
It might be more trouble than it is worth. Especially if you want to execute this on your remote server, you might consider doing a Rake task instead. The => :environment in the rake task ensures that your Rails environment is loaded.
# in deploy.rb
namespace :sidekiq do
desc "Do something with queues"
task :queues, :roles => :web do
run "cd #{current_path} ; RAILS_ENV=#{rails_env} bundle exec rake sidekiq:queues"
end
end
# you'll need to decide when to execute this in your deployment process,
# something like this:
after "deploy:update_code", "sidekiq:queues"
# in lib/tasks/sidekiq.rake
namespace :sidekiq do
desc "Do something with queues"
task :queues => :environment do
queues = (User.scoped.pluck(:id) + [0]).map{|id| "-q parsing_user_#{id}"}.join(" ")
# do what you need to do
end
end

Load resque worker without rails environment?

The resque jobs I have do not depend on anything in Rails, but I'm having a hard time starting workers without the rails env. I've seen this post, but it didn't help (ruby resque without loading rails environment)
Here is my current rake file:
require "resque/tasks"
task "resque:setup" do
root_path = "#{File.dirname(__FILE__)}/../.."
require "#{root_path}/app/workers/myworker.rb"
end
#task "resque:setup" => :environment
The commented task would load the Rails env and everything works, but that's not what I want. When running rake resque:work I get this error:
rake aborted!
No such file to load -- application_controller
Tasks: TOP => resque:work => resque:preload
If you've only added a lib/tasks/resque.rake file and haven't modified your Rakefile, you'll still be loading your Rails environment when you call rake resque:work. Try this for Rakefile:
unless ENV['RESQUE_WORKER'] == 'true'
require File.expand_path('../config/application', __FILE__)
My::Application.load_tasks
else
ROOT_PATH = File.expand_path("..", __FILE__)
load File.join(ROOT_PATH, 'lib/tasks/resque.rake')
end
And then this for your resque.rake file:
require "resque/tasks"
task "resque:setup" do
raise "Please set your RESQUE_WORKER variable to true" unless ENV['RESQUE_WORKER'] == "true"
root_path = "#{File.dirname(__FILE__)}/../.."
require "#{root_path}/app/workers/myworker.rb"
end
Then call rake resque:work RESQUE_WORKER=true
I referred the link here It worked perfectly for me:
This error was resolved, by running
$> QUEUE=* rake environment resque:work
a cleaner solution was to define a rake task:
task "resque:setup" => :environment do
ENV['QUEUE'] ||= '*'
#for redistogo on heroku http://stackoverflow.com/questions/2611747/rails-resque-workers-fail-with-pgerror-server-closed-the-connection-unexpectedl
Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }
end
and now
rake resque:work
Worked perfectly
Thanks.

Adding a COUNT option to my resque.rake file

I have an existing resque.rake file that lets me run heroku rake jobs:work to spawn a resque worker. Here is the file:
require 'resque/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
Now I want to add a specified number of workers by adding the COUNT=X option, but I am unsure where and how to add it to my rake file. How should I do this?
I would thing you could just add another ENV[] key/value:
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
ENV['COUNT'] = 5
end

How to just start 1 production Resque worker from Capistrano?

I've got Redis/Resque installed but I'm having a hard time starting/restarting it from Capistrano when I deploy.
My needs are fairly simple: I only have 1 queue, only need 1 worker, and I'm not using monit or god.
Based on this blog post, my deploy.rb has this:
namespace :deploy do
task :start do ; end
task :stop do ; end
task :restart, :roles => :app, :except => { :no_release => true } do
run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
end
task :restart_resque do
pid_file = "#{shared_path}/pids/resque.pid"
run "test -f #{pid_file} && cd #{current_path} && kill -s QUIT `cat #{pid_file}` || rm -f #{pid_file}"
run "cd #{current_path}; RAILS_ENV=production QUEUE='*' VERBOSE=1 nohup rake environment resque:work& > #{shared_path}/log/resque.log && echo $! > #{pid_file}"
end
end
after 'deploy:restart', 'deploy:restart_resque'
My resque.rake file (based on the Railscast) has:
require "resque/tasks"
task "resque:setup" => :environment
When I cap deploy, everything seems to work but there are no workers running when I look at resque-web ...
I suggest you look at this StackOverflow question and this gist for quite a few suggestions.

Resources