Rake tasks and rails initializers - ruby-on-rails

Question

Kinda new to Rails, so please cope with me. What i'm doing now is background processing some Ruby code use Resque. To get the Rescque rake task started, I've been using (on heroku), I have a resque.rake file with that recommended code to attach into heroku's magical(or strange) threading architecture:
require "resque/tasks"
require 'resque_scheduler/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
Since I need access to the Rails code, I reference :environment. If I set at least 1 worker dyno in the background on heroku, my Resque does great, gets cleared, everything is happy. Until i try to automate stuff...
So I wanted to evolve the code and automatically fill the queue with relevant tasks every minute or so. Do that (without using cron, because heroku is not adequate with cron), I declare an initializer named task_scheduler.rb that uses Rufus scheduler to run tasks:
scheduler = Rufus::Scheduler.start_new
scheduler.in '5s' do
autoprocessor_method
end
scheduler.every '1m' do
autoprocessor_method
end
Things appear to work awesome for a while....then the rake process just stops picking up from the queue unexplainably. The queue just gets larger and larger. Even if i have multiple worker dynos running, they all eventually get tired and stop processing the queue. I'm not sure what I am doing wrong, but I suspect the referencing of the Rails environment in my rake task is causing the task_scheduler.rb code to run again, causing duplicate scheduling. I'm wondering how to solve that problem if someone knows, and I'm also curious if that is the reason for the rake task to stop working.
Thank you

Answer 1

You should not be booting the scheduler in an initializer, you should have a daemon process running the scheduler and filling up your queue. It would be something like this ("script/scheduler"):
#!/usr/bin/env ruby
root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
Dir.chdir(root)
require 'rubygems'
gem 'daemons'
require 'daemons'
options = {
:dir_mode => :normal,
:dir => File.join(root, 'log'),
:log_output => true,
:backtrace => true,
:multiple => false
}
Daemons.run_proc("scheduler", options) do
Dir.chdir(root)
require(File.join(root, 'config', 'environment'))
scheduler = Rufus::Scheduler.start_new
scheduler.in '5s' do
autoprocessor_method
end
scheduler.every '1m' do
autoprocessor_method
end
end
And you can call this script as a usual daemon from your app:
script/scheduler start
This is going to make sure you have only one process sending work for the resque workers instead of one for each mongrel that you're running.

Answer 2

First of all, if you are not running on Heroku, i would not recommend this approach. I'd look at Mauricio's answer, or consider using a classic cron job or using Whenever to schedule the cron job.
But if you are in the pain of running on heroku and trying to do this, here is how i got this to work.
I kept the same original Resque.rake code in place, as i pasted in the original question. In addition, i created another rake task that i attached to the jobs:work rake process, just like the first case:
desc "Scheduler processor"
task :scheduler => :environment do
autoprocess_method
scheduler = Rufus::Scheduler.start_new
scheduler.every '1m' do
twitter_autoprocess
end
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "scheduler"
Couple of notes:
This will be imperfect once you use more than one worker dyno because the scheduler will run in more than one spot. you can solve that by saving state somewhere, but its not as clean as I would like.
I found the original reason why the process would hang. It was this line of code:
scheduler.in '5s' do
autoprocessor_method
end
I'm not sure why, but when I removed that, it never hung again.