I have an existing resque.rake file that lets me run heroku rake jobs:work to spawn a resque worker. Here is the file:
require 'resque/tasks'
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
end
desc "Alias for resque:work (To run workers on Heroku)"
task "jobs:work" => "resque:work"
Now I want to add a specified number of workers by adding the COUNT=X option, but I am unsure where and how to add it to my rake file. How should I do this?
I would thing you could just add another ENV[] key/value:
task "resque:setup" => :environment do
ENV['QUEUE'] = '*'
ENV['COUNT'] = 5
end
Related
The rescue-scheduler gem installation says to create a separate Rake task for scheduling:
https://github.com/resque/resque-scheduler#rake-integration
namespace :resque do
task :setup_schedule => :setup do
...
task :scheduler => :setup_schedule
$ rake resque:scheduler
However my Heroku Procfile already has a worker process to run Resque:
resque: env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work COUNT=1
I don't want to create yet another worker process just to run scheduled jobs. This old blog posts says you can set it up without using Rake tasks.
https://www.perfectline.co/blog/2011/07/cron-tasks-for-your-rails-application-with-resque-2/
How do you schedule Resque jobs on Heroku without use 2 worker dynos?
I tried to add multiple dependencies but it gave me a circular dependency error.
namespace :resque do
task :setup => [:environment, :scheduler]
Console
>env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work COUNT=1
rake aborted!
Circular dependency detected: TOP => resque:work => resque:preload => resque:setup => resque:scheduler => resque:setup_schedule => resque:setup
Tasks: TOP => resque:work => resque:preload => resque:setup => resque:scheduler => resque:setup_schedule
This works
https://grosser.it/2012/04/14/resque-scheduler-on-heroku-without-extra-workers/
require 'resque/tasks'
require 'resque/scheduler/tasks'
namespace :resque do
desc "schedule and work, so we only need 1 dyno"
task :schedule_and_work do
if Process.respond_to? :fork
if Process.fork
sh "rake environment resque:work"
else
sh "rake resque:scheduler"
Process.wait
end
else # windows
pid = Process.spawn "rake environment resque:work"
Rake::Task["resque:scheduler"].invoke
Process.wait pid
end
end
task :setup => :environment
task :setup_schedule => :setup do
require 'resque-scheduler'
# The schedule doesn't need to be stored in a YAML, it just needs to
# be a hash. YAML is usually the easiest.
# Resque.schedule = YAML.load_file('your_resque_schedule.yml')
Resque.schedule = {merit_rules: {every: '1m', class: 'Merit::ReputationChangeObserver', queue: :badge_queue,
description: 'This job runs Merit\'s rules to award badges.'}}
end
task :scheduler => :setup_schedule
end
And to launch it
env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:schedule_and_work COUNT=1
I have rake tasks which installs and starts neo4j.
rake neo4j:install[community-latest, stable]
rake neo4j:start[stable] where `stable` is environment.
Now I want to write another rake task something like rake setup and create dependency on rake neo4j:start[stable] and rake neo4j:install[community-latest, stable]
I have tried,
task :setup_dev_env => [:neo4j:install[community-latest, stable], :neo4j:start[stable]] do
puts "Created Rake task"
end
obviously this doesn't work, because in the above task neo4j is namespace. Then I have changed my task to something like,
task :setup_dev_env => [:'neo4j:install[community-latest, stable]', :'neo4j:start[stable]'] do
puts "Hello rake task working"
end
so, at least this solved my issue with neo4j namespace, but still couldn't solve the problem.
When I run rake setup_dev_env It says
rake aborted!
Don't know how to build task 'neo4j:install[community-latest, stable]' (see --tasks)
You can modify that task as:
desc 'Some description'
# setup_dev_env is dependent on neo4j:start
task :setup_dev_env,[:stable] => :environment do |t, arg|
param = arg[:stable].nil? ? 'stable' : arg[:stable]
Rake::Task['neo4j:start'].invoke(param)
puts "Created Rake task"
end
You can write your task as follow:
desc 'Some description'
# setup_dev_env is dependent on neo4j:start
task :setup_dev_env,[:stable] => "neo4j:start" do
puts "Created Rake task"
end
The above lines of code accept a parameter stable and pass it to neo4j:start. To use stable parameter in neo4j:start, you must have to receive as like:
desc 'Some description'
# it will be inside namespace neo4j
task :start, [:stable] do |t, args|
puts "Created Rake task #{args.inspect}"
end
I have a problem when I do:
namespace :xaaron do
task :get_roles do
roles = Xaaron::Role.all
puts roles
end
task :get_role, [:name] do |t, args|
role = Xaaron::Role.find(args[:name].parameterize)
puts role
end
end
The first task will work fine. I can even add binding.pry and run Xaaron::Role and get information about Roles back. But the second task fails with:
NameError: uninitialized constant Xaaron::Role
I run each task in my main app because these tasks are inside an engine, using:
bin/rake xaaron:get_roles` and `bin/rake xaaron:get_role
I can run bin/rails c in the main application that uses the engine and run Xaaron::Role and get information about Roles table.
Why is the second one failing but the first one is not? Is there scoping with arguments?
I'm not sure why either works, but if this is Rails and those are Rails models, your tasks should depend on the environment:
task :get_roles => [ :environment ] do
By depending on the :environment task, it first loads Rails.
Also see: What's the 'environment' task in Rake?.
You can also run a Rake task as
bundle exec rake environment xaaron:get_role
This will load the Rails environment first.
I kept getting uninitialized constant errors for a Rake task, even after depending on :environment and running with bundle exec.
The issue was that I was making a Rake::TestTask and, even though the Rake task had access to all constants, the test files themselves did not have access to constants.
The solution was to add this line to the top of my test file:
require_relative '../config/environment'
This is the Rake task:
require "rake/testtask"
Rake::TestTask.new(:test) do |t|
t.libs << "test"
t.libs << "lib"
t.test_files = FileList["test/**/test_*.rb"]
end
To add, as of Ruby 1.9 and above, you can use this hash syntax:
namespace :xaaron do
desc "Rake task to get roles"
task get_roles: :environment do
roles = Xaaron::Role.all
puts roles
end
#####
end
And then you can run the command below to run the Rake task:
rake xaaron:get_roles
or
bundle exec rake xaaron:get_roles
I'am using Capistrano for deployment and Sidekiq for queues. I need to load user ids to set :sidekiq_queues variable in deploy.rb file to perform sidekiq queues. Now I use following code
set :sidekiq_queues, ::User.all.map {|u| "-q parsing_user_#{u.id}"}.join(" ") + " -q parsing_user_0"
but it throws following error
./config/deploy.rb:29:in `load': uninitialized constant User (NameError)
I tried to require 'rubygems' and 'active_record' into deploy.rb, but it didn't help.
In result I should have
sidekiq_queues == "-q parsing_user_1 -q parsing_user_2 -q parsing_user_3 -q parsing_user_4 -q parsing_user_5 -q parsing_user_0".
Hardcoding queue names isn't a solution.
Capistrano executes deploy.rb locally, and it doesn't load your Rails environment in deploy.rb.
It might be more trouble than it is worth. Especially if you want to execute this on your remote server, you might consider doing a Rake task instead. The => :environment in the rake task ensures that your Rails environment is loaded.
# in deploy.rb
namespace :sidekiq do
desc "Do something with queues"
task :queues, :roles => :web do
run "cd #{current_path} ; RAILS_ENV=#{rails_env} bundle exec rake sidekiq:queues"
end
end
# you'll need to decide when to execute this in your deployment process,
# something like this:
after "deploy:update_code", "sidekiq:queues"
# in lib/tasks/sidekiq.rake
namespace :sidekiq do
desc "Do something with queues"
task :queues => :environment do
queues = (User.scoped.pluck(:id) + [0]).map{|id| "-q parsing_user_#{id}"}.join(" ")
# do what you need to do
end
end
The resque jobs I have do not depend on anything in Rails, but I'm having a hard time starting workers without the rails env. I've seen this post, but it didn't help (ruby resque without loading rails environment)
Here is my current rake file:
require "resque/tasks"
task "resque:setup" do
root_path = "#{File.dirname(__FILE__)}/../.."
require "#{root_path}/app/workers/myworker.rb"
end
#task "resque:setup" => :environment
The commented task would load the Rails env and everything works, but that's not what I want. When running rake resque:work I get this error:
rake aborted!
No such file to load -- application_controller
Tasks: TOP => resque:work => resque:preload
If you've only added a lib/tasks/resque.rake file and haven't modified your Rakefile, you'll still be loading your Rails environment when you call rake resque:work. Try this for Rakefile:
unless ENV['RESQUE_WORKER'] == 'true'
require File.expand_path('../config/application', __FILE__)
My::Application.load_tasks
else
ROOT_PATH = File.expand_path("..", __FILE__)
load File.join(ROOT_PATH, 'lib/tasks/resque.rake')
end
And then this for your resque.rake file:
require "resque/tasks"
task "resque:setup" do
raise "Please set your RESQUE_WORKER variable to true" unless ENV['RESQUE_WORKER'] == "true"
root_path = "#{File.dirname(__FILE__)}/../.."
require "#{root_path}/app/workers/myworker.rb"
end
Then call rake resque:work RESQUE_WORKER=true
I referred the link here It worked perfectly for me:
This error was resolved, by running
$> QUEUE=* rake environment resque:work
a cleaner solution was to define a rake task:
task "resque:setup" => :environment do
ENV['QUEUE'] ||= '*'
#for redistogo on heroku http://stackoverflow.com/questions/2611747/rails-resque-workers-fail-with-pgerror-server-closed-the-connection-unexpectedl
Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }
end
and now
rake resque:work
Worked perfectly
Thanks.