I am using resque, and resque-scheduler in my rails application. I am facing strange problem in using resque-scheduler. One of my job is not getting removed from queue, once it finishes with the 'perform' method. I need to kill it explicitly to get out of the queue, then other jobs in the queue starts executing.
Job class is simple, like:
class FooJob
#queue = :high_volume
def self.perform
puts "FooJob#perform:"
# some method call
end
end
And resque_schedule.yml contains:
add_jobs_from_foo:
cron: "15 * * * *"
class: FooJob
description: "enqueue jobs from Foo"
Can it be problem with gem versions? or any other?
Did you actually verify by looking at redis directly? The job gets removed from the queue before perform even starts execution.
lets try this gem "resque-status"
get job status-
status = Resque::Plugins::Status::Hash.get(job_id)
Resque::Plugins::Status::Hash.get(job_id) object gonna returns:
status.pct_complete #=> 0
status.status #=> 'queued'
status.queued? #=> true
status.working? #=> false
status.time #=> Time object
status.message #=> "Created at ..."
get this gem and for details: https://github.com/quirkey/resque-status
Related
I am trying to call a job in the console but always get errors.
Followed the following documentation:
http://tutorials.jumpstartlab.com/topics/performance/background_jobs.html
https://www.youtube.com/watch?v=UrE47tCrZXc
https://guides.rubyonrails.org/active_job_basics.html
https://github.com/resque/resque
How can I run an ActiveJob in Rails console for debugging?
Created the following file:
(lib/jobs/archive_survey_jobs.rb)
module Jobs
class ArchiveSurveysJob < ActiveJob::Base
#queue = :setup
def perform(survey_type)
if SurveyTypes.all_classes.map(&:to_s).include?(survey_type)
surveys = Survey.where(something: 'stuff')\
.where.not(something: 'stuff',
something: 'stuff')
surveys.each do |survey|
do_something
end
end
end
end
end
I understand that I can do something like Resque.enqueue(ArchiveSurveysJob, 'string_here')
How can I call this in the console? If I try:
Jobs::ArchiveSurveysJob.create(survey_type: 'string_here'),
when I check the Resque Statuses it resulted in an error: `The task failed because of an error:
undefined local variable or method `args' for #
If I try this:
Jobs::ArchiveSurveysJob.perform(`string_here`)
Or:
Jobs::ArchiveSurveysJob.perform_now('string_here')
I get:
ArgumentError: wrong number of arguments (given 0, expected 1..2)
Please let me know if I am missing some documentation or if I am doing something wrong.
In your example, perform is an instance method. youre calling the class level perform which might not have any arguments. I dont know about your version of resque specifically, however you can enqueue jobs from the console like this in at least one version: Resque.enqueue(Jobs::ArchiveSurveysJob)
ps, make sure your resque workers are running.
In dev environments I typically have resque running in line:
# config/initializers/resque.rb
Resque.inline = true if Rails.env.dev? || Rails.env.test?
How I managed to do this was by creating a Rake task and calling the Job there.
I have a Rails application that runs some background jobs via ActiveJob and Sidekiq. The sidekiq logs in both the terminal and the log file show the following:
2016-10-18T06:17:01.911Z 3252 TID-oukzs4q3k ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-97318b38b1391672d21feb93 INFO: start
Is there some way to show the class names of the jobs here similar to how logs work for a regular Sidekiq Worker?
Update:
Here is how a Sidekiq worker logs:
2016-10-18T11:05:39.690Z 13678 TID-or4o9w2o4 ClientJob JID-b3c71c9c63fe0c6d29fd2f21 INFO: start
Update 2:
My sidekiq version is 3.4.2
I'd like to replace ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper with Client Job
So I managed to do this by removing Sidekiq::Middleware::Server::Logging from the middleware configuration and adding a modified class that displays the arguments in the logs. The arguments themself contain the job and action names as well.
For latest version, currently 4.2.3, in sidekiq.rb
require 'sidekiq'
require 'sidekiq/middleware/server/logging'
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def log_context(worker, item)
klass = item['wrapped'.freeze] || worker.class.to_s
"#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}"
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.remove Sidekiq::Middleware::Server::Logging
chain.add ParamsLogging
end
end
For version 3.4.2, or similar, override the call method instead:
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def call(worker, item, queue)
klass = item['wrapped'.freeze] || worker.class.to_s
Sidekiq::Logging.with_context("#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}") do
begin
start = Time.now
logger.info { "start" }
yield
logger.info { "done: #{elapsed(start)} sec" }
rescue Exception
logger.info { "fail: #{elapsed(start)} sec" }
raise
end
end
end
end
You must be running some ancient version. Upgrade.
Sorry, looks like that's a Rails 5+ feature only. You'll need to upgrade Rails. https://github.com/rails/rails/commit/8d2b1406bc201d8705e931b6f043441930f2e8ac
What is the best way to test something that requires background jobs with Cucumber? I need to run DelayedJob and Sneakers workers in background while tests are running.
You can run any application in the background:
#pid = Process.spawn "C:/Apps/whatever.exe"
Process.detach(#pid)
And even kill it after tests are done:
Process.kill('KILL', #pid) unless #pid.nil?
You can create your own step definition in features/step_definitions/whatever_steps.rb (hopefully with a better name)
When /^I wait for background jobs to complete$/ do
Delayed::Worker.new.work_off
end
That can be extended for any other scripts you'd like to run with that step. Then in the test, it goes something like:
Then I should see the text "..."
When I wait for background jobs to complete
And I refresh the page
Then I should see the text "..."
If anyone has similar problem I ended up writing this (thanks to Square blog post):
require "timeout"
class CucumberExternalWorker
attr_accessor :worker_pid, :start_command
def initialize(start_command)
raise ArgumentError, "start_command was expected" if start_command.nil?
self.start_command = start_command
end
def start
puts "Trying to start #{start_command}..."
self.worker_pid = fork do
start_child
end
at_exit do
stop_child
end
end
private
def start_child
exec({ "RAILS_ENV" => Rails.env }, start_command)
end
def stop_child
puts "Trying to stop #{start_command}, pid: #{worker_pid}"
# send TERM and wait for exit
Process.kill("TERM", worker_pid)
begin
Timeout.timeout(10) do
Process.waitpid(worker_pid)
puts "Process #{start_command} stopped successfully"
end
rescue Timeout::Error
# Kill process if could not exit in 10 seconds
puts "Sending KILL signal to #{start_command}, pid: #{worker_pid}"
Process.kill("KILL", worker_pid)
end
end
end
This can be called as following (added it to env.rb for cucumber):
# start delayed job
$delayed_job_worker = CucumberExternalWorker.new("rake jobs:work")
$delayed_job_worker.start
I am using Resque and Resque Schedule to start a job that has to be run immediately on the application start. Other scheduled jobs are loaded every 30 seconds.
This is the code for my config/initializers/redis.rb
require 'rake'
require 'resque'
require 'resque/server'
require 'resque_scheduler/tasks'
# This will make the tabs show up.
require 'resque_scheduler'
require 'resque_scheduler/server'
uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.redis = REDIS
Dir["#{Rails.root}/app/workers/*.rb"].each { |file| require file }
Resque.enqueue(AllMessageRetriever)
Resque.schedule = YAML.load_file(Rails.root.join('config', 'schedule.yml'))
When the application is started up, the AllMessageRetriever gets run 2-3 times rather than only once. Do the initializers get called more than once? This happens both on Heroku and my local environment?
Is it possible to set a delayed job in Resque-Scheduler which will only get executed once and immediately on runtime?
The code for AllMessageRetriever. Basically it loops over a table and calls an external API to get data and then updates it to the table. This entire task happens 2-3 times if I add the enqueue method in initializer file
require 'socialcast'
module AllMessageRetriever
#queue = :message_queue
def self.perform()
Watchedgroup.all.each do |group|
puts "Running group #{group.name}"
continueLoading=true
page=1
per_page=500
while(continueLoading == true)
User.first.refresh_token_if_expired
token = User.first.token
puts "ContinueLoading: #{continueLoading}"
#test = Socialcast.get_all_messages(group.name,token,page,per_page)
messagesArray = ActiveSupport::JSON.decode(#test)["messages"]
puts "Message Count: #{messagesArray.count}"
if messagesArray.count == 0
puts 'count is zero now'
continueLoading = false
else
messagesArray.each do |message|
if not Message.exists?(message["id"])
Message.create_with_socialcast(message, group.id)
else
Message.update_with_socialcast(message)
end
end
end
page += 1
end
Resqueaudit.create({:watchedgroup_id => group.id,:timecompleted => DateTime.now})
end
# Do anything here, like access models, etc
puts "Doing my job"
end
end
Rake
Firstly, why are you trying to queue on init?
You'd be much better delegating to a rake task which is called from an initializer.
This will remove dependency on the initialize process, which should clear things up a lot. I wouldn't put this in an initializer itself, as it will be better handled elsewhere (modularity)
Problem
I think this line is causing the issue:
Resque.enqueue(AllMessageRetriever)
Without seeing the contents of AllMessageRetriever, I'd surmise that you're AllMessageRetriever (module / class?) will be returning the results 2/3 times, causing Resque to add the (2 / 3 times) data-set to the queue
Could be wrong, but it would make sense, and mean your issue is not with Resque / Initializers, but your AllMessageRetriever class
Would be a big help if you showed it!
I have two questions:
How can I add a heroku worker just before running a delayed job and remove it after it finishes?
Is my cron.rake ok?
cron.rake:
desc "This task is called by the Heroku cron add-on"
task :cron => :environment do
puts "requesting homepage to refresh cache"
uri = URI.parse('http://something.com')
Net::HTTP.get(uri)
puts "end requesting homepage"
puts "start sending daily mail"
User.notified_today.each do |user|
Delayed::Job.enqueue UserMailer.daily_mail(user).deliver
end
puts "end sending daily mail"
end
I use collectiveidea delayed_job.
I've had good success with HireFire.
Easy setup:
Add gem 'hirefire' to your Gemfile
Create Rails.root/config/initializers/hirefire.rb with the config information.
To add remove/remove workers, hook into your ORM's after :create / after :destroy
With DataMapper on Heroku, I did it like this (You must set the ENV vars yourself)
MAX_CONCURRENT_WORKERS = 5
if ENV["HEROKU_APP"]
Delayed::Job.after :create do
workers_needed = [Delayed::Job.count, MAX_CONCURRENT_WORKERS].min
client = Heroku::Client.new(ENV['HEROKU_USERNAME'], ENV['HEROKU_PASSWORD'])
client.set_workers(ENV['HEROKU_APP'], workers_needed)
puts "- Initialized Heroku workers for ZipDecoder"
end
Delayed::Job.after :destroy do
workers_needed = [Delayed::Job.count, MAX_CONCURRENT_WORKERS].min
client = Heroku::Client.new(ENV['HEROKU_USERNAME'], ENV['HEROKU_PASSWORD'])
client.set_workers(ENV['HEROKU_APP'], workers_needed)
puts "- Cleaned Up a Delayed Job for ZipDecoder ---------------------------------"
end
end
You maybe can use an "autoscale" plugin like workless or heroku-autoscale.
About the cron I don't see any problem on it...