Resque workers fail immediately: undefined method `write' for nil:NilClass - ruby-on-rails

I am using Resque (and resque-scheduler) in my Rails app to run a recurring job. This was working fine for me, until today. I made some code changes, which I thought were unrelated, but now every worker fails before the perform method is even entered (checked with a debug statement). The same worker method works fine when I run it in the rails console. It only fails via resque on the development localhost (Postgres DB).
The error shown in the resque console for the failed worker is:
Exception
NoMethodError
Error
undefined method `write' for nil:NilClass
There is no additional stack trace for the error. Any idea why this is failing?
Additional info:
lib/tasks/resque.rake
# Resque tasks
require 'resque/tasks'
require 'resque_scheduler/tasks'
namespace :resque do
task :setup do
require 'resque'
require 'resque_scheduler'
require 'resque/scheduler'
# you probably already have this somewhere
Resque.redis = 'localhost:6379'
# If you want to be able to dynamically change the schedule,
# uncomment this line. A dynamic schedule can be updated via the
# Resque::Scheduler.set_schedule (and remove_schedule) methods.
# When dynamic is set to true, the scheduler process looks for
# schedule changes and applies them on the fly.
# Note: This feature is only available in >=2.0.0.
#Resque::Scheduler.dynamic = true
# The schedule doesn't need to be stored in a YAML, it just needs to
# be a hash. YAML is usually the easiest.
Resque.schedule = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
# If your schedule already has +queue+ set for each job, you don't
# need to require your jobs. This can be an advantage since it's
# less code that resque-scheduler needs to know about. But in a small
# project, it's usually easier to just include you job classes here.
# So, something like this:
# require 'jobs'
end
end
task "resque:setup" => :environment do
#ENV['QUEUE'] = '*'
Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }
end
config/resque.yml
development: localhost:6379
test: localhost:6379:1
staging: redis1.se.github.com:6379
fi: localhost:6379
production: redis1.ae.github.com:6379
initializers/resque.rb
rails_root = Rails.root || File.dirname(__FILE__) + '/../..'
rails_env = Rails.env || 'development'
resque_config = YAML.load_file(rails_root.to_s + '/config/resque.yml')
Resque.redis = resque_config[rails_env]
# This will make the tabs show up.
require 'resque_scheduler'
require 'resque_scheduler/server'
config/resque_schedule.yml
populate_game_data:
# you can use rufus-scheduler "every" syntax in place of cron if you prefer
every: 1m
# By default the job name (hash key) will be taken as worker class name.
# If you want to have a different job name and class name, provide the 'class' option
class: PopulateDataWorker
queue: high
args:
description: "This job populates the game and data"
Should note that the above files were not changed between working and non-working state.

We had the same issue this morning, and we pinned it down to a gem update by New Relic.
Version 3.5.6.46 of newrelic_rpm was yanked on rubygems, but it was somehow installed by bundle update.
They are still on the beta track for 3.5.6 and had some issues with Resque. See https://github.com/newrelic/rpm/commit/e81889c2bce97574ec682dafee12015e13ccb2e1
The fix was to add '~> 3.5.5.38' in our Gemfile for newrelic_rpm

Related

View Resque log output in Rails server logs

I've got a Rails 4 app on a Puma server with Resque/Resque-Scheduler running background jobs. What I'd like to know is how I merge the log output of my two Resque workers into my server log, or, of that is not possible, how I can view the log output of my Resque workers. Currently I have not been able to figure out how to view the log output for the workers, so I have no idea what's happening under the hood. I found this blogpost, which suggests adding the following likes to my resque.rake file:
task "resque:setup" => :environment do
Resque.before_fork = Proc.new {
ActiveRecord::Base.establish_connection
# Open the new separate log file
logfile = File.open(File.join(Rails.root, 'log', 'resque.log'), 'a')
# Activate file synchronization
logfile.sync = true
# Create a new buffered logger
Resque.logger = ActiveSupport::BufferedLogger.new(logfile)
Resque.logger.level = Logger::INFO
Resque.logger.info "Resque Logger Initialized!"
}
end
That didn't work. I also tried the suggestion in the comments, which was to replace Resque.logger = ActiveSupport::BufferedLogger.new(logfile) with Resque.logger = ActiveSupport::Logger.new(logfile), however that didn't work either. With the second option, I still get a NoMethodError: undefined method 'logger=' for Resque:Module error when I try to boot up a worker.
Here is my current resque.rake file:
require 'resque/tasks'
require 'resque_scheduler/tasks'
namespace :resque do
puts "Loading Rails environment for Resque"
task :setup => :environment do
require 'resque'
require 'resque_scheduler'
require 'resque/scheduler'
require 'postman'
end
end
I've looked at the Resque docs on logging, but am not sure how to use what's there as I admittedly don't know very much about logging in Rails. I haven't had any luck finding other useful resources on the subject.
How I fix it, it is not perfect but just works.
my environment: rails 5.0.1, resque: 1.26.0
at the first time, I set the Resque.logger and Resque.logger.level in config/initializers/resque.rb as most docs suggest:
# config/initializers/resque.rb
Resque.logger = Logger.new("#{Rails.root}/log/resque.log")
Resque.logger.level = Logger::DEBUG
then in the job, I output log by Resque.logger.info:
# update_xxx_job.rb
class UpdateXxxJob
def self.perform
Resque.logger.info 'Job starts'
...
end
end
it doesn't work, I can see nothing in log/resque.log.
then someone said should set the logfile sync always, no buffer, so I update the config/initializers/resque.rb according a question from stackoverflow:
# config/initializers/resque.rb
logfile = File.open(File.join(Rails.root, 'log', 'resque.log'), 'a')
logfile.sync = true
Resque.logger = ActiveSupport::Logger.new(logfile)
Resque.logger.level = Logger::INFO
still doesn't work.
I also tried config resque logger in lib/tasks/resque.rake:
# lib/tasks/resque.rake
require 'resque'
require 'resque/tasks'
require 'resque/scheduler/tasks'
require 'resque-scheduler'
require 'resque/scheduler/server'
namespace :resque do
task setup: :environment do
Resque.schedule = YAML.load_file(Rails.root + "config/resque_scheduler_#{Rails.env}.yml")
Resque.redis.namespace = "xxx_#{Rails.env}"
Resque.logger = Logger.new("#{Rails.root}/log/resque.log")
Resque.logger.level = Logger::INFO
end
end
doesn't work.
finally, I decide to move the logger configuration from initializer to the job, so the job now looks like:
# update_xxx_job.rb
class UpdateXxxJob
def self.perform
Resque.logger = Logger.new("#{Rails.root}/log/resque.log")
Resque.logger.level = Logger::DEBUG
Resque.logger.info 'Job starts'
...
end
end
then I can get what I want in the log/resque.log.
you can try it.
I've had the same problem while setting up mine. Here's what I did:
Resque.before_fork do
# Your code here
end
It seems before_fork accepts a block as an argument rather than assigning a block to it.
I faced the same issue so that I check the source of resque and finally I needed to do the followings at initialization process:
define log formatter.
then define logger with log-file path.
set any log level.
Here the example is at my config/initializers/resque.rb in rails case:
...
Resque.logger = Logger.new("#{Rails.root}/log/resque.log")
Resque.logger.level = Logger::DEBUG
Resque.logger.formatter = ::Logger::Formatter.new # This is important
Resque default logger formatter is set here and its definitions is here. That apparently just ignores the output...

how setup active_job and rescue scheduler?

I'm trying to create background jobs for email notification and scraper.
I use resque-scheduler (4.0.0), resque (1.25.2) and rails 4.2.1.
My config.ru file:
# This file is used by Rack-based servers to start the application.
require ::File.expand_path('../config/environment', __FILE__)
run Rails.application
require 'resque/server'
run Rack::URLMap.new "/" => AppName::Application, "/resque" => Resque::Server.new
My /lib/tasks/resque.rake:
require 'resque/tasks'
require 'resque/scheduler/tasks'
namespace :resque do
task :setup do
require 'resque'
require 'resque-scheduler'
Resque.schedule = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }
end
end
My /config/resque_scheduler.yml:
CheckFsUpdatesJob:
queue: fs_updates
every:
- '1h'
- :first_in: '10s'
class: CheckFsUpdatesJob
args:
description: scrape page
My /config/initializer/active_job.rb
ActiveJob::Base.queue_adapter = :resque
My /config/initializer/resque.rb:
#config/initializers/resque.rb
require 'resque-scheduler'
require 'resque/scheduler/server'
uri = URI.parse("redis://localhost:6379/")
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }
Resque.schedule = YAML.load_file(Rails.root.join('config', 'resque_schedule.yml'))
Resque::Server.use(Rack::Auth::Basic) do |user, password|
user = 'admin'
password = 'admin'
end
My first job for emails notifications:
class EmailNotificationJob < ActiveJob::Base
queue_as :email_notifications
def perform(episode_id, email)
NotificationMailer.new_record_appears(record_id, email).deliver_now
end
end
My second job for scheduled runs:
class CheckFsUpdatesJob < ActiveJob::Base
queue_as :fs_updates
def perform()
FsStrategy.new.check_for_updates
end
end
So I have to jobs:
1. emails notifications - should sends email when new record in DB appears
2. scrape a page - should runs every hour
How I run it:
redis-server
rake environment resque:work QUEUE=fs_updates
rake environment resque:work QUEUE=email_notifications
rake environment resque:scheduler
rails s
After running these commands I see in Resque Dashboard two workers and two queues, as it is expected.
But!
After clicking on 'queue now' button at resque Schedule tab, I see that task was created and wroted to "fs_updates" queue. But it's not running and in a few second it dissapears.
When I run a job for emails sending from rails console - it does not work at all.
Please, help me to fix my configurations.
Thanks kindly!
As I understood: rails and active_job developers is not responsible for resque plugins. Maybe this problem will be fixed in new gem versions, but now it does not work (active_job does not work fine with resque-scheduler).
Currently I use gem 'active_scheduler' to fix current problem.
I had the same issue trying to configure Sucker Punch on rails 4.2.1 In the end I moved my custom initialiser logic into application.rb, not great but it got me up and running.
Not sure if there is an issue with the initialisers in this release. Try moving your code from active_job.rb and resque.rb into application.rb or the appropriate environment file. Obviously this isn't a long term solution but it will at least help you you identify whether it's an initialiser issue or Resque config problem.

Resque scheduled jobs are stuck in pending queue

I'm creating a DripEmail campaign for my app and using resque scheduler to schedule the tasks.
I've sceduled a static job, which runs every day at a specific time and collect the user's list based on the drip settings and sends out an emailer to them.
This is my resque job user_follow_up.rb
class UserFollowUp
#queue = :user_follow_up
def self.perform
User.each do |u|
# Send the emailers to only those who are not converted
if !user.is_converted and Date.today <= user.next_email
stage(u)
end
end
end
end
This is my scheduler.yml
UserFollowUp:
cron: "0 16 * * *"
I have 2 resque workers, one has my default set of tasks and the other for scheduler.
rake environment resque:work QUEUE=publish_story,accept_story,image_queue,Mango_mailer
and
rake environment resque:scheduler QUEUE=user_follow_up
When I open the resque admin interface, I'm able to see my static job detected in the list. I clicked the Queue Now button to test it. It properly enqueues the task to the queue, but doesn't execute. It keeps these tasks in the pending queue forever.
This is my resque.rake, it's required
require 'resque/tasks'
require 'resque/scheduler/tasks'
require 'resque/scheduler/server'
require 'active_record'
require 'mongoid'
require 'action_controller/railtie'
require 'active_support/buffered_logger'
# load the Rails app all the time
namespace :resque do
puts 'Loading Rails environment for Resque'
task :setup => :environment do
# The schedule doesn't need to be stored in a YAML, it just needs to
# be a hash. YAML is usually the easiest.
Resque.schedule = YAML.load_file("#{Rails.root}/config/scheduler.yml")
Resque::Scheduler.dynamic = true
Resque.logger.info 'Resque Scheduler Initialized!'
Resque.before_first_fork do
# Open the new separate log file
logfile = File.open(File.join(Rails.root, 'log', 'resque.log'), 'a')
# Activate file synchronization
logfile.sync = true
# Create a new buffered logger
Resque.logger = ActiveSupport::BufferedLogger.new(logfile)
Resque.logger.level = Logger::INFO
Resque.logger.info 'Resque Logger Initialized!'
puts 'Resque Logger Initialized!'
end
end
task 'resque:pool:setup' do
Resque::Pool.after_prefork do |job|
Resque.redis.client.reconnect
end
end
end
And this is my Gemfile
gem 'resque', github: 'resque/resque' , branch: '1-x-stable'
gem 'resque_mailer', github: 'zapnap/resque_mailer'
gem 'resque-scheduler'
I'm not sure what is the issue here. Please help me with this fix.

Schedule a one-time Resque job on heroku on application start in Rails

I am using Resque and Resque Schedule to start a job that has to be run immediately on the application start. Other scheduled jobs are loaded every 30 seconds.
This is the code for my config/initializers/redis.rb
require 'rake'
require 'resque'
require 'resque/server'
require 'resque_scheduler/tasks'
# This will make the tabs show up.
require 'resque_scheduler'
require 'resque_scheduler/server'
uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.redis = REDIS
Dir["#{Rails.root}/app/workers/*.rb"].each { |file| require file }
Resque.enqueue(AllMessageRetriever)
Resque.schedule = YAML.load_file(Rails.root.join('config', 'schedule.yml'))
When the application is started up, the AllMessageRetriever gets run 2-3 times rather than only once. Do the initializers get called more than once? This happens both on Heroku and my local environment?
Is it possible to set a delayed job in Resque-Scheduler which will only get executed once and immediately on runtime?
The code for AllMessageRetriever. Basically it loops over a table and calls an external API to get data and then updates it to the table. This entire task happens 2-3 times if I add the enqueue method in initializer file
require 'socialcast'
module AllMessageRetriever
#queue = :message_queue
def self.perform()
Watchedgroup.all.each do |group|
puts "Running group #{group.name}"
continueLoading=true
page=1
per_page=500
while(continueLoading == true)
User.first.refresh_token_if_expired
token = User.first.token
puts "ContinueLoading: #{continueLoading}"
#test = Socialcast.get_all_messages(group.name,token,page,per_page)
messagesArray = ActiveSupport::JSON.decode(#test)["messages"]
puts "Message Count: #{messagesArray.count}"
if messagesArray.count == 0
puts 'count is zero now'
continueLoading = false
else
messagesArray.each do |message|
if not Message.exists?(message["id"])
Message.create_with_socialcast(message, group.id)
else
Message.update_with_socialcast(message)
end
end
end
page += 1
end
Resqueaudit.create({:watchedgroup_id => group.id,:timecompleted => DateTime.now})
end
# Do anything here, like access models, etc
puts "Doing my job"
end
end
Rake
Firstly, why are you trying to queue on init?
You'd be much better delegating to a rake task which is called from an initializer.
This will remove dependency on the initialize process, which should clear things up a lot. I wouldn't put this in an initializer itself, as it will be better handled elsewhere (modularity)
Problem
I think this line is causing the issue:
Resque.enqueue(AllMessageRetriever)
Without seeing the contents of AllMessageRetriever, I'd surmise that you're AllMessageRetriever (module / class?) will be returning the results 2/3 times, causing Resque to add the (2 / 3 times) data-set to the queue
Could be wrong, but it would make sense, and mean your issue is not with Resque / Initializers, but your AllMessageRetriever class
Would be a big help if you showed it!

testing using Resque with Rspec examples?

I am processing my background jobs using Resque.
My model looks like this
class SomeClass
...
repo = Repo.find(params[:repo_id])
Resque.enqueue(ReopCleaner, repo.id)
...
end
class RepoCleaner
#queue = :repo_cleaner
def self.perform(repo_id)
puts "this must get printed in console"
repo = Repo.find(repo_id)
# some more action here
end
end
Now to test in synchronously i have added
Resque.inline = Rails.env.test?
in my config/initializers/resque.rb file
This was supposed to call #perform method inline without queuing it into Redis and without any Resque callbacks as Rails.env.test? returns true in test environment.
But
"this must get printed in console"
is never printed while testing. and my tests are also failing.
Is there any configurations that i have missed.
Currently i am using
resque (1.17.1)
resque_spec (0.7.0)
resque_unit (0.4.0)
I personally test my workers different. I use RSpec and for example in my user model I test something like this:
it "enqueue FooWorker#create_user" do
mock(Resque).enqueue(FooWorker, :create_user, user.id)
user.create_on_foo
end
Then I have a file called spec/workers/foo_worker_spec.rb with following content:
require 'spec_helper'
describe FooWorker do
describe "#perform" do
it "redirects to passed action" do
...
FooWorker.perform
...
end
end
end
Then your model/controller tests run faster and you don't have the dependency between model/controller and your worker in your tests. You also don't have to mock so much things in specs which don't have to do with the worker.
But if you wan't to do it like you mentioned, it worked for me some times ago. I put Resque.inline = true into my test environment config.
It looks like the question about logging never got answered. I ran into something similar to this and it was from not setting up the Resque logger. You can do something as simple as:
Resque.logger = Rails.logger
Or you can setup a separate log file by adding this to your /lib/tasks/resque.rake. When you run your worker it will write to /log/resque.log
Resque.before_fork = Proc.new {
ActiveRecord::Base.establish_connection
# Open the new separate log file
logfile = File.open(File.join(Rails.root, 'log', 'resque.log'), 'a')
# Activate file synchronization
logfile.sync = true
# Create a new buffered logger
Resque.logger = ActiveSupport::Logger.new(logfile)
Resque.logger.level = Logger::INFO
Resque.logger.info "Resque Logger Initialized!"
}
Mocking like daniel-spangenberg mentioned above ought to write to STDOUT unless your methods are in the "private" section of your class. That's tripped me up a couple times when writing rspec tests. ActionMailer requires it's own log setup too. I guess I've been expecting more convention than configuration. :)

Resources