I have a rails application with a dynamically configured time zone. It is stored in a database table containing other options, and the rails application itself is configured to UTC (default).
I've made the application itself aware of the timezone with a simple around filter using Time.use_zone(..., &block).
I would like to do something similar for my Sidekiq workers. Some of them process data that has timezone relevance, so they need it. I don't see any filtering options available in Sidekiq itself, no callbacks, before/after type things I can hook into. My current solution is to a prepend a module, like so:
module TimeZoneAwareWorker
def perform(*args)
Time.use_zone(Options.time_zone) do
super
end
end
end
and mixed in:
class MyWorker
include Sidekiq::Worker
prepend TimeZoneAwareWorker
...
end
This works fine for simple workers, but breaks down if the prepend occurs in the same class as the include Sidekiq::Worker. If the worker is subclassed, the hierarchy doesn't work out for the prepended perform to wrap the implementation.
Is there a better way? Ultimately it seems what I really want is a foolproof method of wrapping a single method with another method, and yielding the wrapped implementation.
I know my other option is monkeypatching before/after/around type callbacks into Sidekiq's implementation, but I'd like to only go there if forced.
Sidekiq has its own middleware solution:
Sidekiq has a similar notion of middleware to Rack: these are small
bits of code that can implement functionality. Sidekiq breaks
middleware into client-side and server-side.
Client-side middleware runs before the pushing of the job to Redis and allows you to modify/stop the job before it gets pushed. Client
middleware may receive the class argument as a Class object or a
String containing the name of the class.
Server-side middleware runs 'around' job processing. Sidekiq's retry feature is implemented as a simple middleware.
You can easily create your own middleware agent to add the timezone awareness code.
Related
I'm fairly new to using RSpec, so there's a lot I still don't know. I'm currently working on speccing out a section of functionality which is supposed to run a script when a button is pressed. The script is currently called in a controller, which I don't know if there's a good way to test.
I'm currently using
expect_any_instance_of(ConfigurationsController)
.to receive(:system)
.with('sh bin/resque/kill_resque_workers')
.and_return(true)
in a feature spec and it works, but rubocop is complaining about using expect_any_instance_of and I've been told to only use that method if there was no better way.
Is there any better way to test this? Like is there a way to get the instance of the controller being used, or a better kind of test for this?
A better pattern would be to not inline the system call in your controller in the first place. Instead create a seperate object that knows how to kill your worker processes and call that from your controller. The service object pattern is often used for this. It makes it much easier to stub/spy/mock the dependency and make sure it stops at your application boundry.
It also lets you test the object in isolation. Testing plain old ruby objects is really easy. Testing controllers is not.
module WorkerHandler
def self.kill_all
system 'sh bin/resque/kill_resque_workers'
end
end
# in your test
expect(WorkerHandler).to receive(:kill_all)
If your service object method runs on instances of a class you can use stub_const to stub out the new method so that it returns mocks/spies.
Another more novel solution is dependency injection via Rack middleware. You just write a piece of middleware that injects your object into env. env is the state variable thats passed all the way down the middleware stack to your application. This is how Warden for example works. You can pass env along in your spec when you make the http calls to your controller or use before { session.env('foo.bar', baz) }.
It turns out that Spring caches my wisper listener method (I'm writing quite simple Engine).
Example:
app/models/myengine/my_class.rb
class Myengine::MyClass
include Wisper::Publisher
def something
# some logic
publish(:after_something, self)
end
end
config/initializers/wisper.rb
Wisper.subscribe(Myengine::MyObserver.new)
app/observers/myengine/my_observer.rb
class Myengine::MyObserver
def after_something my_class_instance
# any changes here requires Spring manual restart in order to be reflected in tests
another_method
end
def another_method
# all changes here or in any other class methods works ok with Spring and are instantly visible in tests
return true
end
end
By Spring restart I mean manual execution of spring stop command which is really annoying.
What is more mysterious I may change another_method return value to false and then tests are failing which is OK, but when I change after_something method body to let say return false it doesn't have any effect on tests (like the body of the after_something is somehow cached).
It is not high priority problem because this strange behaviour is only visible inside listener method body and easy to overcome by moving all logic to another method in the class. Anyway it might be confusing (especially at the beginning when I didn't know the exact problem).
The problem is properly caused because when you subscribe a listener globally, even if its class is reloaded, the object remains in memory pointing to the class it was originally constructed from, even if the class has been reloaded in the meantime.
Try this in config/initializers/wisper.rb:
Rails.application.config.to_prepare do
Wisper.clear if Rails.env.development?
Wisper.subscribe(Myengine::MyObserver.new)
end
to_prepare will run the block before every request for development environment, but once, as normal for production environment. Therefore provided your listener does not maintain any state it should work as expected.
The Wisper.clear is needed to remove the existing listeners subscribed before we re-subscribe a new instance from the reloaded class. Be aware that #clear will clear all subscribers, so if you have similar code as the above in more than one engine only the last engine to be loaded will have its listeners subscribed.
I have a piece of code that performs the same queries over and over, and it's doing that in a background worker within a thread.
I checkout out the activerecord query cache middleware but apparently it needs to be enabled before use. However I'm not sure if it's a safe thing to do and if it will affect other running threads.
you can see the tests here: https://github.com/rails/rails/blob/3e36db4406beea32772b1db1e9a16cc1e8aea14c/activerecord/test/cases/query_cache_test.rb#L19
my question is: can I borrow and/or use the middleware directly to enable query cache for the duration of a block safely in a thread?
when I tried ActiveRecord::Base.cache do my CI started failing left and right...
EDIT: Rails 5 and later: the ActiveRecord query cache is automatically enabled even for background jobs like Sidekiq (see: https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#activerecord-query-cache for information on how to disable it).
Rails 4.x and earlier:
The difficulty with applying ActiveRecord::QueryCache to your Sidekiq workers is that, aside from the implementation details of it being a middleware, it's meant to be built during the request and destroyed at the end of it. Since background jobs don't have a request, you need to be careful about when you clear the cache. A reasonable approach would be to cache only during the perform method, though.
So, to implement that, you'll probably need to write your own piece of Sidekiq middleware, based on ActiveRecord::QueryCache but following Sidekiq's middleware guide. E.g.,
class SidekiqQueryCacheMiddleware
def call(worker, job, queue)
connection = ActiveRecord::Base.connection
enabled = connection.query_cache_enabled
connection_id = ActiveRecord::Base.connection_id
connection.enable_query_cache!
yield
ensure
ActiveRecord::Base.connection_id = connection_id
ActiveRecord::Base.connection.clear_query_cache
ActiveRecord::Base.connection.disable_query_cache! unless enabled
end
end
I have a question about delayed_job in Rails that doesn't seem to be mentioned much.
When you run a delayed job, it doesn't seem to load anything from ApplicationController. We have some code in ApplicationController to use a custom logger:
def setup_logger
logfile = File.open("#{RAILS_ROOT}/log/audit.log", 'a')
#audit_log = Logger.new(logfile)
$audit_log = #audit_log
end
We then reference $audit_log all through our code. But because DelayedJob doesn't load the ApplicationController this variable is nil and we get errors.
So Delayed_job is just running the specified method blindly, which could also be dangerous
if you rely on before_filters for checking data or validating things.
How can we fix our problem of getting DelayedJob to know about our global logging variable? We don't want to explicitly define the logger all through our code.
How else are people dealing with this problem, as it seems like it should be common, but its not talked about much.
Thanks
Why you think, a Job must run ApplicationController ?
A Job, precisely the worker running that job, loads the environment, sure, but not a controller...
If you don't want have it in a initializer, why not using a kind of config object for storing such data and reference to it in the job ?
And instead of do it in each job individually, I would suggest set up the logger in the BaseJob, and use it in the inherited jobs for what you need.
And another suggestion, please if you can, don't use delayed job, please use http://mperham.github.io/sidekiq/ or at least https://github.com/resque/resque
In my rails application, I have a background process runner, model name Worker, that checks for new tasks to run every 10 seconds. This check generates two SQL queries each time - one to look for new jobs, one to delete old completed ones.
The problem with this - the main log file gets spammed for each of those queries.
Can I direct the SQL queries spawned by the Worker model into a separate log file, or at least silence them? Overwriting Worker.logger does not work - it redirects only the messages that explicitly call logger.debug("something").
The simplest and most idiomatic solution
logger.silence do
do_something
end
See Logger#silence
Queries are logged at Adapter level as I demonstrated here.
How do I get the last SQL query performed by ActiveRecord in Ruby on Rails?
You can't change the behavior unless tweaking the Adapter behavior with some really really horrible hacks.
class Worker < ActiveRecord::Base
def run
old_level, self.class.logger.level = self.class.logger.level, Logger::WARN
run_outstanding_jobs
remove_obsolete_jobs
ensure
self.class.logger.level = old_level
end
end
This is a fairly familiar idiom. I've seen it many times, in different situations. Of course, if you didn't know that ActiveRecord::Base.logger can be changed like that, it would have been hard to guess.
One caveat of this solution: this changes the logger level for all of ActiveRecord, ActionController, ActionView, ActionMailer and ActiveResource. This is because there is a single Logger instance shared by all modules.