My counter cache is locking the row under heavy load so I found wanelo/counter-cache gem which seems to be perfect for my problem but I can't set it up and it must be something really simple but I can't see it.
https://github.com/wanelo/counter-cache
I want it to use my already working delayed jobs and Redis.
In my config file
Counter::Cache.configure do |c|
c.default_worker_adapter = here??? DelayedJob ??
c.recalculation_delay = 5.hours
c.redis_pool = Redis.new
c.counting_data_store = Counter::Cache::Redis
end
If I don't put the line c.default_worker_adapter when executing it says
undefined method 'enqueue' for nil:NilClass
Any idea on what's going on? What should I put in the Worker Adapter? Nothing seems to work.
Thank you for your time
default_worker_adapter is the name of the class that will be handling your updates. An example is given on the github page of the gem. For example if you're using sidekiq, you would make sidekiq worker class and name it whatever you want. On the github page, this class is called CounterWorker and you can copy it exactly as its given, though you can use whatever delayed job framework you want. From then on, any counter_cache_on definitions on your models will use that class to make the counter updates.
Related
I have a question about delayed_job in Rails that doesn't seem to be mentioned much.
When you run a delayed job, it doesn't seem to load anything from ApplicationController. We have some code in ApplicationController to use a custom logger:
def setup_logger
logfile = File.open("#{RAILS_ROOT}/log/audit.log", 'a')
#audit_log = Logger.new(logfile)
$audit_log = #audit_log
end
We then reference $audit_log all through our code. But because DelayedJob doesn't load the ApplicationController this variable is nil and we get errors.
So Delayed_job is just running the specified method blindly, which could also be dangerous
if you rely on before_filters for checking data or validating things.
How can we fix our problem of getting DelayedJob to know about our global logging variable? We don't want to explicitly define the logger all through our code.
How else are people dealing with this problem, as it seems like it should be common, but its not talked about much.
Thanks
Why you think, a Job must run ApplicationController ?
A Job, precisely the worker running that job, loads the environment, sure, but not a controller...
If you don't want have it in a initializer, why not using a kind of config object for storing such data and reference to it in the job ?
And instead of do it in each job individually, I would suggest set up the logger in the BaseJob, and use it in the inherited jobs for what you need.
And another suggestion, please if you can, don't use delayed job, please use http://mperham.github.io/sidekiq/ or at least https://github.com/resque/resque
I have an app with both sidekiq and delayed job gems installed. When I trigger handle_asynchronously in active record models it appear to be handled by sidekiq while I would like to trigger delayed_job.
Is there a way to desactivate sidekiq for a specific model?
UPDATE:
Sidekiq now provides ways to either disable its delay module completely or alias it as sidekiq_delay. Please check this to see how to do it. https://github.com/mperham/sidekiq/wiki/Delayed-Extensions#disabling-extensions
For older version of sidekiq:
I use this monkey patch to make it so that calling .sidekiq_delay() goes to sidekiq and .delay() is goes to DelayedJob. According the answer by Viren, I think this may also solve your problem.
The patch is less complex (just a bunch of aliases), and gives you the power to consciously decide which delay you are actually calling.
As I mention in the comment In order to get it working you have to redefine/basically monkey patch the handle_asynchronously method something like this
Anywhere you like (but make sure it loaded )
in your config/initializers/patch.rb the code look like this
module Patch
def handle_asynchronously(method, opts = {})
aliased_method, punctuation = method.to_s.sub(/([?!=])$/, ''), $1
with_method, without_method = "#{aliased_method}_with_delay#{punctuation}", "#{aliased_method}_without_delay#{punctuation}"
define_method(with_method) do |*args|
curr_opts = opts.clone
curr_opts.each_key do |key|
if (val = curr_opts[key]).is_a?(Proc)
curr_opts[key] = if val.arity == 1
val.call(self)
else
val.call
end
end
end
## Replace this with other syntax
# delay(curr_opts).__send__(without_method, *args)
__delay__(curr_opts).__send__(without_method, *args)
end
alias_method_chain method, :delay
end
end
Module.send(:include,Patch)
And I believe rest all will follow then they way it should :)
Reason:
Delayed::Job include delay method on Object and Sidekiq include it delay method over ActiveRecord
Hence when the class try to invoke delay it look up it ancestors class (including the Eigen Class)
and it find the method define or included in ActiveRecord::Base class (which is sidekiq delay)
why does __delay__ work because alias define the copy of the existing method which is delay method of DelayedJob , hence when you invoke the __delay__ method it invoke delay method define DelayedJob
include to Object
Note:
Although the solution is bit patch but the it works . Keeping in mind that every direct .delay methid invocation is invoking delay method of the SideKiq and not DelayedJob to invoke the DelayedJob delay method you always has call it this way __delay__
Suggestion :
Monkey Patching is just a bad practice on my personal note I would rather not use 2 entirely different background processing library for a single application to achieve the same task. If the task is process thing in background why cant it be done with a single library either delayed_job or sidekiq (why it is that you required both of them )
So the point and to simply thing make your background processing an ease with respect to future I sincerely advice you take any one of the two library for background processing and I feel that would the valid answer for your question instead of monkey patching an doing other crazy stuff
Hope this help
I have a rails 3 application and looked around in the internet for daemons but didnt found the right for me..
I want a daemon which fetches data permanently (exchange courses) from a web resource and saves it to the database..
like:
while true
Model.update_attribte(:course, http::get.new("asdasd").response)
end
I've only seen cron like jobs, but they only run after a specific time... I want it permanently, depending on how long it takes to end the query...
Do you understand what i mean?
The gem light-daemon I wrote should work very well in your case.
http://rubygems.org/gems/light-daemon
You can write your code in a class which has a perform method, use a queue system like this and at application startup enqueue the job with Resque.enqueue(Updater).
Obviously the job won't end until the application is stopped, personally I don't like that, but if this is the requirement.
For this reason if you need to execute other tasks you should configure more than one worker process and optionally more than one queue.
If you can edit your requirements and find a trigger for the update mechanism the same approach still works, you only have to remove the while true loop
Sample class needed:
Class Updater
#queue = :endless_queue
def self.perform
while true
Model.update_attribute(:course, http::get.new("asdasd").response)
end
end
end
Finaly i found a cool solution for my problem:
I use the god gem -> http://god.rubyforge.org/
with a bash script (link) for starting / stopping a simple rake task (with an infinite loop in it).
Now it works fine and i have even some monitoring with god running that ensures that the rake task runs ok.
I'm using delayed_job (tried both tobi's and collective_idea's) on site5.com shared hosting, with passenger as rails environment.
I managed to make jobs done.
However, it seems the plugin ignores any changes in a job class source code after first run.
I have restarted the server on every change (touch tmp/restart.txt) but it still ignores it.
Example:
file: lib/xx_job.rb
class XxJob
def perform
Rails.logger.info "XX START"
TempTest.delete_all
i = 0
10.times {
i+=1
TempTest.create(:name => "XXX")
sleep(1)
}
Rails.logger.info "XX END"
end
end
In a simple controller I call:
Delayed::Job.enqueue(XxJob.new)
Conclusions I have gathered:
If I change xx_job.rb to xx_job1.rb - error on the controller
If I change class XxJob to class XxJob1 - error on the controller
If I delete all the perform method content - the old code old code is executed
New .rb file with class and perform, enqueue this class - works perfectly
If I change something in that new file's perform and run job again - old code is executed
Between every change I made a restart for the server.
It seems like Passenger or something else saves class cache.
How can I delete this cache? Is is stored on the server somewhere? (I hope I have access to it from the shared hosting)
Thanks!
If you run delayed job workers daemonized, then you need to restart them to reload the code. Also, keep in mind that each worker loads its own instance of rails.
Eventually I figured that out - several workers were running in background, each of them caught a job and had their own cache.
I didn't know how to kill them so I changed the table's name for several seconds. That killed them :)
Then I used https://github.com/tobi/delayed_job/wiki/Running-Delayed::Worker-as-a-daemon as worker start, and it works great.
When a new resource is created and it needs to do some lengthy processing before the resource is ready, how do I send that processing away into the background where it won't hold up the current request or other traffic to my web-app?
in my model:
class User < ActiveRecord::Base
after_save :background_check
protected
def background_check
# check through a list of 10000000000001 mil different
# databases that takes approx one hour :)
if( check_for_record_in_www( self.username ) )
# code that is run after the 1 hour process is finished.
user.update_attribute( :has_record )
end
end
end
You should definitely check out the following Railscasts:
http://railscasts.com/episodes/127-rake-in-background
http://railscasts.com/episodes/128-starling-and-workling
http://railscasts.com/episodes/129-custom-daemon
http://railscasts.com/episodes/366-sidekiq
They explain how to run background processes in Rails in every possible way (with or without a queue ...)
I've just been experimenting with the 'delayed_job' gem because it works with the Heroku hosting platform and it was ridiculously easy to setup!!
Add gem to Gemfile, bundle install, rails g delayed_job, rake db:migrate
Then start a queue handler with;
RAILS_ENV=production script/delayed_job start
Where you have a method call which is your lengthy process i.e
company.send_mail_to_all_users
you change it to;
company.delay.send_mail_to_all_users
Check the full docs on github: https://github.com/collectiveidea/delayed_job
Start a separate process, which is probably most easily done with system, prepending a 'nohup' and appending an '&' to the end of the command you pass it. (Make sure the command is just one string argument, not a list of arguments.)
There are several reasons you want to do it this way, rather than, say, trying to use threads:
Ruby's threads can be a bit tricky when it comes to doing I/O; you have to take care that some things you do don't cause the entire process to block.
If you run a program with a different name, it's easily identifiable in 'ps', so you don't accidently think it's a FastCGI back-end gone wild or something, and kill it.
Really, the process you start should be "deamonized," see the Daemonize class for help.
you ideally want to use an existing background job server, rather than writing your own. these will typically let you submit a job and give it a unique key; you can then use the key to periodically query the jobserver for the status of your job without blocking your webapp. here is a nice roundup of the various options out there.
I like to use backgroundrb, its nice it allows you to communicate to it during long processes. So you can have status updates in your rails app
I think spawn is a great way to fork your process, do some processing in background, and show user just some confirmation that this processing was started.
What about:
def background_check
exec("script/runner check_for_record_in_www.rb #{self.username}") if fork == nil
end
The program "check_for_record_in_www.rb" will then run in another process and will have access to ActiveRecord, being able to access the database.