As per the resque-status home page on GitHub I should be able to pass back data from a job. For some reason this does not seem to be working for me. If anyone else has encountered this problem and worked around it I would like to know how.
I am using resque-status with JRuby 1.6.5 in a Rails 3.2.3 application.
Passing back data from the job
You may want to save data from inside the job to access it from outside the job.
A common use-case is web-triggered jobs that create files, later available for download by the user.
A Status is actually just a hash, so inside a job you can do:
status['filename'] = '/myfilename'
Also, all the status setting methods take any number of hash arguments. So you could do:
complete('filename' => '/myfilename')
Apparently such functionality is not implemented, as read on
https://github.com/quirkey/resque-status/issues/66
we've found a work around using the function set_status to add the required data to the status hash:
set_status({"my variable" => "my value" })
hope this helps!
Related
So I've been looking for the simplest way to send an e-mail when X column of Payments table in the database is == 'condition'. Basically what I want is to add a payment and set a date like 6 months. When 6 months have passed I want to send the mail. I've seen many solutions like using Whenever cron jobs and others but I want to know the absolute simplest way (perhaps using Rails only without relying on outside source) to keep my application light and clean. I was thinking I could use the auto generated created_at to evaluate when x time has passed.
Since you have a column in your db for the time to send email, make it a datetime datatype and you can set the email date as soon as the event payment event is created. Then, you can have a rake task where,
range = Time.now.beginning_of_day..Time.now.end_of_day
Payment.where(your_datetime_custom_column: range).each do |payment|
payment.user.send_email
end
and you can run this task everyday from the scheduler.
The "easiest" way is to use Active Job in conjunction with a state machine:
EmailJob.set(wait: 6.months).perform_later(user.id) if user.X_changed?
The problem with this is that the queue will accumulate jobs since jobs don't get handled right away. This may lead to other performance issues since there are now more jobs to scan and they're taking up more memory.
Cron jobs are well suited for this kind of thing. Depending on your hosting platform, there may be various other ways to handle this; for example, Heroku has Heroku Scheduler.
There are likely other ways to schedule repeating tasks without cron, such as this SO answer.
edit: I did use a gem once called 'fist_of_fury', but it's not currently maintained and I'm not sure how it would perform in a production environment. Below are some snippets for how I used it in a rails project:
in Gemfile
gem 'fist_of_fury'
in config/initializers/fist_of_fury.rb
# Ensure the jobs run only in a web server.
if defined?(Rails::Server)
FistOfFury.attack! do
ObserveAllJob.recurs { minutely(1) }
end
end
in app/jobs/observe_all_job.rb
class ObserveAllJob
include SuckerPunch::Job
include FistOfFury::Recurrent
def perform
::Task.all.each(&:observe)
end
end
Here is my scenario, i'm using resque to queue a job in redis, the usual way its done in ROR. The format of my key looks something like this (as per my namespace convention)
"resque:lock:Jobs::XYZ::SomeCreator-{:my_ids=>[101]}"
The job runs successfully to completition. But the key still exists in redis. For a certain flow, i need to queue and execute the job again for the same parameters (the key will essentially be same). But seems like the job does not get queued.
My guess is that since the key already exists in Redis, it does not queue the job again.
Questions:
Is this behavior of resque normal (not removing the key after successful completition)?
If Yes, how should i tackle this scenario (as per best practices)?
If No, can you help me understand what is going wrong?
After a couple of hours of debugging, finally this is the observed behavior:
I was creating the job and passing the options (parameters) with symbolized keys which when created the Redis key for the same job with symbolized param in the key.
Example:
Jobs::Abc::SomeJobCreator.create({:some_ids => [101]}) would create the "redis key" as "resque:lock:Jobs::Abc::SomeJobCreator.create({:some_ids => [101]})" (Notice the key being a symbol in the key)
Now when the after_perform_hook executes, it tries to remove the Redis Key but it searches the key with Stringified keys: "resque:lock:Jobs::Abc::SomeJobCreator-({\"some_ids\"=>[101]}" Which obviously won't be found, as the key in Redis has symbolized params in key.
To fix this issue i had to change the calls to job creation in the code and use stringified params like this: Jobs::Abc::SomeJobCreator.create({'some_ids' => [101]}). This works fine.
Not sure if this has anything to do with the version of Resque. Since its a old codebase i haven't yet updated the version. Its currently at Resque v1.25.2
I have a case scenario where I need to run multiple record updates in the background(using resque) and I want to give user visual indicator of how the task is running(eg started/running/finished).
One way of achieving this(which I can think of) is saving the current state into a table, then showing the state to user by simple page refresh.
Can anyone suggest a better solution of doing it?I want to avoid creating the whole migration, model, controller for this.
Thanks
As I've commented, resque-status gem could be useful for you. I am not sure if that is an answer but since you said that you do not want to create migration, model and controller for this. Thus, a gem might be the way to go.
From the job id you can get the status you are looking for, for example:
status = Resque::Plugins::Status::Hash.get(job_id)
status.working? #=> true
There is also a front-end called resque-web, check that out too.
You may use ruby's global variable $var_name = 'foo'. However I am not sure about it, because they are considered bad practice in rails, but in this case I see them reasonable, as soon as their name is very unique.
It can be done like (in case of resque):
class UpdateJob
#queue = data
def self.perform
$my_job_name_is_running = true
MyJobName.new.run
$my_job_name_is_running = nil
end
end
then you can access them from anywhere in the app:
while $my_job_name_is_running
puts "job is running..." if $my_job_name_is_running
sleep 3 # important to not overload your processor
end
Ruby global vars are not very popular. Check docs for more info https://ruby-doc.org/docs/ruby-doc-bundle/UsersGuide/rg/globalvars.html
What I'm doing
I'm using the twitter gem (a Ruby wrapper for the Twitter API) in my app, which is run on Heroku. I use Heroku's Scheduler to periodically run caching tasks that use the twitter gem to, for example, update the list of retweets for a particular user. I'm also using delayed_job so scheduler calls a rake task, which calls a method that is 'delayed' (see scheduler.rake below). The method loops through "authentications" (for users who have authenticated twitter through my app) to update each authorized user's retweet cache in the app.
My question
What am I doing wrong? For example, since I'm using Heroku's Scheduler, is delayed_job redundant? Also, you can see I'm not catching (rescuing) any errors. So, if Twitter is unreachable, or if a user's auth token has expired, everything chokes. This is obviously dumb and terrible because if there's an error, the entire thing chokes and ends up creating a failed delayed_job, which causes ripple effects for my app. I can see this is bad, but I'm not sure what the best solution is. How/where should I be catching errors?
I'll put all my code (from the scheduler down to the method being called) for one of my cache methods. I'm really just hoping for a bulleted list (and maybe some code or pseudo-code) berating me for poor coding practice and telling me where I can improve things.
I have seen this SO question, which helps me a little with the begin/rescue block, but I could use more guidance on catching errors, and one the higher-level "is this a good way to do this?" plane.
Code
Heroku Scheduler job:
rake update_retweet_cache
scheduler.rake (in my app)
task :update_retweet_cache => :environment do
Tweet.delay.cache_retweets_for_all_auths
end
Tweet.rb, update_retweet_cache method:
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
end
end
User.rb, twitter method:
def twitter
authentication = Authentication.find_by_user_id_and_provider(self.id, "twitter")
if authentication
#twitter ||= Twitter::Client.new(:oauth_token => authentication.oauth_token, :oauth_token_secret => authentication.oauth_secret)
end
end
Note: As I was posting this, I noticed that I'm finding all "twitter" authentications in the "cache_retweets_for_all_auths" method, then calling the "User.twitter" method, which specifically limits to "twitter" authentications. This is obviously redundant, and I'll fix it.
First what is the exact error you are getting, and what do you want to happen when there is an error?
Edit:
If you just want to catch the errors and log them then the following should work.
def self.cache_retweets_for_all_auths
#authentications = Authentication.find_all_by_provider("twitter")
#authentications.each do |authentication|
being
authentication.user.twitter.retweeted_to_me(include_entities: true, count: 200).each do |tweet|
# Actually build the cache - this is good - removing to keep this short
end
rescue => e
#Either create an object where the error is log, or output it to what ever log you wish.
end
end
end
This way when it fails it will keep moving on to the next user but will still making a note of the error. Most of the time with twitter its just better to do something like this then try to do with each error on its own. I have seen so many weird things out of the twitter API, and random errors, that trying to track down every error almost always turns into a wild goose chase, though it is still good to keep track just in case.
Next for when you should use what.
You should use a scheduler when you need something to happen based on time only, delayed jobs for when its based on an user action, but the 'action' you are going to delay would take to long for a normal response. Sometimes you can just put the thing plainly in the controller also.
So in other words
The scheduler will be fine as long as the time between updates X is less then the time it will take for the update to happen, time Y.
If X < Y then you might want to look at calling the logic from the controller when each indvidual entry is accessed, isntead of trying to do them all at once. The idea being you would only update it after a certain time as passed so. You could store the last time update either on the model itself in a field like twitter_udpate_time or in a redis or memecache instance at a unquie key for the user/auth.
But if the individual update itself is still too long, then thats when you should do the above, but instead of doing the actually update, call a delayed job.
You could even set it up that it only updates or calls the delayed job after a certain number of views, to further limit stuff.
Possible Fancy Pants
Or if you want to get really fancy you could still do it as a cron job, but have a point system based on views that weights which entries should be updated. The idea being certain actions would add points to certain users, and if their points are over a certain amount you update them, and then remove their points. That way you could target the ones you think are the most important, or have the most traffic or show up in the most search results etc etc.
Next off a nick picky thing.
http://api.rubyonrails.org/classes/ActiveRecord/Batches.html
You should be using
#authentications.find_each do |authentication|
instead of
#authentications.each do |authentication|
find_each pulls in only 1000 entries at a time so if you end up with a lof of Authentications you don't end up pulling a crazy amount of entries into memory.
I am using Ruby on Rails 3.2.2 and I would like to know if it is possible to use something like caching or IdentityMapping techniques or, at least, "protect" the database from hitting (for example, if the current object data isn't changed the database should not be hit at all) when running the ActiveRecord::Persistence#update_attribute method multiple times in the same controller action and view rendering work flows.
object.update_attribute(foo, bar) if object.changed?
For more infos...
http://ar.rubyonrails.org/classes/ActiveRecord/Dirty.html
hope this helps,...