I have a simple resque job that accepts a model id, fetches it, and then calls a method on the fetched item.
class CupcakeWorker
#queue = :cupcake_queue
def self.perform(cupcake_id)
#cupcake = Cupcake.find(cupcake_id)
#cupcake.bake
end
end
I queue it from a controller action using the 'enqueue' method
def bake
Resque.enqueue(CupcakeWorker, params[:cupcake_id])
render :json => 'Baking...'
end
The job queues and executes correctly. However if I modify the record's data in the database and proceded to queue the job again the operation doesn't execute using the new values.
I can only get it to fetch the new values if I restart the resque worker process. Is resque caching the object? Is there a way to ensure it refetches from the database every time the worker is called?
This answer is a little crude, but could work. You could call reload on the ActiveRecord model before doing any processing. This forces ActiveRecord to update the data.
The worker would then look like this:
class CupcakeWorker
#queue = :cupcake_queue
def self.perform(cupcake_id)
#cupcake = Cupcake.find(cupcake_id).reload
#cupcake.bake
end
end
I'm afraid I don't know why Resque might be doing this, however. Try Sidekiq and see if you get better results.
Related
I have a before_destroy callback on my model named Connection.
It looks like this:
def run_my_worker
SomeWorker.perform_async(self.id)
end
The method calls a Sidekiq Worker to be performed. By the time the Sidekiq Worker is running, the model has been destroyed, and it can't "find" it when I query for it based on the id passed through to the worker.
How can I get around this/what are my alternatives to this situation?
The simplest approaches are:
do the work synchronously, or
pass all the data you need to the asynchronous method (like in Reyko's answer)
If neither of those work, you'll need to do the asynchronous work, then destroy the object once you're done with it.
One approach is to write a new method (like do_whatever_then_destroy). You can use Sidekiq's Batches feature to get a callback when the work has completed. At this point you can destroy the model object, since you're finally done with it.
You could pass the whole object which should be available to your worker even if the record gets destroyed.
def run_my_worker
SomeWorker.perform_async(self)
end
Update 1
Parse the json then inside your worker
def perform(my_object)
# parsed_object will store a hash representation of my_object
parsed_object = JSON.parse(my_object)
end
I'm trying to build a reference number for booking in Rails. I'm using
DateTime.now.strftime('Y%m%d%H%M%S')
for the reference number and concatenate some string. I used Sidekiq for performing this as a background job but then was confused with the result because it gave a different value in strftime.
def to_reserve
venue = Venue.find(params[:id])
venue_name = venue.name.delete(' ')
reference_number = ReferencesNumberWorker.perform_in(5.seconds, venue_name)
................
end
Sidekiq:
class ReferencesNumberWorker
include Sidekiq::Worker
def perform(venue_name)
reference_number = DateTime.now.strftime('Y%m%d%H%M%S')
return venue_name + reference_number
end
end
This is a sample result:
HotelMatt67fb02cb5da7871f92347df9
which I didn't expect with the right result.
That's appending the hash for a Sidekiq job to your reference number where you are expecting it to append the return of the Sidekiq result.
So essentially this method
def to_reserve
venue = Venue.find(params[:id])
venue_name = venue.name.delete(' ')
reference_number = ReferencesNumberWorker.perform_in(5.seconds, venue_name) <- This line will return Sidekiq job number, not the actual result from Sidekiq perform operation.
................
end
When you define reference number you're assuming it'll give you the Sidekiq return, which it won't. This is the fundamental point of background jobs. You're disconnecting the Sidekiq job from what you're doing, i.e. offloading to a separate process. So you can't expect a return, you're offloading the method to Sidekiq. Sidekiq's return is basically the job number.
If you want this to work then move the reference number computation that you're doing to Sidekiq in the to_reserve method OR if it must be done in Sidekiq move all of the logic from to_reserve to Sidekiq's perform method.
Also note, to use Sidekiq you should pass (to Sidekiq) the object's id so you can relook it up in Sidekiq and perform some idempotent operation on the object. In my opinion, I don't think in this case you need to use Sidekiq. Its unnecessary complexity, but to each their own.
Sidenote: Prefer the Time class (Time#now vs. DateTime#now) over DateTime to compute the strftime method. Its a pure Ruby method and is faster than the DateTime class (a known performance bottleneck). Also your code won't have a hard dependency on Rails.
I have a rake task send_emails which send e-mails to lot of people. I call this rake task from controller as mentioned in Rake in Background railscast. But I want to schedule this rake task to run at a particular date and time, which is not same for everyday (it's not a cron job). The date and time are set dynamically from a form.
For the above implemented rake task for sending emails, I want to show the status of the mailing process to the end-user. For instance, say there is a response object in the rake task which I can use as response.status,response.delivered?,response.address, etc. How can I access this object ( or any variable) in the rake file in my controller?
I don't want to use delayed_job but want to implement it's functionality of run_at and in_the_future. Also the whenever gem won't be able to solve my first problem coz I won't be able to pass date and time to it's scheduler.
First thing, calling rake task from controller is a bad practice. Ryan published that video at 2008 since that many better solution have came up. You shouldn't ignore it.
I suggest you to use delayed_job, it serves your needs in a great way. Since, if you want to invoke task dynamically, there should be some checker which will continuously check the desire field every second. Delayed job keep checking its database every time, you can use that.
Anyway,You can use something like this
def self.run_when
Scheduler.all.each do |s|
if d.dynamically_assigned_field < 1.second.ago
d.run_my_job!
d.status = "finished"
d.save
end
end
end
And, in model you can do something like this
def run_my_job!
self.status = "processing"
self.save
long_running_task
end
One thing also you should keep in mind that if too many workers/batch/cron job starts at run at same it will fight for resources and may enter into deadlock state. As per your server capacity, you should limit the running jobs.
Sidekiq is also a good option you can consider. Personally, i like sidekiq because it doesn't hit my database everytime , scales very effectively. It uses redis but it is expensive.
I would create new model for mail job, like this:
app/models/mail_job.rb
class MailJob
attr_accessible :email, :send_at, :delivered
scope :should_deliver, -> { where(delivered: false).where('send_at <= ?', Time.now) }
def should_deliver?
!delivered? && send_at <= Time.now
end
...
end
And use Sidekiq + Sidetiq, running every minute (or any other interval) and checking for mail jobs that should be delivered.
Hope this helps!
I am currently working on a rails project, i was asked to save progress of a sidekiq workers and store it, so the user who is using the application can see the progress. Now i am faced with this dilemna, is it better to just write out to a text file or save it in a database.
If it is a database, then how to save it in a model object. I know we can store the progress of workers by just sending out the info to log file.
class YourWorker
include Sidekiq::Worker
def perform
logger.info { "Things are happening." }
logger.debug { "Here's some info: #{hash.inspect}" }
end
So if i want to save the progress of workers in a data model, then how?
Your thread title says that the data is unstructured, but your problem description indicates that the data should be structured. Which is it? Speed is not always the most important consideration, and it doesn't seem to be very important in your case. The most important consideration is how your data will be used. Will the way your data is used in the future change? Usually, a database with an appropriate model is the better answer because it allows flexibility for future requirements. It also allows other clients access to your data.
You can create a Job class and then update some attribute of the currently working job.
class Job < ActiveRecord::Base
# assume that there is a 'status' attribute that is defined as 'text'
end
Then when you queue something to happen you create a new Job and pass the id of the Job to perform or perform_async.
job = Job.create!
YourWorker.perform_async job.id
Then in your worker, you'd receive the id of the job to be worked on, and then retrieve and update that record.
def perform(job_id)
job = Job.find job_id
job.status = "It's happening!"
job.save
end
So i have many database operations that i put into my helpers that i want to do in the background. As an example, I define a record_activity method in my Users helper. When a Post gets created I want to record this activity ie in the create method in the Posts controller:
def create
#operations to save the post
record_activity(user, post)
end
For performance reasons, I want to delay this record_activity and others, and run them with workers on the back-end. I use delayed_job for delaying mailers and it works excellently. In rails console, method.delay works great ie I could in rails console do:
record_activity.delay
However, the same .delay doesn't work when written in a controller ie the following still runs live, not delayed:
def create
#operations to save the post
record_activity(user, post).delay
end
Why is this? I'm using Rails 3.0.9 in one app and Rails 3.1.3 in another, plus I have delayed_job version 2.1.4.
Can anyone suggest how to make this work?
EDIT *
I think the answer provided by mu_is_too_short is the right path. It creates a delayed job, only it doesn't execute the record_activity method properly. When the delayed_job worker starts, it executes the delayed_job and has no errors, and deletes the record as if it worked. But no activity gets recorded. To give some context, here is the call and the method i am troubleshooting now:
self.delay.record_activity(user, #comment)
ANd the method:
def record_activity(current_user, act)
activity = Activity.new
activity.user_id = current_user.id
activity.act_id = act.id
activity.act_type = act.class.name
activity.created_at = act.created_at
activity.save
end
I then thought that maybe I couldn't pass user variables through, in this case, so I tried to just pass integer values and so on. I restarted the delayed_job workers and tried these methods, to no avail:
self.delay.record_activity(user.id, #comment.id, #comment.class.name, #comment.created_at)
And the altered method:
def record_activity(current_user_id, act_id, act_type, act_created_at)
activity = Activity.new
activity.user_id = current_user_id
activity.act_id = act_id
activity.act_type = act_type
activity.created_at = act_created_at
activity.save
end
I don't think record_activity.delay in the console is working the way you think it is. That will execute record_activity before delay has a chance to do anything.
The delay call has to go first, then you call your delayed method on what delay returns:
def create
self.delay.record_activity(user, post)
end
The delay call will return an object that intercepts all method calls (probably through method_missing), YAMLizes them, and adds the YAML to its delayed job queue table in the database. So, just saying record_activity.delay doesn't do anything useful, it just executes record_activity, creates the delayed-job interceptor object, and throws away what delay created.