Probably the title is not self explanatory, the situation is this:
# user.points: 0
user.update!(points: 1000)
UserMailer.notify(user).deliver_later. # user.points = 0 => Error !!!!
user instance is updated and after that the Mailer is called with the user as a parameter, and in the email that changes are non-existent: user.points=0 instead of 1000
But, with a sleep 1 just after the user_update the email is sent with the changes updated, so it seems that the email job is faster than updating data to database.
# user.points: 0
user.update!(points: 1000)
sleep 1
UserMailer.notify(user).deliver_later. # user.points = 1000 => OK
What's the best approach to solve this avoiding this two possible solutions?
One solution could be calling UserMailer.notify not with the user instance but with the user values
Another solution, it could be sending the mail in the user callback after_commit
So, is there another way to solve this keeping the user instance as the parameter and avoiding the after_commit callback?
Thanks
Remember, Sidekiq runs copy of your Rails app in a separate process, using Redis as the medium. When you call deliver_later, it does not actually 'pass' user to the mailer job. It spawns a thread that enqueues the job in Redis, passing a serialized hash of user properties, including the ID.
When the mailer job runs in the Sidekiq process, it loads a fresh copy of user from the database. If the transaction containing your update! in the main Rails app has not yet finished committing, Sidekiq gets the old record from the database. So, it's a race condition.
(update! already wraps an implicit transaction around itself if there isn't one, so wrapping it in your own transaction is redundant, and doesn't help the race condition since nested ActiveRecord transactions commit only when the outermost transaction commits.)
In a pinch, you could delay enqueuing the job with something hacky like .deliver_later(wait_until: 10.seconds.from_now), but your best bet is to put the mailer notification in an after_commit callback on your model.
class User < ApplicationRecord
after_commit :send_points_mailer
def send_points_mailer
return unless previous_changes.includes?(:points)
UserMailer.notify(self).deliver_later
end
end
A model's after_commit callbacks are guaranteed to run after the final transaction is committed, so, like nuking from orbit, it's the only way to be sure.
You didn't mention it, but I'm assuming you are using ActiveRecord? If so you likely need to assure to flush the database transaction before your sidekiq job is scheduled.
https://api.rubyonrails.org/v6.1.4/classes/ActiveRecord/Transactions/ClassMethods.html
Related
I have a before_destroy callback on my model named Connection.
It looks like this:
def run_my_worker
SomeWorker.perform_async(self.id)
end
The method calls a Sidekiq Worker to be performed. By the time the Sidekiq Worker is running, the model has been destroyed, and it can't "find" it when I query for it based on the id passed through to the worker.
How can I get around this/what are my alternatives to this situation?
The simplest approaches are:
do the work synchronously, or
pass all the data you need to the asynchronous method (like in Reyko's answer)
If neither of those work, you'll need to do the asynchronous work, then destroy the object once you're done with it.
One approach is to write a new method (like do_whatever_then_destroy). You can use Sidekiq's Batches feature to get a callback when the work has completed. At this point you can destroy the model object, since you're finally done with it.
You could pass the whole object which should be available to your worker even if the record gets destroyed.
def run_my_worker
SomeWorker.perform_async(self)
end
Update 1
Parse the json then inside your worker
def perform(my_object)
# parsed_object will store a hash representation of my_object
parsed_object = JSON.parse(my_object)
end
I have around 10 workers that performs a job that includes the following:
user = User.find_or_initialize_by(email: 'some-email#address.com')
if user.new_record?
# ... some code here that does something taking around 5 seconds or so
elsif user.persisted?
# ... some code here that does something taking around 5 seconds or so
end
user.save
The problem is that at certain times, two or more workers run this code at the exact time, and thus I later found out that two or more Users have the same email, in which I should always end up only unique emails.
It is not possible for my situation to create DB Unique Indexes for email as unique emails are conditional -- some Users should have unique email, some do not.
It is noteworthy to mention that my User model has uniqueness validations, but it still doesn't help me because, between .find_or_initialize_by and .save, there is a code that is dependent if the user object is already created or not.
I tried Pessimistic and Optimistic locking, but it didn't help me, or maybe I just didn't implement it properly... should you have some suggestions regarding this.
The solution I can only think of is to lock the other threads (Sidekiq jobs) whenever these lines of codes get executed, but I am not too sure how to implement this nor do I know if this is even a suggestable approach.
I would appreciate any help.
EDIT
In my specific case, it is gonna be hard to put email parameter in the job, as this job is a little more complex than what was just said above. The job is actually an export script in which a section of the job is the code above. I don't think it's also possible to separate the functionality above into another separate worker... as the whole job flow should be serial and that no parts should be processed parallely / asynchronously. This job is just one of the jobs that are managed by another job, in which ultimately is managed by the master job.
Pessimistic locking is what you want but only works on a record that exists - you can't use it with new_record? because there's nothing to lock in the DB yet.
I managed to solve my problem with the following:
I found out that I can actually add a where clause in Rails DB Uniqueness Partial Index, and thus I can now set up uniqueness conditions for different types of Users on the database-level in which other concurrent jobs will now raise an ActiveRecord::RecordNotUnique error if already created.
The only problem now then is the code in between .find_or_initialize_by and .save, since those are time-dependent on the User objects in which always only one concurrent job should always get a .new_record? == true, and other concurrent jobs should then trigger the .persisted? == true as one job would always be first to create it, but... all of these doesn't work yet because it is only at the line .save where the db uniqueness index validation gets called. Therefore, I managed to solve this problem by putting .save before those conditions, and at the same time I added a rescue block for .save which then adds another job to the queue of itself should it trigger the ActiveRecord::RecordNotUnique error, to make sure that async jobs won't get conflicts. The code now looks like below.
user = User.find_or_initialize_by(email: 'some-email#address.com')
begin
user.save
is_new_record = user.new_record?
is_persisted = user.persisted?
rescue ActiveRecord::RecordNotUnique => exception
MyJob.perform_later(params_hash)
end
if is_new_record
# do something if not yet created
elsif is_persisted
# do something if already created
end
I would suggest a different architecture to bypass the problem.
How about a producer-worker model, where one master Sidekiq process gets a list of email addresses, and then spawns a worker Sidekiq process for each email? Sidekiq makes this easy with a dedicated queue for master and workers to communicate.
Doing so, the email address becomes an input parameter of workers, so we know by construction that workers will not stump on each other data.
I'm running into a weird bug on Heroku, which I believe may be a race condition, and I'm looking for any sort of advice for solving it.
My application has a model that calls an external API (Twilio, if you're curious) after it's created. In this call, it passes a url to be called back once the third party completes their work. Like this:
class TextMessage < ActiveRecord::Base
after_create :send_sms
def send_sms
call.external.third.party.api(
:callback => sent_text_message_path(self)
)
end
end
Then I have a controller to handle the callback:
class TextMessagesController < ActiveController::Base
def sent
#textmessage = TextMessage.find(params[:id])
#textmessage.sent = true
#textmessage.save
end
end
The problem is that the third party is reporting that they're getting a 404 error on the callback because the model hasn't been created yet. Our own logs confirm this:
2014-03-13T18:12:10.291730+00:00 app[web.1]: ActiveRecord::RecordNotFound (Couldn't find TextMessage with id=32)
We checked and the ID is correct. What's even crazier is that we threw in a puts to log when the model is created and this is what we get:
2014-03-13T18:15:22.192733+00:00 app[web.1]: TextMessage created with ID 35.
2014-03-13T18:15:22.192791+00:00 app[web.1]: ActiveRecord::RecordNotFound (Couldn't find TextMessage with id=35)
Notice the timestamps. These things seem to be happening 58 milliseconds apart, so I'm thinking it's a race condition. We're using Heroku (at least for staging) so I think it might be their virtual Postgres databases that are the problem.
Has anyone had this sort of problem before? If so, how did you fix it? Are there any recommendations?
after_create is processed within the database transaction saving the text message. Therefore the callback that hits another controller cannot read the text message. It is not a good idea to have an external call within a database transaction, because the transaction blocks parts of the database for the whole time the slow external request takes.
The simples solution is to replace after_save with after_commit (see: http://apidock.com/rails/ActiveRecord/Transactions/ClassMethods/after_commit)
Since callbacks tend to become hard to understand (and may lead to problems when testing), I would prefer to make the call explicit by calling another method. Perhaps something like this:
# use instead of .save
def save_and_sent_sms
save and sent_sms
end
Perhaps you want to sent the sms in the background, so it does not slow down the web request for the user. Search for the gems delayed_job or resque for more information.
Do you have master/slave database where you always write to master but read from slave? This sounds like the db replication lag.
We solved such problems by forcing a read being made from the master database in the specific controller action.
Another way would be to call send_sms after the replication has been finished.
Related to Run rails code after an update to the database has commited, without after_commit, but I think deserving its own question.
If I have code like this:
my_instance = MyModel.find(1)
MyModel.transaction do
my_instance.foo = "bar"
my_instance.save!
end
new_instance = MyModel.find(1)
puts new_instance.foo
Is this a guarantee that new_instance.foo will always output "bar" and not its previous value? I'm looking for a way to ensure that all the database actions that occur in a previous statement are committed BEFORE executing my next statements. Rails has an after_commit hook for this, but I don't want this code executed every time... only in this specific context.
I can't find anything in the documentation on Transactions that would indicate if Transaction blocks are "blocking". If they are blocking, that will satisfy my requirement. Unfortunately, I can't think of a practical way to test this behavior to confirm my suspicions one way or another.
Still researching this, but I think a transaction does block code execution until after the database confirms that it has written. Since "save!" is automatically wrapped in a transaction by Rails, the relevant code should run synchronously. The extra transaction block should be unnecessary.
I don't think Rails returns as soon as it hands off the call to the DB when the DB calls are within a transaction. The confusion I had was with after_save callbacks. After_save callbacks suffer from race conditions because they are in fact part of the transaction that saves are automatically wrapped in, so any code called by an after_save callback is not race condition safe, it is not protected by the transaction. Only after_commit calls are safe. Within the transaction Rails will hand off to the DB and then execute after_save callbacks before the DB has finished committing.
Studying this for more insights:
https://github.com/rails/rails/blob/bfdd3c2182156fa2cb81ed4f048b065a2d6f1341/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb
UPDATE
Changing my answer to "no". It doesn't appear that save! or save blocks execution. From these two resources, looks like this is a common problem:
https://github.com/resque/resque/wiki/FAQ#how-do-you-make-a-resque-job-wait-for-an-activerecord-transaction-commit
https://blog.engineyard.com/2011/the-resque-way
So I'm using Delayed::Job workers (on Heroku) as an after_create callback after a user creates a certain model.
A common use case, though, it turns out, is for users to create something, then immediately delete it (likely because they made a mistake or something).
When this occurs the workers are fired up, but by the time they query for the model at hand, it's already deleted, BUT because of the auto-retry feature, this ill-fated job will retry 25 times, and definitely never work.
Is there any way I can catch certain errors and, when they occur, prevent that specific job from ever retrying again, but if it's not that error, it will retry in the future?
Abstract the checks into the function you call with delayed_job. Make the relevant checks wether your desired job can proceed or not and either work on that job or return success.
To expand on David's answer, instead of doing this:
def after_create
self.send_later :spam_this_user
end
I'd do this:
# user.rb
def after_create
Delayed::Job.enqueue SendWelcomeEmailJob.new(self.id)
end
# send_welcome_email_job.rb
class SendWelcomeEmailJob < Struct(:user_id)
def perform
user = User.find_by_id(self.user_id)
return if user.nil? #user must have been deleted
# do stuff with user
end
end