I have a case scenario where I need to run multiple record updates in the background(using resque) and I want to give user visual indicator of how the task is running(eg started/running/finished).
One way of achieving this(which I can think of) is saving the current state into a table, then showing the state to user by simple page refresh.
Can anyone suggest a better solution of doing it?I want to avoid creating the whole migration, model, controller for this.
Thanks
As I've commented, resque-status gem could be useful for you. I am not sure if that is an answer but since you said that you do not want to create migration, model and controller for this. Thus, a gem might be the way to go.
From the job id you can get the status you are looking for, for example:
status = Resque::Plugins::Status::Hash.get(job_id)
status.working? #=> true
There is also a front-end called resque-web, check that out too.
You may use ruby's global variable $var_name = 'foo'. However I am not sure about it, because they are considered bad practice in rails, but in this case I see them reasonable, as soon as their name is very unique.
It can be done like (in case of resque):
class UpdateJob
#queue = data
def self.perform
$my_job_name_is_running = true
MyJobName.new.run
$my_job_name_is_running = nil
end
end
then you can access them from anywhere in the app:
while $my_job_name_is_running
puts "job is running..." if $my_job_name_is_running
sleep 3 # important to not overload your processor
end
Ruby global vars are not very popular. Check docs for more info https://ruby-doc.org/docs/ruby-doc-bundle/UsersGuide/rg/globalvars.html
Related
Sidekiq recommends that all jobs be idempotent (able to run multiple times without being an issue) as it cannot guarantee a job will only be run one time.
I am having trouble understanding the best way to achieve that in certain cases. For example, say you have the following table:
User
id
email
balance
The background job that is run simply adds some amount to their balance
def perform(user_id, balance_adjustment)
user = User.find(user_id)
user.balance += balance_adjustment
user.save
end
If this job is run more than once their balance will be incorrect. What is best practice for something like this?
If I think about it a potential solution I can come up with is to create a record before scheduling the job that is something like
PendingBalanceAdjustment
user_id
balance_adjustment
When the job runs it will need to acquire a lock for this user so that there's no chance of a race condition between two workers and then will need to both update the balance and delete the record from pending balance adjustment before releasing the lock.
The job then looks something like this?
def perform(user_id, balance_adjustment_id)
user = User.find(user_id)
pba = PendingBalanceAdjustment.where(:balance_adjustment_id => balance_adjustment_id).take
if pba.present?
$redis.lock("#{user_id}/balance_adjustment") do
user.balance += pba.balance_adjustment
user.save
pba.delete
end
end
end
This seems to solve both
a) Race condition between two workers taking the job at the same time (though you'd think Sidekiq could guarantee this already?)
b) A job being run multiple times after running successfully
Is this pattern a good solution?
You're on the right track; you want to use a database transaction, not a redis lock.
I think you're on the right track too but you're solution might be overkill since I don't have full knowledge of your application.
BUT, a simpler solution would simply be to have a flag on you User model like balance_updated:datetime. So, you could check that before updating.
As Mike mentions using a Transaction block should ensure it's thread safe.
In any case, to answer your question more generally... having an updated_ column is usually good enough to start with, and then if it gets complicated you can move this stuff to another model.
I'm trying to battle some race cases with my background task manager. Essentially, I have a Thing object (already exists) and assign it some properties, and then save it. After it is saved with the new properties, I queue it in Resque, passing in the ID.
thing = Thing.find(1)
puts thing.foo # outputs "old value"
thing.foo = "new value"
thing.save
ThingProcessor.queue_job(thing.id)
The background job will load the object from the database using Thing.find(thing_id).
The problem is that we've found Resque is so fast at picking up the job and loading the Thing object from the ID, that it loads a stale object. So within the job, calling thing.foo will still return "old value" like 1/100 times (not real data, but it does not happen often).
We know this is a race case, because rails will return from thing.save before the data has actually been commit to the database (postgresql in this case).
Is there a way in Rails to only execute code AFTER a database action has commit? Essentially I want to make sure that by the time Resque loads the object, it is getting the freshest object. I know this can be achieved using an after_commit hook on the Thing model, but I don't want it there. I only need this to happen in this one specific context, not every time the model has commit changed to the DB.
You can put in a transaction as well. Just like the example below:
transaction do
thing = Thing.find(1)
puts thing.foo # outputs "old value"
thing.foo = "new value"
thing.save
end
ThingProcessor.queue_job(thing.id)
Update: there is a gem which calls After Transaction, with this you may solve your problem. Here is the link:
http://xtargets.com/2012/03/08/understanding-and-solving-race-conditions-with-ruby-rails-and-background-workers/
What about wrapping a try around the transaction so that the job is queued up only upon success of the transaction?
I had a similar issue, where by I needed to ensure that a transaction had commited before running a series of action. I ended up using this Gem:
https://github.com/Envek/after_commit_everywhere
It meant that I could do the following:
def finalize!
Order.transaction do
payment.charge!
# ...
# Ensure that we only send out items and perform after actions when the order has defintely be completed
after_commit { OrderAfterFinalizerJob.perform_later(self) }
end
end
One gem to allow that is https://github.com/Ragnarson/after_commit_queue
It is a little different than the other answer's after_commit_everywhere gem. The after_commit_everywhere call seems decoupled from current model being saved or not.
So it might be what you expect or not expect, depending on your use case.
I don't get it.
Per Sidekiq documentation, each worker (mine is called FeedWorker) can only contain one method called perform. Well, what if I want to run mulitple methods through the same worker?
For instance, my FeedWorker (you guessed it, it processes an activity feed) should run the following 3 methods:
announce_foo
announce_bar
invite_to_foo
I don't think this is an unreasonable expectation. I'm sure other folks have considered this. I'm no genius, but I know I'm not breaking new ground in expectations here. Yet it's not clear how one would do this.
Right now, it looks like I have to code this way:
def perform(id, TYPE)
if TYPE == BAR
Bar.find(id) and_announce_bar
else
Foo.find(id) and_announce_foo
end
end
Boring and ugly code. There must be better out there.
Any help appreciated!
perform method is the entry point of your Worker. Inside of it you can create as many instance methods as you want, to organize your code as it best fits your need. It's a good practice though to keep worker code as slim as possible. Calling other objects from inside of it for example is a way to achieve that. You'll find your code will be easier to test too.
I had the same question for awhile and now have a rather simple solution: use the Delayed Extension method on any class, as explained in docs, for ex:
# Usage: EmailWorker.delay.do_something
class EmailWorker
class << self
def send_this(attrs)
MyMailer.some_action(attrs).deliver
end
def send_that(attrs)
MyMailer.another_action(attrs).deliver
end
end
end
Any class can be delayed, so no need to include Sidekiq::Worker if you're not going to use perform_async method.
The problem I've had with this is that the per-worker options won't be used unless you go thru the perform_async method.
Lets say that on top of my Rails app there is a bar with piece of text displayed - latest hot deal, scheduled downtime notfication, something like that. It's a single, on of a kind information that needs to be accessed on basically every request, and may be updated from time to time. What is the best way to achieve this?
What I'd like to do is some kind of permanent global variable (accessible from controllers).
It will be updated very rarely, so there's no problem if for some time after update there will be an inconsistency between workers.
On the other hand, it should be persistent in case of server fault (periodic backup is enough).
It will be accessed really often, so it should be as fast as possible - preferably stay in memory.
Also, it's only one of a kind, so I'd really prefer not to bloat the app with a dedicated database model.
Something like that is damn easy in Node.js for example, but I couldn't find a single way to achieve this in Rails. What shall I do?
EDIT
Thanks for the answers so far, but while they're inspiring, I think that I should stress out one key functionality that they're all missing. The variable should be editable inside the app and persistent. While it's possible to edit your variables, in case of server restart I'm back to the default - which is bad.
It really depends on what you are looking for. You could do something very simply by putting it in your application_controller.rb
class ApplicationController < ActionController::Base
def system_message
"Come buy our amazing .99 iphone chocolate bar apps, with 100% more gamification!"
end
end
That function (and string) is then accessible from any controller in your application. You could also specify something in the after_initialize block in your application.rb file.
config.after_initialize do
::MYTEXT = "MY SUPER AMAZING TEXT"
end
You could also create your own file under the initializers directory, which is preloaded in rails.
so siteAnnounce.rb
MYANNOUNCEMENT = "NOW LISTEN TO ME!"
You may also want to check out this Railscast video about site wide announcements
I would store it in the database and let caching take care of it.
I feel that global variables are fine, when appropriate, for code that needs to share that common value in many places but that is the code, not the the user view.
This is clearly true in this case as the OP has bolded 'editable by the app'. So I would have a view that lets the users enter it, it gets stored in a db table and then recalled as needed (as cached once used once).
Well I had faced a similar problem.
My problem was I needed a global variable in all the levels (MVC).
We went to use Memcache to store the variable.
May be you can go for a similar solution.
And as an added bonus you can change it throughout the program.
You could declare it as a constant in an initializer:
config/initialzers/foo.rb:
MYVARIABLE = 'some string'
Accessible from anywhere in your application as MYVARIABLE
Ok, so here's what I did. Instead of just putting the value to an initializer, I've made there a simple class that handles it. The variable itself is stored in a predefined file. Besides of reading the file upon the initialization, the class updates file when the value is changed, and also re-read the file periodically to maintain consistency across workers. I've also put there some basic JSON handling and backup functionality to make life easier.
For anyone interested, here's the important code:
class Pomegranate
def initialize
#delay = 30.minutes
#path = "db/pomegranate.json"
#valid = Time.now - 1
validate
end
def get(*p)
validate
p.inject(#data) {|object,key| object[key] if object}
end
def set(*p, q, v)
hash = p.inject(#data) {|object,key| object[key]||={}}
hash[q] = v
end
def save
#valid = Time.now + #delay
File.open(#path,"w") {|f| f.write(#data.to_json)}
end
private
def validate
if #valid < Time.now
#data = ActiveSupport::JSON.decode(File.read(#path)) rescue {}
#valid = Time.now + #delay
#valid = Time.now - 1 if #data.empty?
end
end
end
$pom = Pomegranate.new
Source:
Where to put Global variables in Rails 3
Try putting it in your applicaton.rb like this:
module MyAppName
class Application < Rails::Application
YOUR_GLOBAL_VAR = "test"
end
end
Then you can call it with the namespace in your controllers, views or whatever..
MyAppName::Application::YOUR_GLOBAL_VAR
Another alternative would be using something like settingslogic. With settingslogic, you just create a yml config file and a model (Settings.rb) that points to the config file. Then you can access these settings anywhere in your rails app with:
Settings.my_setting
I've started putting constants and variables like this in the configuration object, e.g.
TestApp::Application.config.foo = 'bar'
TestApp::Application.config.something = { :a => 1, :b => 2 }
I'm sure "global variable" will get the hair on the back of everyone's neck standing up. What I'm trying to do is store a hierarchical menu in an acts_as_tree data table (done). In application_helper.rb, I create an html menu by querying the database and walking the tree (done). I don't want to do this for every page load.
Here's what I tried:
application.rb
config.menu = nil
application_helper.rb
def my_menu_builder
return MyApp::Application.config.menu if MyApp::Application.config.menu
# All the menu building code that should only run once
MyApp::Application.config.menu = menu_html
end
menu_controller.rb
def create
# whatever create code
expire_menu_cache
end
protected
def expire_menu_cache
MyApp::Application.config.menu = nil
end
Where I stand right now is that on first page load, the database is, indeed, queried and the menu built. The results are stored in the config variable and the database is never again hit for this.
It's the cache expiration part that's not working. When I reset the config.menu variable to nil, presumably the next time through my_menu_builder, it will detect that change and rebuild the menu, caching the new results. Doesn't seem to happen.
Questions:
Is Application.config a good place to store stuff like this?
Does anyone see an obvious flaw in this caching strategy?
Don't say premature optimization -- that's the phase I'm in. The premature-optimization iteration :)
Thanks!
I would avoid global variables, and use Rails' caching facilities.
http://guides.rubyonrails.org/caching_with_rails.html
One way to achieve this is to set an empty hash in your application.rb file:
MY_VARS = {}
Then you can add whatever you want in this hash which is accessible everywhere.
MY_VARS[:foo] = "bar"
and elsewhere:
MY_VARS[:foo]
As you felt, this is not the Rails way to behave, even if it works. There are different ways to use caching in Rails:
simple cache in memory explained here:
Rails.cache.read("city") # => nil
Rails.cache.write("city", "Duckburgh")
Rails.cache.read("city") # => "Duckburgh"
use of a real engine like memcached
I encourage you to have a look at http://railslab.newrelic.com/scaling-rails
This is THE place to learn caching in all it's shapes.