I currently have the following simple controller:
class SimpleController < ApplicationController
def index
#results = fetch_results
end
end
fetch_results is a fairly expensive operation so although the above works, I don't want to run it every time the page is refreshed. How can I decouple the updating of #results so that it's updated on a fixed schedule, let's say every 15 minutes.
That way each time the page is loaded it will just return the current #results value, which at worst would be 14 minutes and 59 seconds out of date.
You might want to use Rails` low level caching for this:
def index
#results = Rails.cache.fetch('fetched_results', expires_in: 15.minutes) do
fetch_results
end
end
Read more about how to configure Rails' caching stores in the Rails Guide.
Well I would store this in a database table and update regularly via a background job. The updates will be relative to some events, like if the user has done something that may change the result, then the result will be recalculated and updated.
Another solution is to update the result regularly, say every hour, using cron jobs. There is a good gem that can handle it.
Related
In my app there is a financial overview page with quite a lot of queries. This page is refreshed once a month after executing a background job, so I added caching:
#iterated_hours = Rails.cache.fetch("productivity_data", expires_in: 24.hours) do
FinancialsIterator.new.create_productivity_iterations(#company)
end
The cache must expire when the background job finishes, so I created a model CacheExpiration:
class CacheExpiration < ApplicationRecord
validates :cache_key, :expires_in, presence: true
end
So in the background job a record is created:
CacheExpiration.create(cache_key: "productivity_data", expires_in: DateTime.now)
And the Rails.cache.fetch is updated to:
expires_in = get_cache_key_expiration("productivity_data")
#iterated_hours = Rails.cache.fetch("productivity_data", expires_in: expires_in) do
FinancialsIterator.new.create_productivity_iterations(#company)
end
private def get_cache_key_expiration(cache_key)
cache_expiration = CacheExpiration.find_by_cache_key(cache_key)
if cache_expiration.present?
cache_expiration.expires_in
else
24.hours
end
end
So now the expiration is set to a DateTime, is this correct or should it be a number of seconds? Is this the correct approach to make sure the cache is expired only once when the background job finishes?
Explicitly setting an expires_in value is very limiting and error prone IMO. You will not be able to change the value once a cache value has been created (well you can clear the cache manually) and if ever you want to change the background job to run more/less often, you also have to remember to update the expires_in value. Additionally, the time when the background job is finished might be different from the time the first request to the view is made. As a worst case, the request is made a minute before the background job updates the information for the view. Your users will have to wait a whole day to get current information.
A more flexible approach is to rely on updated_at or in their absence created_at fields of ActiveRecord models.
For that, you can either rely on the CacheExpiration model you already created (it might already have the appropriate fields) or use the last of the "huge number of records" you create. Simply order them and take the last SomeArModel.order(created_at: :desc).first
The benefit of this approach is that whenever the AR model you create is updated/created, you cache is busted and a new one will be created. There is no longer a coupling between the time a user called the end point and the time the background job ran. In case a record is created by any means other than the background job, it will also simply be handled.
ActiveRecord models are first class citizens when it comes to caching. You can simply pass them in as cache keys. Your code would then change to:
Rails.cache.fetch(CacheExpiration.find_by_cache_key("productivity_data")) do
FinancialsIterator.new.create_productivity_iterations(#company)
end
But if at all possible, try to find an alternative model so you no longer have to maintain CacheExpiration.
Rails also has a guide on that topic
Currently I am building an application that allows users to place bids on products and admins to approve them. The 'transactions' themselves take place outside of the scope of the application. Currently, users see the price of an asset on the transaction/new page and submit a bid by submitting the form. Admins click a button to approve the bid.
class TransactionsController < ApplicationController
before_action :get_price
def new
#price = get_price
#tansaction = Transaction.new
end
###
def get_price
#price = <<Some External Query>>
end
def approve
t = Transaction.find(params[:id])
t.status = "Approved"
t.update!
end
Obviously this is not ideal. I don't want to query the API every time a user wants to submit a bid. Ideally, I could query this API every 5-10 seconds in the background and use the price in that manner. I have looked at a couple of techniques for running background jobs including delayed_job, sidekiq, and resque. For example in sidekiq I could run something like this:
#app/workers/price_worker.rb
class PriceWorker
include Sidekiq::Worker
def perform(*args)
get_price
end
def get_price
#price = <<Some External Query>>
end
end
#config/initializers/sidekiq.rb
schedule_file = "config/schedule.yml"
if File.exists?(schedule_file) && Sidekiq.server?
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
#config/schedule.yml
my_price_job:
cron: "*/10 * * * * * "
class: "PriceWorker"
That code runs. The problem is I am kind of lost on how to handle the price variable and pass it back to the user from the worker. I have watched the Railscasts episodes for both sidekiq and resque. I have written background workers and jobs that queue and run properly, but I cannot figure out how to implement them into my application. This is the first time I have dealt with background jobs so I have a lot to learn. I have spent sometime researching this issue and it seems like more background jobs are used for longer running tasks like updating db indexes rather than constantly recurring jobs (like an API request every 5 seconds).
So to sum up, What is the proper technique for running a constantly recurring task such as querying an external API in Rails? Any feedback on how to do this properly will be greatly appreciated! Thank you.
That is not how background jobs work. You're right, you have a lot of reading up to do. Think of running an asynchronous job processor like Sidekiq as running an entirely separate app. It shares the same code base as your Rails app but it runs completely separately. If you want these two separate apps to talk to each other then you have to design and write that code.
For example, I would define a cache with reader and writer methods, then have the cache populated when necessary:
someone loads product "foo" for the first time on your site
Rails checks the cache and finds it empty
Rails calls the external service
Rails saves the external service response to the cache using its writer method
Rails returns the cached response to the client
The cache would be populated thereafter by Sidekiq:
someone loads product "foo" for the second time on your site
Rails checks the cache and finds the cached value from above
Rails fires a Sidekiq job telling it to refresh the cache
Rails returns the cached response to the client
Continuing from step 3 above:
Sidekiq checks to see when the cache was last refreshed, and if it was more than x seconds ago then continue, else quit
Sidekiq calls the external service
Sidekiq saves the external service response to the cache using its writer method
When the next client loads product "foo", Rails will read the cache that was updated (or not updated) by Sidekiq.
With this type of system, the cache must be an external store of some kind like a relational database (MySQL, Postgres, Sqlite) or a NoSQL database (Redis, memcache). You cannot use the internal Rails cache because the Rails cache exists only within the memory space of the Rails app, and is not readable by Sidekiq. (because Sidekiq runs as a totally separate app)
I guess in this case you should use rails cache. Put something like this in your controller:
#price = Rails.cache.fetch('price') do
<<Some external query>>
end
you can also configure cache expiration date, by setting expires_in argument, see https://apidock.com/rails/ActiveSupport/Cache/Store/fetch for more information.
Regarding using background jobs to update your "price" value, you would need to store retrieved data anyways (use some kind of database) and fetch it in your controller.
I have a "Play" button in my app that checks a stock value from an API and creates a Position object that holds that value. This action uses Resque to make a background job using Resque and Redis in the following way:
Controller - stock_controller.rb:
def start_tracking
#stock = Stock.find(params[:id])
Resque.enqueue(StockChecker, #stock.id)
redirect_to :back
end
Worker:
class StockChecker
#queue = :stock_checker_queue
def self.perform(stock_id)
stock = Stock.find_by(id: stock_id)
stock.start_tracking_position
end
end
Model - stock.rb:
def start_tracking_position
// A Position instance that holds the stock value is created
end
I now want this to happen every 15 minutes for every Stock object. I looked at the scheduling section in the Ruby Toolbox website and have a hard time deciding what fits to my needs and how to start implementing it.
My concern is that my app will create tons of Position objects so I need something that is simple, uses Resque and can withstand this type of object creating without overloading the app.
What gem should I use and what is the simplest way to make my Resque Job happen every 15 minutes when the start_tracking action happens on a Stock object?
I've found resque scheduler to be useful: https://github.com/resque/resque-scheduler.
Configure up the schedule.yml for 15 mins
The biggest issue I found was ensuring it's running post releases etc. In the end I set up God to shutdown and restart
In terms of load. Not sure I follow, the schedulers will trigger events but the load is determined by the number of workers you have and how you decide to implement the creation. You can set the priority of the queues, and workers for the queue, but I guess if you don't process them in a timely way you get a backlog, is that acceptable. ? Normally you would run them of a separate server, this minimising impact to front end
I want to display a random record from the database for a certain amount of time, after that time it gets refreshed to another random record.
How would I go about that in rails?
Right now I'm looking in the directions of cronjobs, also the whenever gem, .. but I'm not 100% sure I really need all that for what seems to be a pretty simple action?
Use the Rails.cache mechanism.
In your controller:
#record = Rails.cache("cached_record", :expires_in => 5.minutes) do
Model.first( :offset =>rand(Model.count))
end
During the first execution, result gets cached in the Rails cache. A new random record is retrieved after 5 minutes.
I would have an expiry_date in my model and then present the user with a javascript timer. After the time has elapsed, i would send a request back to the server(ajax probably, or maybe refreshing the page) and check whether the time has indeed expired. If so, i would present the new record.
You could simply check the current time in your controller, something like:
def show
#last_refresh ||= DateTime.now
#current ||= MyModel.get_random
#current = MyModel.get_random if (DateTime.now - #last_refresh) > 5.minutes
end
This kind of code wouldn't scale to more servers (as it relies on class variables for data storage), so in reality you would wan't to store the two class variables in something like Redis (or Memcache even) - that is for high performance. Depends really on how accurately you need this and how much performance you need. You could as well use your normal database to store expiry times and then load the record whose time is current.
My first though was to cache the record in a global, but you could end up with different records being served by different servers. How about adding a :chosen_at datetime column to your record...
class Model < AR::Base
def self.random
##random = first(:conditions => 'chosen_at NOT NULL')
return ##random unless ##random.nil? or ##random.chosen_at < 5.minutes.ago
##random.update_attribute(:chosen_at,nil) if ##random
ids = connection.select_all("SELECT id FROM things")
##random = find(ids[rand(ids.length)]["id"].to_i)
end
end
Is there any way i can fetch start and time for my every job. I am using delayed_job.
Depending on your setup, you could include actions to record the start time and end time of your job within the job itself.
class SomeJob < Struct.new(:param1, :param2)
def perform
start_time = Time.now
## Do Something
SomeModel.find(id).update_parameters({:start_time => start_time, :end_time => Time.now})
end
end
Might be easier than forking the repository and I am not crazy about the idea of keeping all of those jobs around, it would slow down the queue over time depending on load.
delayed_job has no ability to track the start time, duration or end time of a job. It also by default removes the table entry upon success.
You would have to fork the github version to and create a patch to track and record this information or utilise an external method ( http://helderribeiro.net/?p=87 uses monit ) to track this data (again which uses a forked version).