What's the best way to cache a paginated result set with rails and memcached?
For example, posts controller:
def index
#posts = Rails.cache.fetch('all_posts') do
Post.paginate(:conditions => ['xx = ?', yy], :include => [:author], :page => params[:page], :order => 'created_at DESC')
end
end
This obviously doesn't work when the params[:page] changes. I can change the key to "all_posts_#{params[:page]}_#{params[:order]_#{last_record.created_at.to_i}", but then there could be several possible orders (recent, popular, most voted etc) and there will be a combination of pages and orders ... lots of keys this way.
Problem #2 - It seems that when I implement this solution, the caches get written correctly and the page loads fine during the first call to a paginated action. When I click back on the same page i.e. page1, with recent order, it seems the browser does not even make a call to the server. I don't see any controller action being called in the production log.
I am using passenger, REE, memcached, and rails 2.3.5. Firebug shows no requests being made....
Is there a simples/more graceful way of handling this?
When it comes to caching there is no easy solution. You might cache every variant of the result, and thats ok if you implement auto-expiration of entries. You can't just use all_posts, because this way you will have to expire dozens of keys if posts will get changed.
Every AR model instance has the .cache_key based on updated_at method, which is prefered way, so use this instead of last record. Also don't base your key on last record, because if some post in the middle will get deleted your key wont change. You can use logic like this instead.
class ActiveRecord::Base
def self.newest
order("updated_at DESC").first
end
def self.cache_key
newest.nil? ? "0:0" : "#{newest.cache_key}:#{count}"
end
end
Now you can use Post.cache_key, which will get changed if any post will get changed/deleted or created.
In general I would just cache Post.all and then paginate on this object. You really need to do some profiling to find bottle necks in your application.
Besides, if you want to cache every variant, then do fragment/page caching instead.
If up to you how and where to cache. No one-way here.
As for the second part of the question, there is way to few hints for me to figure an answer. Check if the browser is making a call at all LiveHTTPHeaders, tcpdump, etc.
Related
For instance:
#examples = #user.examples.mostrecent.paginate(page: params[:page])
Where "mostrecent" is defined as:
def self.mostrecent
self.order('created_at DESC')
end
So basically the first call to the database is pull every User's example, and then on top of that, order them by most recent first. It seems like this should be doable, but for some reason I can't get it to work.
There is no defined order scope in the model I'm working with, and other calls to order work just fine. By checking the development.log I can see only the first database pulling example by users is respected. The mostrecent order is never called.
Is there a Rails way of doing this all in one line?
You could use a scope, as in:
scope :by_recent, lambda
{ |since_when| order("created_at") }
I'm probably missing something very simple here, but can't understand what.
I'm trying to cache a simple active record query but every time I touch the cache, it runs the query against the DB again.
Controller Code:
products = Rails.cache.read("search_results")
if products
render :text => products[0].id
else
products = Product.where('name LIKE ?", 'product_name?')
Rails.cache.write("search_results", products)
end
I can see that in my second call I get to the if block and not the else, but any time I'm trying to touch products (like rendering it) I also see an active record call to the DB.
What am I missing?
The line
products = Product.where('name LIKE ?", 'product_name?')
returns an ActiveRecord::Relation, but does not hit the database unless a kicker method is called on it.
While I would still recommend using fetch as mentioned in my comment above, try changing the line to:
products = Product.where('name LIKE ?", 'product_name?').all
which will force the database hit, and save the actual results of the query into the cache, instead of the relation.
How would you cache an ActiveResource model? Preferably in memcached. Right now it's pulling a model from my REST API fine but it pulls dozens of records each time. Would be best to cache them.
I've been playing around with the same thing and I think I've found a pretty simple way to check redis for the cached object first. This will only work when you use the find method, but for my needs, I think this is sufficient.
By overriding find, I can check the checksum of the arguments to see if I already have the response saved in redis. If I do, I can pull the JSON response out of redis and create a new object right there. If I don't, I'll pass the find through to ActiveResource::Base's find and the normal action will happen.
I haven't implemented the saving of the responses into redis with ActiveResource yet, but my plan is to populate those caches elsewhere. This way, normally I can rely on my caches being there, but if they aren't, I can fall back to the API.
class MyResource < ActiveResource::Base
class << self
def find(*arguments)
checksum = Digest::MD5.hexdigest(arguments.md5key)
cached = $redis.get "cache:#{self.element_name}:#{checksum}"
if cached
return self.new JSON.parse(cached)
end
scope = arguments.slice!(0)
options = arguments.slice!(0) || {}
super scope, options
end
end
end
and a little patch so we can get an md5key for our array:
require 'digest/md5'
class Object
def md5key
to_s
end
end
class Array
def md5key
map(&:md5key).join
end
end
class Hash
def md5key
sort.map(&:md5key).join
end
end
Does that help?
Caching in rails is configurable. You can configure the cache to be backed by memcached. Typically you can cache when you retrieve. It's unclear if you are a rest consumer or service but it's really not relevant. If you cache on read (or retrieve) and then read the cache the next time, everything will work just fine. If you are pulling the data from a database, serve up the cache and if no cache is available then cache the read from the database.
I wrote a blog post about it here:
http://squarism.com/2011/08/30/memcached-with-rails-3/
However what I wrote about is really pretty simple. Just showing how to avoid an expensive operation with what is sort of similar to the ||= operator. For a better example, new relic has a scaling rails episode. For example, they show how to cache the latest 10 posts:
def self.recent
Rails.cache.fetch("recent_posts", :expires_in => 30.minutes) do
self.find(:all, :limit => 10)
end
end
Rails.cache has been configured to be a memcached cache, this is the configurable part I was talking about.
I would suggest looking into https://github.com/Ahsizara/cached_resource, almost all of the work is done for you through the gem.
I am implementing caching into my Rails project via Memcached and particularly trying to cache side column blocks (most recent photos, blogs, etc), and currently I have them expiring the cache every 15 minutes or so. Which works, but if I can do it more up-to-date like whenever new content is added, updated or whatnot, that would be better.
I was watching the episode of the Scaling Rails screencasts on Memcached http://content.newrelic.com/railslab/videos/08-ScalingRails-Memcached-fixed.mp4, and at 8:27 in the video, Gregg Pollack talks about intelligent caching in Memcached in a way where intelligent keys (in this example, the updated_at timestamp) are used to replace previously cached items without having to expire the cache. So whenever the timestamp is updated, the cache would refresh as it seeks a new timestamp, I would presume.
I am using my "Recent Photos" sideblock for this example, and this is how it's set up...
_side-column.html.erb:
<div id="photos"">
<p class="header">Photos</p>
<%= render :partial => 'shared/photos', :collection => #recent_photos %>
</div>
_photos.html.erb
<% cache(photos) do %>
<div class="row">
<%= image_tag photos.thumbnail.url(:thumb) %>
<h3><%= link_to photos.title, photos %></h3>
<p><%= photos.photos_count %> Photos</p>
</div>
</div>
<% end %>
On the first run, Memcached caches the block as views/photos/1-20110308040600 and will reload that cached fragment when the page is refreshed, so far so good. Then I add an additional photo to that particular row in the backend and reload, but the photo count is not updated. The log shows that it's still loading from views/photos/1-20110308040600 and not grabbing an updated timestamp. Everything I'm doing appears to be the same as what the video is doing, what am I doing wrong above?
In addition, there is a part two to this question. As you see in the partial above, #recent_photos query is called for the collection (out of a module in my lib folder). However, I noticed that even when the block is cached, this SELECT query is still being called. I attempted to wrap the entire partial in a block at first as <% cache(#recent_photos) do %>, but obviously this doesn't work - especially as there is no real timestamp on the whole collection, just it's individual items of course. How can I prevent this query from being made if the results are cached already?
UPDATE
In reference to the second question, I found that unless Rails.cache.exist? may just be my ticket, but what's tricky is the wildcard nature of using the timestamp...
UPDATE 2
Disregard my first question entirely, I figured out exactly why the cache wasn't refreshing. That's because the updated_at field wasn't being updated. Reason for that is that I was adding/deleting an item that is a nested resource in a parent, and I probably need to implement a "touch" on that in order to update the updated_at field in the parent.
But my second question still stands...the main #recent_photos query is still being called even if the fragment is cached...is there a way using cache.exists? to target a cache that is named something like /views/photos/1-2011random ?
One of the major flaws with Rails caching is that you cannot reliably separate the controller and the view for cached components. The only solution I've found is to embed the query in the cached block directly, but preferably through a helper method.
For instance, you probably have something like this:
class PhotosController < ApplicationController
def index
# ...
#recent_photos = Photos.where(...).all
# ...
end
end
The first instinct would be to only run that query if it will be required by the view, such as testing for the presence of the cached content. Unfortunately there is a small chance that the content will expire in the interval between you testing for it being cached and actually rendering the page, something that will lead to a template rendering error when the nil-value #recent_photos is used.
Here's a simpler approach:
<%= render :partial => 'shared/photos', :collection => recent_photos %>
Instead of using an instance variable, use a helper method. Define your helper method as you would've the load inside the controller:
module PhotosHelper
def recent_photos
#recent_photos ||= Photos.where(...).all
end
end
In this case the value is saved so that multiple calls to the same helper method only triggers the query once. This may not be necessary in your application and can be omitted. All the method is obligated to do is return a list of "recent photos", after all.
A lot of this mess could be eliminated if Rails supported sub-controllers with their own associated views, which is a variation on the pattern employed here.
As I've been working further with caching since asking this question, I think I'm starting to understand exactly the value of this kind of caching technique.
For example, I have an article and through a variety of things I need for the page which include querying other tables, maybe I need to do five-seven different queries per article. However, caching the article in this way reduces all those queries to one.
I am assuming that with this technique, there always needs to have at least "one" query, as there needs to be "some" way to tell whether the timestamp has been updated or not.
Is there a way to do this in Rails:
I have an activerecord query
#posts = Post.find_by_id(10)
Anytime the query is called, SQL is generated and executed at the DB that looks like this
SELECT * FROM 'posts' WHERE id = 10
This happens every time the AR query is executed. Similarly with a helper method like this
<%= f.textarea :name => 'foo' %>
#=> <input type='textarea' name='foo' />
I write some Railsy code that generates some text that is used by some other system (database, web browser). I'm wondering if there's a way to write an AR query or a helper method call that generates the text in the file. This way the text rendering is only done once (each time the code changes) instead of each time the method is called?
Look at the line, it may be going to the database for the first one but ones after it could be saying CACHE at the start of the line meaning it's going to ActiveRecord's query cache.
It also sounds to me like you want to cache the page, not the query. And even if it were the query, I don't think it's as simple as find_by_id(10) :)
Like Radar suggested you should probably look into Rails caching. You can start with something simple like the memory store or file cache and then move to something better like memcached if necessary. You can throw in some caching into the helper method which will cache the result after it is queried once. So for example you can do:
id = 10 # id is probably coming in as a param/argument
cache_key = "Post/#{id}"
#post = Rails.cache.read(cache_key)
if #post.nil?
#post = Post.find_by_id(id)
# Write back to the cache for the next time
Rails.cache.write(cache_key,#post)
end
The only other thing left to do is put in some code to expire the cache entry if the post changes. For that take a look at using "Sweepers" in Rails. Alternatively you can look at some of the caching gems like Cache-fu and Cached-model.
I'm not sure I understand your question fully.
If you're asking about the generated query, you can just do find_by_sql and write your own SQL if you don't want to use the active record dynamic methods.
If you're asking about caching the resulset to a file, it's already in the database, I don't know that if it was in a file it would be much more efficient.