So I've got a forum thread full of posts, and I want to eager load the users from the posts and those users roles, to reduce the total number of database queries (which seems to be the best way to do things)
(to make things extra fun, forum and posts are nested resources, if that makes a differnece)
def show
#forum = Forum.find(params[:forum_id])
#posts = #forum.posts.includes(:user => :role).where('id = ? OR parent_post_id =?',params[:id], params[:id])
end
The page renders, but I can't really tell by looking at the console if it's actually reducing the number of queries or not. The console appears to have a slight reduction in number of queries to the CACHE, but the Load calls holds steady. Am I doing this right?
yes, I think so, if the eager loading is successful, the query count will be reduced.
If you are using MYsql, I suggest you use "query_reviewer". ( https://github.com/dsboulder/query_reviewer) it will show your how many queries were executed for opening a page.
Related
Let's say we have these two outputs in a view:
#post.user.first_name
current_user.posts.size
If the outputs above would be called multiple times each in a single view, is Rails "smart enough" to not hit the database every time?
If the answer is yes - are there any "general rule" about this worth knowing?
If the answer is no - would a good practice then be to store the associated object/objects in it's own variable?
ActiveRecord by default caches queries for performance. If you do AC query in console a few times you will see that the second query executes much faster due to AC cache. So I guess this works for queries in the view as well.
You can manually cache objects with Rails Fragment Caching feature.
Fragment Caching allows a fragment of view logic to be wrapped in a
cache block and served out of the cache store when the next request
comes in.
Also there is Cache Stores for you to use.
Rails provides different stores for the cached data (apart from SQL
and page caching).
Queries for a view are done in the controller action that renders the view. You will notice that you define #post in your controller action but you may not see current_user defined. This is typically because you're using the devise gem and the code defining the current_user method is part of the gem.
Everything you need to render a view should be queried in your controller by ActiveRecord and in memory of the app in preparation of the rendering of the view. So multiple calls of #post or current_user shouldn't matter.
Sometimes objects are called via associations for a view eg. #post.user.name Is going to have to query for the user. This will work but it is better Model-View-Controller separation to eager load the users along with the posts in the controller. Following MVC and making sure your queries happen in the controller will also help you avoid N + 1 query performance issues. See Rails Active Record Query Eager Load
An example of querying users with their posts.
#post = Post.find(post_params).includes(:user)
I have a RoR application which contains an API to manage applications, each of which contain recipes (and groups, ingredients, measurements).
Once the user has finished managing the recipes, they download a JSON file of the entire application. Because each application could have hundreds of recipes, the files can be large. It also means there is a lot of DB calls to get all the required data to export.
Now because of this, the request to download the application can take upwards of 30 seconds, sometimes more.
My current code looks something like this:
application.categories.each do |c|
c.recipes.each do |r|
r.groups.each do |r|
r.ingredients.each do |r|
Within each loop I'm storing the data in a HASH and then giving it to the user.
My question is: where do I go from here?
Is there a way to grab all the data I require from the DB in one query? From looking at the log, I can see it is running hundreds of queries.
If the above solution is still slow, is this something I should put into a background process, and then email the user a link (or similar)?
There are of course ways to grab more data at once. This is done with Rails includes or joins, depending on your needs. See this article for some detailed information.
The basic idea is that you can join between your tables so that each time new queries aren't generated. When you do application.categories, that's one query. For each of those categories, you'll do another query: c.recipes - this creates N+1 queries, where N is the number of categories you have. Rather, you can include them off the get go to create 1 or 2 queries (depending on what Rails does).
The basic syntax is easy:
Application.includes(:categories => :recipes).each do |application| ...
This generates 1 (or 2 - again, see article) query that grabs all applications, their categories, and each categories recipies all at once. You can tack on the groups and ingredients too.
As for putting the work in the background, my suggestion would be to just have a loading image, or get fancy by using a progress bar.
First of all I have to assume that the required has_many and belongs_to associations exist.
Generally you can do something like
c.recipes.includes(:groups)
or even
c.recipes.includes(:groups => :ingredients)
which will fetch recipes and groups (and ingredients) at once.
But since you have a quite big data set IMO it would be better if you limited that technique to the deepest levels.
The most usefull approach would be to use find_each and includes together.
(find_each fetches the items in batches in order to keep the memory usage low)
perhaps something like
application.categories.each do |c|
c.recipes.find_each do |r|
r.groups.includes(:ingredients).each do |r|
r.ingredients.each do |r|
...
end
end
end
end
Now even that can take quite a long time (for an http request) so you can consider using some async processing where the client will generate a request that is going to be processed by the server as a background job, and when that is ready, you can provide a download link (or send an email) to the client.
Resque is one possible solution for handling the async part.
In my user model I have a friends method that returns the hash of all the user's facebook friends. In my view I iterate through the entire list to paginate that list alphabetically. I can't tell if my server is running really slow or if this is extremely inefficient. How can I make this process faster? Is it better to maybe create a friendsmodel? Please let me know if my method is inefficient, why, and how I might be able to make it faster. Thanks!
In my Home.html.erb view I have <%letter ='a'%> which changes when the user selects a different letter and the page refreshes.
<% current_user.friends.each do |user| %>
<% if user['name'].downcase.start_with? letter %>
do something
<% end %>
User Model
def facebook
#facebook ||= Koala::Facebook::API.new(token)
block_given? ? yield(#facebook) : #facebook
rescue Koala::Facebook::APIError => e
logger.info e.to_s
nil
end
def friends
facebook {|fb| fb.get_connections("me","friends")}.sort{|a,b| a['name']<=>b['name']}
end
You are making an external API call for every request. Plus user may have good number of friends like 500, 1000.
I in my fb app processing the data in background job(delayed job). You can use resque or sidekiq or some other background to process user data.
I would suggest you to make Friend model and have its association with users model. Then if you have some n+1 query problem you can use includes and instead of using sort use order it would be much faster then sort. Moreover instead of using each use find_each it will process the data in chunks you can google the difference between each and find_each. hope it would be helpful
One thing that will be slowing down each request for sure is the fact that your making an external API call in the middle of the request. The second thing to note is that your potentially bringing back a large amount of data, easily getting into the hundreds, if not thousands.
A more appropriate way to handle this would be to create a Friend model where each friend has a belongs to relationship to the User. In a background processor (ie delayed job, resque, sidekiq), iterate through your users and update their friends at some interval that your server can tolerate. It will cause some lag as to when the user's friends will show up. You'll have to be the judge as to how much lag time is tolerable, and it depends largely on your number of users and budget for hardware.
This is effectively a caching mechanism, and you may want to account for the fact that data will change, friends may be removed and so on. You could delete all the friends and recreate the whole list on each refresh. Doing so inside a transaction will keep the deletes from showing up until it is commited.
I'm hoping to get advice on the proper use of caching to speed up a timeline query in Rails. Here's the background:
I'm developing an iPhone app with a Rails backend. It's a social app, and like other social apps, its primary view is a timeline (i.e., newsfeed) of messages. This works very much like Twitter, where the timeline is made up of messages of the user and of his/her followers. The main query in the API request to retrieve the timeline is the following:
#messages = Message.where("user_id in (?) OR user_id = ?", current_user.followed_users.map(&:id), current_user)
Now this query gets quite inefficient, particularly at scale, so I'm looking into caching. Here are the two things I'm planning to do:
1) Use Redis to cache timelines as lists of message ids
Part of what makes this query so expensive is figuring out which messages to display on-the-fly. My plan here is to keep create a Redis list of message ids for each user. Assuming I build this correctly when a Timeline API request comes in I can call Redis to get a pre-processed ordered list of the ids of the messages to display. For example, I might get something like this: "[21, 18, 15, 14, 8, 5]"
2) Use Memcached to cache individual message objects
While I believe the first point will help a great deal, there's still the potential problem of retrieving the individual message objects from the database. The message objects can get quite big. With them, I return related objects like comments, likes, the user, etc. Ideally, I would cache these individual message objects as well. This is where I'm confused.
Without caching, I would simply make a query call like this to retrieve the message objects:
#messages = Message.where("id in (?)", ids_from_redis)
Then I would return the timeline:
respond_with(:messages => #messages.as_json) # includes related likes, comments, user, etc.
Now given my desire to utilize Memcache to retrieve individual message objects, it seems like I need to retrieve the messages one at a time. Using psuedo-code I'm thinking something like this:
ids_from_redis.each do |m|
message = Rails.cache.fetch("message_#{m}") do
Message.find(m).as_json
end
#messages << message
end
Here are my two specific questions (sorry for the lengthy build):
1) Does this approach generally make sense (redis for lists, memcached for objects)?
2) Specifically, on the pseudo-code below, is this the only way to do this? It feels inefficient grabbing the messages one-by-one but I'm not sure how else to do it given my intention to do object-level caching.
Appreciate any feedback as this is my first time attempting something like this.
On the face of it, this seems reasonable. Redis is well suited to storing lists etc, can be made persistent etc, and memcached will be very fast for retrieving individual messages, even if you call it sequentially like that.
The issue here is that you're going to need to clear/supplement that redis cache each time a message is posted. It seems a bit of a waste just to clear the cache in this circumstance, because you'll already have gone to the trouble of identifying every recipient of the message.
So, without wishing to answer the wrong question, have you thought about 'rendering' the visibility of messages into the database (or redis, for that matter) when each message is posted? Something like this:
class Message
belongs_to :sender
has_many :visibilities
before_create :render_visibility
sender.followers.each do |follower|
visibilities.build(:user => follower)
end
def
end
You could then render the list of messages quite simply:
class User
has_many :visibilities
has_many :messages, :through => :visibilities
end
# in your timeline view:
<%= current_user.messages.each { |message| render message } %>
I would then add of individual messages like this:
# In your message partial, caching individual rendered messages:
<%= cache(message) do %>
<!-- render your message here -->
<% end %>
I would also then add caching of entire timelines like this:
# In your timeline view
<%= cache("timeline-for-#{current_user}-#{current_user.messages.last.cache_key}") do %>
<%= current_user.messages.each { |message| render message } %>
<% end %>
What this should achieve (I've not tested it) is that the entire timeline HTML will be cached until a new message is posted. When that happens, the timeline will be re-rendered, but all the individual messages will come from the cache rather than being rendered again (with the possible exception of any new ones that haven't been viewed by anyone else!)
Note that this assumes that the message rendering is the same for every user. If it isn't, you'll need to cache the messages per user too, which would be a bit of a shame, so try not to do this if you can!
FWIW, I believe this is vaguely (and I mean vaguely) what twitter do. They have a 'big data' approach to it though, where the tweets are exploded and inserted into follower timelines across a large cluster of machines. What I've described here will struggle to scale in a write-heavy environment with lots of followers, although you could improve this somewhat by using resque or similar.
P.S. I've been a bit lazy with the code here - you should look to refactor this to move e.g. the timeline cache key generation into a helper and/or the person model.
Introduction
I have a (mostly) single-page application built with BackboneJS and a Rails backend.
Because most of the interaction happens on one page of the webapp, when the user first visits the page I basically have to pull a ton of information out of the database in one large deeply joined query.
This is causing me some rather extreme load times on this one page.
NewRelic appears to be telling me that most of my problems are because of 457 individual fast method calls.
Now I've done all the eager loading I can do (I checked with the Bullet gem) and I still have a problem.
These method calls are most likely ocurring in my Rabl serializer which I use to serialize a bunch of JSON to embed into the page for initializing Backbone. You don't need to understand all this but suffice to say it could add up to 457 method calls.
object #search
attributes :id, :name, :subscription_limit
# NOTE: Include a list of the members of this search.
child :searchers => :searchers do
attributes :id, :name, :gravatar_icon
end
# Each search has many concepts (there could be over 100 of them).
child :concepts do |search|
attributes :id, :title, :search_id, :created_at
# The person who suggested each concept.
child :suggester => :suggester do
attributes :id, :name, :gravatar_icon
end
# Each concept has many suggestions (approx. 4 each).
node :suggestions do |concept|
# Here I'm scoping suggestions to only ones which meet certain conditions.
partial "suggestions/show", object: concept.active_suggestions
end
# Add a boolean flag to tell if the concept is a favourite or not.
node :favourite_id do |concept|
# Another method call which occurs for each concept.
concept.favourite_id_for(current_user)
end
end
# Each search has subscriptions to certain services (approx. 4).
child :service_subscriptions do
# This contains a few attributes and 2 fairly innocuous method calls.
extends "service_subscriptions/show"
end
So it seems that I need to do something about this but I'm not sure what approach to take. Here is a list of potential ideas I have:
Performance Improvement Ideas
Dumb-Down the Interface
Maybe I can come up with ways to present information to the user which don't require the actual data to be present. I don't see why I should absolutely need to do this though, other single-page apps such as Trello have incredibly complicated interfaces.
Concept Pagination
If I paginate concepts it will reduct the amount of data being extracted from the database each time. Would product an inferior user interface though.
Caching
At the moment, refreshing the page just extracts the entire search out of the DB again. Perhaps I can cache parts of the app to reduce on DB hits. This seems messy though because not much of the data I'm dealing with is static.
Multiple Requests
It is technically bad to serve the page without embedding the JSON into the page but perhaps the user will feel like things are happening faster if I load the page unpopulated and then fetch the data.
Indexes
I should make sure that I have indexes on all my foreign keys. I should also try to think about places where it would help to have indexes (such as favourites?) and add them.
Move Method Calls into DB
Perhaps I can cache some of the results of the iteration I do in my view layer into the DB and just pull them out instead of computing them. Or I could sync things on write rather than on read.
Question
Does anyone have any suggestions as to what I should be spending my time on?
This is a hard question to answer without being able to see the actual user interface, but I would focus on loading exactly only as much data as is required to display the initial interface. For example, if the user has to drill down to see some of the data you're presenting, then you can load that data on demand, rather than loading it as part of the initial payload. You mention that a search can have as many as 100 "concepts," maybe you don't need to fetch all of those concepts initially?
Bottom line, it doesn't sound like your issue is really on the client side -- it sounds like your server-side code is slowing things down, so I'd explore what you can do to fetch less data, or to defer the complex queries until they are definitely required.
I'd recommend separating your JS code-base into modules that are dynamically loaded using an asset loader like RequireJS. This way you won't have so many XHRs firing at load time.
When a specific module is needed it can load and initialize at an appropriate time instead of every page load.
If you complicate your code a little, each module should be able to start and stop. So, if you have any polling occurring or complex code executing you can stop the module to increase performance and decrease the network load.