I'm using Mongoid to work with MongoDB in Rails.
What I'm looking for is something like active record include. Currently I failed to find such method in mongoid orm.
Anybody know how to solve this problem in mongoid or perhaps in mongomapper, which is known as another good alternative.
Now that some time has passed, Mongoid has indeed added support for this. See the "Eager Loading" section here:
http://docs.mongodb.org/ecosystem/tutorial/ruby-mongoid-tutorial/#eager-loading
Band.includes(:albums).each do |band|
p band.albums.first.name # Does not hit the database again.
end
I'd like to point out:
Rails' :include does not do a join
SQL and Mongo both need eager loading.
The N+1 problem happens in this type of scenario (query generated inside of loop):
.
<% #posts.each do |post| %>
<% post.comments.each do |comment| %>
<%= comment.title %>
<% end %>
<% end %>
Looks like the link that #amrnt posted was merged into Mongoid.
Update: it's been two years since I posted this answer and things have changed. See tybro0103's answer for details.
Old Answer
Based on the documentation of both drivers, neither of them supports what you're looking for. Probably because it wouldn't solve anything.
The :include functionality of ActiveRecord solves the N+1 problem for SQL databases. By telling ActiveRecord which related tables to include, it can build a single SQL query, by using JOIN statements. This will result in a single database call, regardless of the amount of tables you want to query.
MongoDB only allows you to query a single collection at a time. It doesn't support anything like a JOIN. So even if you could tell Mongoid which other collections it has to include, it would still have to perform a separate query for each additional collection.
Although the other answers are correct, in current versions of Mongoid the includes method is the best way to achieve the desired results. In previous versions where includes was not available I have found a way to get rid of the n+1 issue and thought it was worth mentioning.
In my case it was an n+2 issue.
class Judge
include Mongoid::Document
belongs_to :user
belongs_to :photo
def as_json(options={})
{
id: _id,
photo: photo,
user: user
}
end
end
class User
include Mongoid::Document
has_one :judge
end
class Photo
include Mongoid::Document
has_one :judge
end
controller action:
def index
#judges = Judge.where(:user_id.exists => true)
respond_with #judges
end
This as_json response results in an n+2 query issue from the Judge record. in my case giving the dev server a response time of:
Completed 200 OK in 816ms (Views: 785.2ms)
The key to solving this issue is to load the Users and the Photos in a single query instead of 1 by 1 per Judge.
You can do this utilizing Mongoids IdentityMap Mongoid 2 and Mongoid 3 support this feature.
First turn on the identity map in the mongoid.yml configuration file:
development:
host: localhost
database: awesome_app
identity_map_enabled: true
Now change the controller action to manually load the users and photos. Note: The Mongoid::Relation record will lazily evaluate the query so you must call to_a to actually query the records and have them stored in the IdentityMap.
def index
#judges ||= Awards::Api::Judge.where(:user_id.exists => true)
#users = User.where(:_id.in => #judges.map(&:user_id)).to_a
#photos = Awards::Api::Judges::Photo.where(:_id.in => #judges.map(&:photo_id)).to_a
respond_with #judges
end
This results in only 3 queries total. 1 for the Judges, 1 for the Users and 1 for the Photos.
Completed 200 OK in 559ms (Views: 87.7ms)
How does this work? What's an IdentityMap?
An IdentityMap helps to keep track of what objects or records have already been loaded. So if you fetch the first User record the IdentityMap will store it. Then if you attempt to fetch the same User again Mongoid queries the IdentityMap for the User before it queries the Database again. This will save 1 query on the database.
So by loading all of the Users and Photos we know we are going to want for the Judges json in manual queries we pre-load the data into the IdentityMap all at once. Then when the Judge requires it's User and Photo it checks the IdentityMap and does not need to query the database.
ActiveRecord :include typically doesn't do a full join to populate Ruby objects. It does two calls. First to get the parent object (say a Post) then a second call to pull the related objects (comments that belong to the Post).
Mongoid works essentially the same way for referenced associations.
def Post
references_many :comments
end
def Comment
referenced_in :post
end
In the controller you get the post:
#post = Post.find(params[:id])
In your view you iterate over the comments:
<%- #post.comments.each do |comment| -%>
VIEW CODE
<%- end -%>
Mongoid will find the post in the collection. When you hit the comments iterator it does a single query to get the comments. Mongoid wraps the query in a cursor so it is a true iterator and doesn't overload the memory.
Mongoid lazy loads all queries to allow this behavior by default. The :include tag is unnecessary.
This could help https://github.com/flyerhzm/mongoid-eager-loading
You need update your schema to avoid this N+1 there are no solution in MongoDB to do some jointure.
Embed the detail records/documents in the master record/document.
In my case I didn't have the whole collection but an object of it that caused n+1 (bullet says that).
So rather than writing below which causes n+1
quote.providers.officialname
I wrote
Quote.includes(:provider).find(quote._id).provider.officialname
That didn't cause a problem but left me thinking if I repeated myself or checking n+1 is unnecessary for mongoid.
Related
Let's say I have a simple model association, where a blog Post has many Comments on it.
class Post < ActiveRecord::Base
has_may :comments
end
If I wanted to avoid "N + 1" queries and eager load all the associations beforehand, I could do -
Post.includes(:comments).where(title: "foo")
Which runs two queries. The first one looks up the Post using the where condition and the second looks up all the associated comments at once.
But what if I already have a Post object? Can I "add" an includes to it after the initial result set to run the 2nd bulk query that looks up all the associations?
It seems counter intuitive to do a delayed eager load, but I assume looking up all the associations at once would still save me from having to look them up individually as I loop through them.
e.g.
p = Post.where(title: "foo")
# Will the below work?
p.includes(:comments)
p.comments.each do |comment|
puts comment.text
end
Let's break down your code.
posts = Post.where(title: 'foo')
That searches for all posts with a title of foo. posts is an ActiveRecord::Relation object and you can chain more ActiveRecord commands like select, limit, includes, etc.
So doing posts.includes(:comments) is valid and should eager load the comments. Just don't forget to assign the result to posts again (or another variable).
What will not work is posts.comments because comments is a method that works on an instance of Post. Changing to the following code will work
posts = Post.where(title: 'foo')
posts = posts.includes(:comments)
posts.each do |post|
post.comments.each do |comment|
puts comment.text
end
end
I don't understand the Rails includes method as well as I'd like, and I ran into an issue that I'm hoping to clarify. I have a Board model that has_many :members and has_many :lists (and a list has_many :cards). In the following boards controller, the show method looks as follows:
def show
#board = Board.includes(:members, lists: :cards).find(params[:id])
...
end
Why is the includes method needed here? Why couldn't we just use #board = Board.find(params[:id]) and then access the members and lists via #board.members and #board.lists? I guess I'm not really seeing why we would need to prefetch. It'd be awesome if someone could detail why this is more effective in terms of SQL queries. Thanks!
Per the Rails docs:
Eager loading is the mechanism for loading the associated records of
the objects returned by Model.find using as few queries as possible.
When you simply load a record and later query its different relationships, you have to run a query each time. Take this example, also from the Rails docs:
clients = Client.limit(10)
clients.each do |client|
puts client.address.postcode
end
This code executes a database call 11 times to do something pretty trivial, because it has to do each lookup, one at a time.
Compare the above code to this example:
clients = Client.includes(:address).limit(10)
clients.each do |client|
puts client.address.postcode
end
This code executes a database call 2 times, because all of the necessary associations are included at the onset.
Here's a link to the pertinent section of the Rails docs.
Extra
A point to note as of recent: if you do a more complex query with an associated model like so:
Board.includes(:members, lists: :cards).where('members.color = ?', 'foo').references(:members)
You need to make sure to include the appended references(:used_eager_loaded_class) to complete the query.
In RoR, it is pretty common mistake for new people to load a class and assiocations like this# the solution to eager load
# The bellow generates an insane amount of queries
# post has many comments
# If you have 10 posts with 5 comments each
# this will run 11 queries
posts = Post.find(:all)
posts.each do |post|
post.comments
end
The solution is pretty simple to eager load
# should be 2 queries
# no matter how many posts you have
posts = Post.find(:all, :include => :comments) # runs a query to get all the comments for all the posts
posts.each do |post|
post.comments # runs a query to get the comments for that post
end
But what if you don't have access to the class methods, and only have access to a collection of instance methods.
Then you are stuck with the query intensive lazy loading.
Is there a way to minimize queries to get all the comments for the collection of posts, from the collection of instances?
Addition for Answer (also added to the code above)
So to eager load from what I can see in the rdoc for rails is a class method on any extension of ActiveRecord::Associations, the problem is say you you don't have the ability to use a class method, so you need to use some sort of instance method
a code example of what I think it would look like would be is something like
post = Posts.find(:all)
posts.get_all(:comments) # runs the query to build comments into each post without the class method.
In Rails 3.0 and earlier you can do:
Post.send :preload_associations, posts, :comments
You can pass arrays or hashes of association names like you can to include:
Post.send :preload_associations, posts, :comments => :users
In Rails 3.1 this has been moved and you use the Preloader like this:
ActiveRecord::Associations::Preloader.new(posts, :comments).run()
And since Rails 4 its invocation has changed to:
ActiveRecord::Associations::Preloader.new.preload(posts, :comments)
I think I get what you're asking.
However, I don't think you have to worry about what methods you have access to. The foreign key relationship (and the ActiveRecord associations, such as has_many, belongs_to, etc.) will take care of figuring out how to load the associated records.
If you can provide a specific example of what you think should happen, and actual code that isn't working, it would be easier to see what you're getting at.
How are you obtaining your collection of model instances, and what version of Rails are you using?
Are you saying that you have absolutely no access to either controllers or models themselves?
giving you the best answer depends on knowing those things.
I've got an ActiveAdmin index page
ActiveAdmin.register Bill
And I am trying to display links to associated models
index do
column "User" do |bill|
link_to bill.user.name, admin_user_path(bill.user)
end
end
But I run into the N+1 query problem - there's a query to fetch each user.
Is there a way of eager loading the bills' users?
The way to do this is to override the scoped_collection method (as noted in Jeff Ancel's answer) but call super to retain the existing scope. This way you retain any pagination/filtering which has been applied by ActiveAdmin, rather than starting from scratch.
ActiveAdmin.register Bill do
controller do
def scoped_collection
super.includes :user
end
end
index do
column "User" do |bill|
link_to bill.user.name, admin_user_path(bill.user)
end
end
end
As noted in official documentation at http://activeadmin.info/docs/2-resource-customization.html
There is an answer on a different post, but it describes well what you need to do here.
controller do
def scoped_collection
Bill.includes(:user)
end
end
Here, you will need to make sure you follow scope. So if your controller is scope_to'ed, then you will want to replace the model name above with the scope_to'ed param.
The existing answers were right at the time, but ActiveAdmin supports eager loading with a much more convenient syntax now:
ActiveAdmin.register Bill do
includes :user
end
See the docs for resource customization
IMPORTANT EDIT NOTE : what follows is actually false, see the comments for an explanation. However I leave this answer where it stands because it seems I'm not the only one to get confused by the guides, so maybe someone else will find it useful.
i assume that
class Bill < ActiveRecord::Base
belongs_to :user
end
so according to RoR guides it is already eager-loaded :
There’s no need to use :include for immediate associations – that is,
if you have Order belongs_to :customer, then the customer is
eager-loaded automatically when it’s needed.
you should check your SQL log if it's true (didn't know that myself, i was just verifying something about :include to answer you when i saw this... let me know)
I've found scoped_collection loads all the entries, instead of just the ones for the page you are displaying. I think a better option is apply_collection_decorator that will only preload the items you are effectively displaying.
controller do
def apply_collection_decorator(collection)
collection.includes(:user)
end
end
I have a users object that I'm exposing through the simple call:
#users = User.all
I also have some more expensive queries I'm doing where I'm using custom SQL to generate the results.
#comment_counts = User.received_comment_counts
This call and others like it will return objects with one element per user in my list ordered by id.
In my view I'd like to loop through the users and the comment_counts object at the same time. Eg:
for user, comment_count in #users, #comment_counts do
end
I can't figure out how to do this. I also can't seem to figure out how to get an individual record from the #comment_counts result without running it through #comment_counts.each
Isn't there some way to build an iterator for each of these lits objects, dump them into a while loop and step the iterator individually on each pass of the loop? Also, if you can answer this where can I go to learn about stepping through lists in rails.
Something like this should work for you:
#users.zip(#comment_counts).each do |user, comments_count|
# Do something
end
I've never come across manual manipulation of iterators in Ruby, not to say that it can't be done. Code like that would likely be messy and so I would likely favor more elegant solutions.
Assuming the arrays have equal values:
In your controller:
#users = User.all
#comments = Comment.all
In your view.
<% #users.each_with_index do |user, i |%>
<% debug user %>
<% debug #comments[i] %>
<% end %>
Then you can just check if the values exists on #comments if you don't know if the arrays have the same number of objects.
It's not obvious because your data requires a bit of restructuring. comment_count clearly belongs to instances of User and this should be reflected so that only loop on one collection (of Users) is necessary. I can think of two possible ways to achieve this:
Eager loading. Only works for associated models, not the case here probably. Say, each of your users has comments, so writing User.includes(:comments).all would return all users with their comments and only do 2 queries to fetch the data.
For counting each user's comments you could just use counter_cache (clickable). Then you'll have one more field in the users table for caching how many comments a user has. Again, works for associated Comments only.