Eager loading does not work as I expect.
I have Products that has_many Variants, and of course each Variant belongs_to a Product.
I use code like this to load a product and all its variants:
products = Product.includes(:variants)
This works: all products and all variants are loaded with only two queries. However, the product of each variant is not loaded, so the following code causes another SQL-query:
puts products[0].variants[0].product.title
Why is that, and how can I fix it? I suppose Product.includes(:variants => :product) would work, but it causes one extra big and unnecessary SQL-query, since the Product-data is already available.
Active Record will only eager loading an association on the level you've specified. In its point of view, variant.product will be treated as another level of association. So, if you want to eager loading it, you'd have to do:
products = Product.includes({:variants => :product})
Related
For this setup with default (unspecified) search_data:
class Item < ActiveRecord::Base
searchkick
has_many :quantities, dependent: :destroy
scope :search_import, -> { includes(:quantities) }
end
When importing all database records by running Item.reindex, each "batch" eager loads the quantities for all of the item ids as expected.
However, if I want to specify the index model document differently than the default attributes using the search_data method including the associated model data with something like this:
class Item < ActiveRecord::Base
searchkick
has_many :quantities, dependent: :destroy
def search_data
{
part_number: standard_part_number,
category_id: category_id.to_i,
content_source_name: content_source_name,
standard_price: standard_price.to_i,
locations: quantities.map {|q| [q.location_code,
quantity: q.value.to_i,
bins: q.location_bins]
}.to_h
}
end
scope :search_import, -> { includes(:quantities) }
end
where I am operating on the quantities using map to define a locations attribute, returning to import using Item.reindex I see that it not only eager loads all of the associated quantities each batch, but it also then loads all quantities per item with a hit to the database again for each item record it is indexing.
Am I missing something here to get Searchkick to eager load the quantities models and be able to customize the search data record using that already loaded associated model without it doing another pull from the database again per item?
Update
I've determined there is something interfering with the way the ActiveRecord normally responds to the method name used for the association with the eager loaded models in our app and may not be exclusively related to us using Searchkick, although it did reveal the issue. I am not sure what it is that is interfering at this time but it has something to do with the target method on the association. Perhaps a gem that is installed is causing this problem. I did manage to find a way to work around this (for now) using item.association(:quantities).target as a replacement for item.quantities and now when running the reindex it makes use of the already eager loaded quantities and doesn't hit the db again for each item
I see that it not only eager loads all of the associated quantities each batch
This is expected behaviour (and very likely performant), since each batch you will get different quantities to load, since they are connecting to different items, so you don't need to keep all quantities in memory.
each batch, but it also then loads all quantities per item with a hit to the database again for each item record it is indexing.
This is actually unexpected, but my guess here is, that in one of the methods in Quantity (#location_code #value or #location_bins) or even one of the methods on Item that you call (#standard_part_number, #category_id, #content_source_name, #standard_price), there is some implementation, that requires reloading of records.
Without knowing the code of that methods it is purely speculative, but the presented part of the code looks fine.
Some of my classes :
class User
embeds_many :notifications
field :first_name
field :last_name
def name{ "#{first_name} #{last_name}" }
class Notification
embedded_in :user
belongs_to :sender, class_name: "User", inverse_of: nil
Now in my views, I implemented a small mailbox system for notifications. However, it's currently hitting N+1 times the database :
<% current_user.notifications.sort{...}.each do |notif|%>
...
<%= notif.sender.name if notif.sender %>
The problem here is the notif.sender.name which causes N hits on the database. Can I somehow preload/eager load this ? Something like current_user.notifications.includes(:sender) (but which would work :D)
I currently only need the sender name.
I think you're half out of luck here. Mongoid has an error message like:
Eager loading in Mongoid only supports providing arguments to M.includes that are the names of relations on the M model, and only supports one level of eager loading. (ie, eager loading associations not on the M but one step away via another relation is not allowed).
Note the last parenthesized sentence in particular:
eager loading associations not on the M but one step away via another relation is not allowed
Embedding is a relation but you want to apply includes to the embedded relation and that's one step too far for Mongoid.
The fine manual does say that:
This will work for embedded relations that reference another collection via belongs_to as well.
but that means that you'd call includes on the embedded relation rather than what the models are embedded in. In your case, that means that you could eager load the senders for each set of embedded Notifications:
current_user.notifications.includes(:sender).sort { ... }
That still leaves you with the N+1 problem that eager loading is supposed to get around but your N will be smaller.
If that's still too heavy then you could denormalize the name into each embedded document (i.e. copy it rather than referencing it through the sender). Of course, you'd need to maintain the copies if people were allowed to change their names.
It's not perfect, but this article presents a possible solution.
You can load all the senders and use set_relation to avoid them to be loaded every time.
def notifications_with_senders
sender_ids = notifications.map(:sender_id)
senders = User.in(id: sender_ids).index_by(&:id)
notifications.each do |notification|
notification.set_relation(:sender, senders[notification.sender_id])
end
end
Would be great to have that as a Relation method (like includes of Rails Active Record)
I have two model with has_many belongs to relation.
Scheme has_many navs
I need to fetch all the Schemes with only last nav value. I have 10 Schemes and each scheme has around 100k navs but I need only last record which is the current value.
With eager loading will load all the navs
Scheme.all.includes(:navs)
How can I apply condition to to get only last row of nav for each schemes while eager loading.
UPDATE with Log
If I run
Scheme.includes(:current_nav).limit(3)
these are the queries executed by AR
SELECT `schemes`.* FROM `schemes` LIMIT 3
SELECT `navs`.* FROM `navs` WHERE `navs`.`schemeCode` IN ('D04', 'D01', 'D30') ORDER BY id DESC
How the second query works, it will take all the navs whose schemeCode falls under list and order those by id DESC , but how it will be associated with particular scheme exactly.
How about creating an another association like this:
class Scheme < ActiveRecord::Base
has_one :current_nav, -> { order('id DESC').limit(1) }, class_name: 'Nav'
end
Now you can:
Schema.includes(:current_nav).all
or:
Schema.includes(:current_nav).last(10)
will eager load only last nav of the queried schemes.
Explanation: includes is one of the methods for retrieving objects from database in ActiveRecord. From the doc itself:
Active Record lets you specify in advance all the associations that
are going to be loaded. This is possible by specifying the includes
method of the Model.find call. With includes, Active Record ensures
that all of the specified associations are loaded using the minimum
possible number of queries.
And, since we have the association setup with current_nav, all we had to do is to use it with includes to eager load the data. Please read ActiveRecord querying doc for more information.
I have a query like this,
company.users.select("users.id, users.state").includes(:organization)
here I'm eager loading the association organization. I was expecting the attributes id and user_id to be fetched in the objects, but then I get all fields fetched.
Is this the way, rails behaves when we eager load or am I missing something here ?
In your case you will get all company users not organizations.
Eager loading means pre-loading the database rows. It will not fetch only attributes. It loads all rows associated.
For example:
comments = Comment.all(:select => "users.name,comment_text", :include => :user)
Here, it will not just load names from user table. It will get all users rows from the database. So you don't have to fire extra queries. And one more thing is when you use include select clause is ignored when you have attributes of included tables. For more info go through ryan bates rialscast on joins vs include : http://railscasts.com/episodes/181-include-vs-joins
Does anyone know a way to determine if a Rails association has been eager loaded?
My situation: I have a result set where sometimes one of the associations is eager loaded, and sometimes it isn't. If it isn't eager-loaded, then I want to look up associations using ActiveRecord's find. If it is eager loaded, I want to use detect.
For example, say that I have a "has_many" array of shipping_info objects in my item model. Then:
If item is eager loaded, most efficient load is:
item.shipping_infos.detect { |si| si.region == "United States" }
If item isn't eager loaded, most efficient load is:
item.shipping_infos.where(region: "United States").first
But unless I know whether it is eager loaded, I don't know which code to call to get the record efficiently. If I use the first method when it wasn't eager loaded, then I have to look up more DB records than necessary. And if I use the second method when it was eager loaded, then my eager loaded objects are ignored.
Use .association(name).loaded? on a record.
For Rails < 3.1 use loaded_foo?.
(It is deprecated since Rails 3.1. See: https://github.com/rails/rails/issues/472.)
item.shipping_infos.loaded? will tell you.
I gotta say, though: this path leads to madness... before writing code that tests loaded? to decide between #detect and #find, make sure this instance really matters, relative to everything else that's going on.
If this isn't the slowest thing your app does, adding extra code paths adds unnecessary complexity. Just because you might waste a little database effort doesn't mean you need to fix it - it probably doesn't matter in any measurable way.
I'd suggest using item.association_cache.keys that will provide a list of the eager loaded associations. So you item.association_cache.keys.include?(:name_of_association)
association_cached? might be a good fit:
item.association_cached?(:shipping_infos)
You can detect whether or not a single association has been loaded with loaded_foo?. For example, if shipping_info was a belongs_to association, then item.loaded_shipping_info? will return true when it's been eager-loaded. Oddly, it appears to return nil (rather than false) when it hasn't been loaded (in Rails 2.3.10 anyway).
Solution to this problem should be foo.association(:bla).loaded?, BUT it works incorrectly - it checks and marks association as dirty:
class Foo; has_one :bla, :autosave => true end
foo.association(:bla).loaded? #=> false
foo.save # saves foo and fires select * from bla
So I've added following extension to ActiveRecord:
module ActiveRecord
class Base
def association_loaded?(name)
association_instance_get(name).present?
end
end
end
and now:
class Foo; has_one :bla, :autosave => true end
foo.association_loaded?(:bla) #=> false
foo.save # saves foo
Have a look at the Bullet gem.. This will tell you when you should and should not use eager loading.