Rails eager loading seems to be querying wrong - ruby-on-rails

I'm attempting to eager load in my Rails 3 app. I've narrowed it down to a very basic sample, and instead of generating the one query I'm expecting, it's generating 4.
First, here's a simple breakdown of my models.
class Profile < ActiveRecord::Base
belongs_to :gender
def to_param
self.name
end
end
class Gender < ActiveRecord::Base
has_many :profiles, :dependent => :nullify
end
I then has a ProfilesController::show action, where's I'm querying for the model.
def ProfilesController < ApplicationController
before_filter :find_profile, :only => [:show]
def show
end
private
def find_profile
#profile = Profile.find_by_username(params[:id], :include => :gender)
raise ActiveRecord::RecordNotFound, "Page not found" unless #profile
end
end
When I look at the queries this generates, it shows the following:
SELECT `profiles`.* FROM `profiles` WHERE `profiles`.`username` = 'matt' LIMIT 1
SELECT `genders`.* FROM `genders` WHERE (`genders`.`id` = 1)
What I expected to see is a single query:
SELECT `profiles`.*, `genders`.* FROM `profiles` LEFT JOIN `genders` ON `profiles`.gender_id = `genders`.id WHERE `profiles`.`username` = 'matt' LIMIT 1
Anyone know what I'm doing wrong here? Everything I've found on eager loading makes it sound like this should work.
Edit: After trying joins, as recommended by sled, I'm still seeing the same results.
The code:
#profile = Profile.joins(:gender).where(:username => params[:id]).limit(1).first
The query:
SELECT `profiles`.* FROM `profiles` INNER JOIN `genders` ON `genders`.`id` = `profiles`.`gender_id` WHERE `profiles`.`username` = 'matt' LIMIT 1
Again, you can see no genders data is being retrieved, and so a second query to genders is being made.
I even tried adding a select, to no avail:
#profile = Profile.joins(:gender).select('profiles.*, genders.*').where(:username => params[:id]).limit(1).first
which correctly resulted in:
SELECT profiles.*, genders.* FROM `profiles` INNER JOIN `genders` ON `genders`.`id` = `profiles`.`gender_id` WHERE `profiles`.`username` = 'matt' LIMIT 1
...but it still performed a second query on genders later when accessing #profile.gender's attributes.
Edit 2: I also tried creating a scope that includes both select and joins in order to get all the fields I require, (similar to the custom left join method sled demonstrated). It looks like this:
class Profile < ActiveRecord::Base
# ...
ALL_ATTRIBUTES = [:photo, :city, :gender, :relationship_status, :physique, :children,
:diet, :drink, :smoke, :drug, :education, :income, :job, :politic, :religion, :zodiac]
scope :with_attributes,
select((ALL_ATTRIBUTES.collect { |a| "`#{reflect_on_association(a).table_name}`.*" } + ["`#{table_name}`.*"]).join(', ')).
joins(ALL_ATTRIBUTES.collect { |a|
assoc = reflect_on_association(a)
"LEFT JOIN `#{assoc.table_name}` ON `#{table_name}`.#{assoc.primary_key_name} = `#{assoc.table_name}`.#{assoc.active_record_primary_key}"
}.join(' '))
# ...
end
This generates the following query, which appears correct:
SELECT `photos`.*, `cities`.*, `profile_genders`.*, `profile_relationship_statuses`.*, `profile_physiques`.*, `profile_children`.*, `profile_diets`.*, `profile_drinks`.*, `profile_smokes`.*, `profile_drugs`.*, `profile_educations`.*, `profile_incomes`.*, `profile_jobs`.*, `profile_politics`.*, `profile_religions`.*, `profile_zodiacs`.*, `profiles`.* FROM `profiles` LEFT JOIN `photos` ON `profiles`.photo_id = `photos`.id LEFT JOIN `cities` ON `profiles`.city_id = `cities`.id LEFT JOIN `profile_genders` ON `profiles`.gender_id = `profile_genders`.id LEFT JOIN `profile_relationship_statuses` ON `profiles`.relationship_status_id = `profile_relationship_statuses`.id LEFT JOIN `profile_physiques` ON `profiles`.physique_id = `profile_physiques`.id LEFT JOIN `profile_children` ON `profiles`.children_id = `profile_children`.id LEFT JOIN `profile_diets` ON `profiles`.diet_id = `profile_diets`.id LEFT JOIN `profile_drinks` ON `profiles`.drink_id = `profile_drinks`.id LEFT JOIN `profile_smokes` ON `profiles`.smoke_id = `profile_smokes`.id LEFT JOIN `profile_drugs` ON `profiles`.drug_id = `profile_drugs`.id LEFT JOIN `profile_educations` ON `profiles`.education_id = `profile_educations`.id LEFT JOIN `profile_incomes` ON `profiles`.income_id = `profile_incomes`.id LEFT JOIN `profile_jobs` ON `profiles`.job_id = `profile_jobs`.id LEFT JOIN `profile_politics` ON `profiles`.politic_id = `profile_politics`.id LEFT JOIN `profile_religions` ON `profiles`.religion_id = `profile_religions`.id LEFT JOIN `profile_zodiacs` ON `profiles`.zodiac_id = `profile_zodiacs`.id WHERE `profiles`.`username` = 'matt' LIMIT 1
Unfortunately, it doesn't seem that calls to relationship attributes (e.g.: #profile.gender.name) are using the data that was returned in the original SELECT. Instead, I see a flood of queries following this first one:
Profile::Gender Load (0.2ms) SELECT `profile_genders`.* FROM `profile_genders` WHERE `profile_genders`.`id` = 1 LIMIT 1
Profile::Gender Load (0.4ms) SELECT `profile_genders`.* FROM `profile_genders` INNER JOIN `profile_attractions` ON `profile_genders`.id = `profile_attractions`.gender_id WHERE ((`profile_attractions`.profile_id = 2))
City Load (0.4ms) SELECT `cities`.* FROM `cities` WHERE `cities`.`id` = 1 LIMIT 1
Country Load (0.3ms) SELECT `countries`.* FROM `countries` WHERE `countries`.`id` = 228 ORDER BY FIELD(code, 'US') DESC, name ASC LIMIT 1
Profile Load (0.4ms) SELECT `profiles`.* FROM `profiles` WHERE `profiles`.`id` = 2 LIMIT 1
Profile::Language Load (0.4ms) SELECT `profile_languages`.* FROM `profile_languages` INNER JOIN `profile_profiles_languages` ON `profile_languages`.id = `profile_profiles_languages`.language_id WHERE ((`profile_profiles_languages`.profile_id = 2))
SQL (0.3ms) SELECT COUNT(*) FROM `profile_ethnicities` INNER JOIN `profile_profiles_ethnicities` ON `profile_ethnicities`.id = `profile_profiles_ethnicities`.ethnicity_id WHERE ((`profile_profiles_ethnicities`.profile_id = 2))
Profile::Religion Load (0.5ms) SELECT `profile_religions`.* FROM `profile_religions` WHERE `profile_religions`.`id` = 2 LIMIT 1
Profile::Politic Load (0.2ms) SELECT `profile_politics`.* FROM `profile_politics` WHERE `profile_politics`.`id` = 3 LIMIT 1

your example is fine and it will end up in two queries because that's how eager loading is implemented in rails. It becomes handy if you have many associated records. You can read more about it here
What you probably want is a simple join:
#profile = Profile.joins(:gender).where(:username => params[:id])
Edit
If the profile consists of many pieces there are multiple approaches here:
Custom left joins - maybe there is a plugin out there which does the job otherwise I'd suggest to do something like:
class Profile < ActiveRecord::Base
# .... code .....
def self.with_dependencies
attr_joins = []
attr_selects = []
attr_selects << "`profiles`.*"
attr_selects << "`genders`.*"
attr_selects << "`colors`.*"
attr_joins << "LEFT JOIN `genders` ON `gender`.`id` = `profiles`.gender_id"
attr_joins << "LEFT JOIN `colors` ON `colors`.`id` = `profiles`.color_id"
prep_model = select(attr_selects.join(','))
attr_joins.each do |c_join|
prep_model = prep_model.joins(c_join)
end
return prep_model
end
end
Now you could do something like:
#profile = Profile.with_dependencies.where(:username => params[:id])
Another solution is to use the :include => [:gender, :color] it may be some queries more but it's the cleaner "rails way". If you run into performance issues you may want to rethink your DB Schema but do you have really such a heavy load?
A friend of mine wrote a nice little solution for this simple 1:n relations (like genders) it's called simple_enum

After working with sled's suggestions, I finally came up with this solution. I'm sure it could be made cleaner with a plugin, but here's what I've got for now:
class Profile < ActiveRecord::Base
ALL_ATTRIBUTES = [:photo, :city, :gender, :relationship_status, :physique, :children,
:diet, :drink, :smoke, :drug, :education, :income, :job, :politic, :religion, :zodiac]
scope :with_attributes,
includes(ALL_ATTRIBUTES).
select((ALL_ATTRIBUTES.collect { |a| "`#{reflect_on_association(a).table_name}`.*" } + ["`#{table_name}`.*"]).join(', '))
end
The two main points are:
A call to includes, which passes the symbols of the relationships I want
A call to select that makes sure to retrieve all columns for the related tables. Note that I call reflect_on_association so that I don't have to hard-code the related tables' names, letting the Rails models do the work for me.
I can now call:
Profile.with_attributes.where(:username => params[:id]).limit(1).first
Going to mark sled's answer as correct since it's his help (answers + comments combined) that led me here, even though this is the code I'm ultimately using.

Related

Write and Activerecord join query

I'm writing an activerecord join query but It doesn't work.
I have these two classes
class User
belongs_to :store, required: true
end
class Store < ActiveRecord::Base
has_many :users, dependent: :nullify
has_one :manager, -> { where role: User.roles[:manager] }, class_name: 'User'
end
I need to get all the stores with a manager and all the stores without a manager.
I write these two queries
Store.includes(:users).where('users.role <> ?', User.roles[:manager]).references(:users).count
Store.includes(:users).where('users.role = ?', User.roles[:manager]).references(:users).count
and the result is
2.2.1 :294 > Store.includes(:users).where('users.role <> ?', User.roles[:manager]).references(:users).count
(6.6ms) SELECT COUNT(DISTINCT "stores"."id") FROM "stores" LEFT OUTER JOIN "users" ON "users"."store_id" = "stores"."id" WHERE (users.role <> 1)
=> 201
2.2.1 :295 > Store.includes(:users).where('users.role = ?', User.roles[:manager]).references(:users).count
(4.0ms) SELECT COUNT(DISTINCT "stores"."id") FROM "stores" LEFT OUTER JOIN "users" ON "users"."store_id" = "stores"."id" WHERE (users.role = 1)
=> 217
Now I know that I have 219 stores, and using
with_manager = 0
without_manager = 0
Store.all.each do |s|
if s.manager.present?
with_manager = with_manager +1
else
without_manager = without_manager +1
end
end
I know also that I have 217 stores with manager and 2 store without manager. One query is working, the second (stores without manager) fails.
So I must fix the query, but I cannot understand how can I fix it...
Usually I use the following thing for, getting al stores with a manager:
Store.joins(:manager)
Joins will use a inner join and not a left join that is includes case.
In the opposite case, that's tricky, but I do in this way:
Store.includes(:manager).where(users: { id: nil })
It's a left join with manager and getting all stores without a user included.

How to eager load child model's sum value for ruby on rails?

I have an Order model, it has many items, it looks like this
class Order < ActiveRecord::Base
has_many :items
def total
items.sum('price * quantity')
end
end
And I have an order index view, querying order table like this
def index
#orders = Order.includes(:items)
end
Then, in the view, I access total of order, as a result, you will see tons of SUM query like this
SELECT SUM(price * quantity) FROM "items" WHERE "items"."order_id" = $1 [["order_id", 1]]
SELECT SUM(price * quantity) FROM "items" WHERE "items"."order_id" = $1 [["order_id", 2]]
SELECT SUM(price * quantity) FROM "items" WHERE "items"."order_id" = $1 [["order_id", 3]]
...
It's pretty slow to load order.total one by one, I wonder how can I load the sum in a eager manner via single query, but still I can access order.total just like before.
Try this:
subquery = Order.joins(:items).select('orders.id, sum(items.price * items.quantity) AS total').group('orders.id')
#orders = Order.includes(:items).joins("INNER JOIN (#{subquery.to_sql}) totals ON totals.id = orders.id")
This will create a subquery that sums the total of the orders, and then you join that subquery to your other query.
I wrote up two options for this in this blog post on using find_by_sql or joins to solve this.
For your example above, using find_by_sql you could write something like this:
Order.find_by_sql("select
orders.id,
SUM(items.price * items.quantity) as total
from orders
join items
on orders.id = items.order_id
group by
order.id")
Using joins, you could rewrite as:
Order.all.select("order.id, SUM(items.price * items.quantity) as total").joins(:items).group("order.id")
Include all the fields you want in your select list in both the select clause and the group by clause. Hope that helps!

Rails + UUID + Eager loading

I have 2 models in my rails app, one with an UUID primary key :
class User < ActiveRecord::Base
belongs_to :country, :foreign_key => 'country_uuid'
end
class Country < ActiveRecord::Base
set_primary_key :uuid
has_many :users
end
When I try something like that:
<% #user = User.find :first, :include => [:country] %>
<%= #user.country.name %>
I have the good result, but I see 2 requests in the log file. Why eager loading is not working when we change the ID key for UUID key ?
User Load (0.4ms) SELECT `users`.* FROM `users` LIMIT 1
Country Load (0.4ms) SELECT `countries`.* FROM `countries` WHERE (`countries`.`uuid` = '1')
And I would have something like:
User Load (0.4ms) SELECT `users`.* FROM `users` INNER JOIN countries ON countries.uuid = users.country_uuid LIMIT 1
Is there a workaround ?
If I change uuid key for id key, but keep the string format to store an uuid, will it be ok ?
Thanks,
Use joins instead of include to get the inner join
includes always issues a 2nd query but not n+1 queries (lazy)
for the direction you are going in user -> 1 country it is not so important
but if you were going the other direction country -> many users
country = Country.first
# => select countries.* from countries where id = xxxx limit 1;
country.users.each do
# select users.* from users where user_id = xxxx;
# this could be bad because of lazy loading, one query per iteration
end
# vs...
country = Country.first.includes(:users)
# => select countries.* from countries where id = xxxx limit 1;
# => select users.* from users where country_uuid IN (xxxx);
country.users.each do
# users are all in memory
end
see http://guides.rubyonrails.org/active_record_querying.html for more info
I don't think the fact you are using UUID should make any difference

How do I write a UNION chain with ActiveRelation?

I need to be able to chain an arbitrary number of sub-selects with UNION using ActiveRelation.
I'm a little confused by the ARel implementation of this, since it seems to assume UNION is a binary operation.
However:
( select_statement_a ) UNION ( select_statement_b ) UNION ( select_statement_c )
is valid SQL. Is this possible without doing nasty string-substitution?
You can do a bit better than what Adam Lassek has proposed though he is on the right track. I've just solved a similar problem trying to get a friends list from a social network model. Friends can be aquired automatically in various ways but I would like to have an ActiveRelation friendly query method that can handle further chaining. So I have
class User
has_many :events_as_owner, :class_name => "Event", :inverse_of => :owner, :foreign_key => :owner_id, :dependent => :destroy
has_many :events_as_guest, :through => :invitations, :source => :event
def friends
friends_as_guests = User.joins{events_as_guest}.where{events_as_guest.owner_id==my{id}}
friends_as_hosts = User.joins{events_as_owner}.joins{invitations}.where{invitations.user_id==my{id}}
User.where do
(id.in friends_as_guests.select{id}
) |
(id.in friends_as_hosts.select{id}
)
end
end
end
which takes advantage of Squeels subquery support. Generated SQL is
SELECT "users".*
FROM "users"
WHERE (( "users"."id" IN (SELECT "users"."id"
FROM "users"
INNER JOIN "invitations"
ON "invitations"."user_id" = "users"."id"
INNER JOIN "events"
ON "events"."id" = "invitations"."event_id"
WHERE "events"."owner_id" = 87)
OR "users"."id" IN (SELECT "users"."id"
FROM "users"
INNER JOIN "events"
ON "events"."owner_id" = "users"."id"
INNER JOIN "invitations"
ON "invitations"."user_id" =
"users"."id"
WHERE "invitations"."user_id" = 87) ))
An alternative pattern where you need a variable number of components is demonstrated with a slight modification to the above code
def friends
friends_as_guests = User.joins{events_as_guest}.where{events_as_guest.owner_id==my{id}}
friends_as_hosts = User.joins{events_as_owner}.joins{invitations}.where{invitations.user_id==my{id}}
components = [friends_as_guests, friends_as_hosts]
User.where do
components = components.map { |c| id.in c.select{id} }
components.inject do |s, i|
s | i
end
end
end
And here is a rough guess as to the solution for the OP's exact question
class Shift < ActiveRecord::Base
def self.limit_per_day(options = {})
options[:start] ||= Date.today
options[:stop] ||= Date.today.next_month
options[:per_day] ||= 5
queries = (options[:start]..options[:stop]).map do |day|
where{|s| s.scheduled_start >= day}.
where{|s| s.scheduled_start < day.tomorrow}.
limit(options[:per_day])
end
where do
queries.map { |c| id.in c.select{id} }.inject do |s, i|
s | i
end
end
end
end
Because of the way the ARel visitor was generating the unions, I kept getting SQL errors while using Arel::Nodes::Union. Looks like old-fashioned string interpolation was the only way to get this working.
I have a Shift model, and I want to get a collection of shifts for a given date range, limited to five shifts per day. This is a class method on the Shift model:
def limit_per_day(options = {})
options[:start] ||= Date.today
options[:stop] ||= Date.today.next_month
options[:per_day] ||= 5
queries = (options[:start]..options[:stop]).map do |day|
select{id}.
where{|s| s.scheduled_start >= day}.
where{|s| s.scheduled_start < day.tomorrow}.
limit(options[:per_day])
end.map{|q| "( #{ q.to_sql } )" }
where %{"shifts"."id" in ( #{queries.join(' UNION ')} )}
end
(I am using Squeel in addition to ActiveRecord)
Having to resort to string-interpolation is annoying, but at least the user-provided parameters are being sanitized correctly. I would of course appreciate suggestions to make this cleaner.
I like Squeel. But don't use it. So I came to this solution (Arel 4.0.2)
def build_union(left, right)
if right.length > 1
Arel::Nodes::UnionAll.new(left, build_union(right[0], right[1..-1]))
else
Arel::Nodes::UnionAll.new(left, right[0])
end
end
managers = [select_manager_1, select_manager_2, select_manager_3]
build_union(managers[0], managers[1..-1]).to_sql
# => ( (SELECT table1.* from table1)
# UNION ALL
# ( (SELECT table2.* from table2)
# UNION ALL
# (SELECT table3.* from table3) ) )
There's a way to make this work using arel:
tc=TestColumn.arel_table
return TestColumn.where(tc[:id]
.in(TestColumn.select(:id)
.where(:attr1=>true)
.union(TestColumn.select(:id)
.select(:id)
.where(:attr2=>true))))

rails named_scope ignores eager loading

Two models (Rails 2.3.8):
User; username & disabled properties; User has_one :profile
Profile; full_name & hidden properties
I am trying to create a named_scope that eliminate the disabled=1 and hidden=1 User-Profiles. The User model is usually used in conjunction with the Profile model, so I attempt to eager-load the Profile model (:include => :profile).
I created a named_scope on the User model called 'visible':
named_scope :visible, {
:joins => "INNER JOIN profiles ON users.id=profiles.user_id",
:conditions => ["users.disabled = ? AND profiles.hidden = ?", false, false]
}
I've noticed that when I use the named_scope in a query, the eager-loading instruction is ignored.
Variation 1 - User model only:
# UserController
#users = User.find(:all)
# User's Index view
<% for user in #users %>
<p><%= user.username %></p>
<% end %>
# generates a single query:
SELECT * FROM `users`
Variation 2 - use Profile model in view; lazy load Profile model
# UserController
#users = User.find(:all)
# User's Index view
<% for user in #users %>
<p><%= user.username %></p>
<p><%= user.profile.full_name %></p>
<% end %>
# generates multiple queries:
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 1) ORDER BY full_name ASC LIMIT 1
SHOW FIELDS FROM `profiles`
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 2) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 3) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 4) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 5) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 6) ORDER BY full_name ASC LIMIT 1
Variation 3 - eager load Profile model
# UserController
#users = User.find(:all, :include => :profile)
#view; no changes
# two queries
SELECT * FROM `users`
SELECT `profiles`.* FROM `profiles` WHERE (`profiles`.user_id IN (1,2,3,4,5,6))
Variation 4 - use name_scope, including eager-loading instruction
#UserConroller
#users = User.visible(:include => :profile)
#view; no changes
# generates multiple queries
SELECT `users`.* FROM `users` INNER JOIN profiles ON users.id=profiles.user_id WHERE (users.disabled = 0 AND profiles.hidden = 0)
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 1) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 2) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 3) ORDER BY full_name ASC LIMIT 1
SELECT * FROM `profiles` WHERE (`profiles`.user_id = 4) ORDER BY full_name ASC LIMIT 1
Variation 4 does return the correct number of records, but also appears to be ignoring the eager-loading instruction.
Is this an issue with cross-model named scopes? Perhaps I'm not using it correctly.
Is this sort of situation handled better by Rails 3?
From railsapi.com:
Eager loading of associations
[...] Since only one table is loaded
at a time, conditions or orders
cannot reference tables other than the
main one. If this is the case Active
Record falls back to the previously
used LEFT OUTER JOIN based strategy.
For example
Post.find(:all, :include => [ :author, :comments ],
:conditions => ['comments.approved = ?', true])
will result in a single SQL query with
joins along the lines of: LEFT OUTER
JOIN comments ON comments.post_id =
posts.id and LEFT OUTER JOIN authors
ON authors.id = posts.author_id.
I believe this answers your question... there's no eager loading in "variation #4" because you references profiles table on your named_scope.
I believe the following may give you what you are looking for:
#users = User.visible.scoped(:include => :profile)
This did the trick for me, but I'm not joining with other tables in the definition of my named scope.
Jim Benton provides an elegant way of adding this to ActiveRecord on his blog: http://autonomousmachine.com/posts/2009/10/28/add-a-scope-for-easier-eager-loading

Resources