I'm having trouble with Rails 3 using conditions on an associated table while eager loading. It appears that Rails is applying the condition when it loads the original model data, so it won't load the parent model unless a non-zero number of the child/associated models match the condition. This is easier to explain in code (simplified for example):
#post = Post.includes(:comments).where(:comments => { :approved => true }).find(1)
This would generate a SQL query similar to:
SELECT DISTINCT `posts`.id FROM `posts`
LEFT OUTER JOIN `comments` ON `comments`.`post_id` = `posts`.`id`
WHERE (`comments`.`approved` = 1) AND (`posts`.`id` = '1')
LIMIT 1
In the case that there aren't any comments that meet the approved = 1 condition, no rows are returned, and thus the Post never gets loaded at all.
What is the right way to load a post and the associated comments eagerly with a condition on the comments?
Update
I'd stil love to hear a better way of doing this, but for now I'm using the following to work around it (works with deeply nested eager loading):
#post = Post.find(1)
#comments = #post.comments.where(:approved => true).all
# allows deeper/more complex nesting without getting into SQL:
#post = Post.includes(:author => [ :websites, :photo ]).find(1)
#comments = #post.comments.includes(:editor).where(:approved => true).all
I guess what you are looking for is joins method, it will let you put your condition within join definition, not outside of it. For example:
#post = Post.joins("LEFT JOIN comments on posts.id = comments.post_id AND comments.approved = 1").first
Not sure about the correctness of the condition itself but you get my point.
Unfortunately you have to use that ugly string as joins is using INNER JOIN if you pass array/hash.
There's more about joins at rails guides
Update: There might be some nugget of wisdom in this post on includes vs eager_load vs preload.
I'd still love to hear a better way of doing this, but for now I'm using the following to work around it (works with deeply nested eager loading, unlike using joins):
#post = Post.find(1)
#comments = #post.comments.where(:approved => true).all
# allows deeper/more complex nesting without getting into SQL:
#post = Post.includes(:author => [ :websites, :photo ]).find(1)
#comments = #post.comments.includes(:editor).where(:approved => true).all
Related
I'm trying to return the another_id for a related record. I would just add a has_many and belongs_to relation for each project, but I need to have the user id in order to return the correct results. However, with the code I have below, it returns all of the possible another_ids for the current_user.
If I enter this into psql, it works fine:
WITH RECURSIVE t(id, parent_id, path) AS (
SELECT thing.id, thing.parent_id, ARRAY[thing.id]
FROM thing, projects
WHERE thing.id = 595
UNION
SELECT i.id, i.parent_id, i.parent_id || t.path
FROM thing i
INNER JOIN t ON t.parent_id = i.id
)
SELECT DISTINCT user_thing.another_id FROM user_thing
INNER JOIN t on t.id = user_thing.thing_id
WHERE t.id = user_thing.thing_id AND user_thing.user_id = 2;
another_id
-----------
52
(1 row)
But if I run the code from the serializer, it returns: [52, 51]:
class ProjectSerializer < ActiveModel::Serializer
attributes :id, :another_id
def another_id__sql
"(WITH RECURSIVE t(id, parent_id, path) AS (
SELECT thing.id, thing.parent_id, ARRAY[thing.id]
FROM thing, projects
WHERE thing.id = projects.thing_id
UNION
SELECT i.id, i.parent_id, i.parent_id || t.path
FROM thing i
INNER JOIN t ON t.parent_id = i.id
)
SELECT DISTINCT user_thing.another_id FROM user_thing
INNER JOIN t on t.id = user_thing.thing_id
WHERE t.id = user_thing.thing_id AND user_thing.user_id = #{options[:current_user].id})"
end
end
class API::V1::ProjectsController < API::APIController
def index
render json: Project.all
end
private
def default_serializer_options
{ current_user: #current_user }
end
end
From what I can gather, I'm not understanding how active_model_serializers serializes more than one record.
I'm using rails 4.2.3 and active_model_serializers 0.8.3. I'm afraid I can't change the schema. Also, it probably doesn't matter, but this is the API for an Ember app.
Thanks in advance. I'm a bit embarrassed that I'm having trouble with this.
Edit:
I should probably mention that this is what my project model looks like:
class Project < ActiveRecord::Base
belongs_to :thing
has_many :user_thing, through: :thing
attr_accessor :another_id
def set_another_id(user)
connection = ActiveRecord::Base.connection
result = connection.execute("(WITH RECURSIVE t(id, parent_id, path) AS (
SELECT thing.id, thing.parent_id, ARRAY[thing.id]
FROM thing, projects
WHERE thing.id = #{thing_id}
UNION
SELECT i.id, i.parent_id, i.parent_id || t.path
FROM thing i
INNER JOIN t ON t.parent_id = i.id
)
SELECT DISTINCT user_thing.another_id FROM user_thing
INNER JOIN t on t.id = user_thing.thing_id
WHERE t.id = user_thing.thing_id AND user_thing.user_id = #{user.id})")
#another_id = result[0]["another_id"].to_i
end
end
And this is the show action in the controller:
def show
#project = Project.find(params[:id])
#project.set_another_id(#current_user)
render json: #project
end
The show action does return the correct id.
Also, I know what I have is incorrect. The thing is that I can't just use the activerecord associations, because it depends on that session's current user.
Edit 2:
I thought I was able to get it to work if I just rendered it using: render json: Project.all.to_json, and got rid of the another_id__sql method in the serializer. That does work if it does have another_id. However, if that's nil, I get the error: "NoMethodError in API::V1::ProjectsController#index undefined method []' for nil:NilClass". It looks like this is a possible bug in 0.8, so I'll either have to ask another Stack Overflow question, or I'll have to see if I can upgrade theactive_model_serializers` gem. I was wrong! See my answer below.
All the DB logic belongs in your model, not in your serializer. The serializers simply state what is supposed to be exposed, but it should not be responsible for computing it.
So here, I'd advise to make this another_id a method on your model, which won't solve your issue (as it seems it is more of an SQL issue than anything else), but it will make it so that you don't have a problem with AMS anymore.
Serializers take a record and return a serialized representation suitable for JSON or XML encoding.
They are meant as an alternative to littering your controllers with this:
render json: #users, except: [:foo, :bar, :baz], include: [..........]
And the mental flatulence that is jbuilder.
SQL queries and scopes instead belong in your models.
You can set the serializer by using the each_serializer option. But in this case it will not do you much good the objects you serialize must at least implement the base methods for a serializable model.
So you need to re-write your query so that it returns a collection or array of records.
see:
http://apidock.com/rails/ActiveRecord/Base/find_by_sql/class
https://github.com/rails-api/active_model_serializers/blob/master/lib/active_model_serializers/model.rb
https://github.com/rails-api/active_model_serializers
Got it! It appears that I needed one more method in the serializer:
project_serializer.rb
def another_id
object.another_id
end
def another_id__sql
# sql from before
end
I'm not 100% sure why this works, but I had noticed that, if I left out the another_id__sql, I would get the error column.projects.another_id does not exist. So, I'm guessing that the another_id__sql is called when it's returning an array, but uses the another_id method when the object is a single project record.
I'd still love to hear better ways to do this!
I'm trying to optimise some N+1 queries in active record for the first time. There are 3 to kill - 2 went very easily with a .includes call, but I can't for the life of me figure out why the third is still calling a bunch of queries. Relevant code below - if anyone has any suggestions, I'd be really appreciative.
CONTROLLER:
#enquiries = Comment.includes(:children).faqs_for_project(#project)
MODEL;
def self.faqs_for_project(project)
Comment.for_project_and_enquiries(project, project.enquiries).where(:published => true).order("created_at DESC")
end
(and the relevant scope)
scope :for_project_and_enquiries, lambda{|p, qs| where('(commentable_type = ? and commentable_id = ?) or (commentable_type = ? and commentable_id IN (?))', "Project", p.id, "Enquiry", qs.collect{|q| q.id})}
VIEW:
...
= render :partial => 'comments/comment', :collection => #enquries
...
(and that offending line in the partial)
...
= 'Read by ' + pluralize(comment.acknowledgers.count, 'lead')
...
Two SQL queries are called for each comment. The 2 queries are:
SQL (2.8ms) SELECT COUNT(*) FROM "users" INNER JOIN "acknowledgements" ON "users".id = "acknowledgements".user_id WHERE (("acknowledgements".feedback_type = 'Comment') AND ("acknowledgements".feedback_id = 177621))
CACHE (0.0ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1295 LIMIT 1
I would have thought appending (:user, :acknowledgements) into the controller's .includes would have solved the problem, but it doesn't seem to have any effect. If anyone has any suggestions on what I'm missing, I'd be really appreciative
I believe in your Comment table you want to add a :acknowledgers_count column as a counter cache
has_many :acknowledgers, ....., counter_cache: true
You will need to create a migration to add the :acknowledgers_count column to the comments table. Rails should take care of the rest.
You can learn more about the ActiveRecord::CounterCache api here.
The count method in comment.acknowledgers.count is overloaded in ActiveRecord to first check if a counter cache column exists, and if it does, it returns that directly from the model (in this case the Comment model) without having to touch the database again.
Finally, there was very recently a great Railscast about a gem call Bullet that can help you identify these query issues and guide you toward a solution. It covers both counter caches and N+1 queries.
As #ismaelga pointed out in a comment to this answer, it's a generally better practice to call .size instead of .count on a relation. Check out the source for size:
def size
loaded? ? #records.length : count
end
If the relation is already loaded it will just call length on it, otherwise it will call count. It's an extra check to try and prevent the database from unnecessarily being queried.
I've written a couple of complex queries (at least to me) with Ruby on Rail's query interface:
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id})
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id})
Both of these queries work fine by themselves. Both return Post objects. I would like to combine these posts into a single ActiveRelation. Since there could be hundreds of thousands of posts at some point, this needs to be done at the database level. If it were a MySQL query, I could simply user the UNION operator. Does anybody know if I can do something similar with RoR's query interface?
Here's a quick little module I wrote that allows you to UNION multiple scopes. It also returns the results as an instance of ActiveRecord::Relation.
module ActiveRecord::UnionScope
def self.included(base)
base.send :extend, ClassMethods
end
module ClassMethods
def union_scope(*scopes)
id_column = "#{table_name}.id"
sub_query = scopes.map { |s| s.select(id_column).to_sql }.join(" UNION ")
where "#{id_column} IN (#{sub_query})"
end
end
end
Here's the gist: https://gist.github.com/tlowrimore/5162327
Edit:
As requested, here's an example of how UnionScope works:
class Property < ActiveRecord::Base
include ActiveRecord::UnionScope
# some silly, contrived scopes
scope :active_nearby, -> { where(active: true).where('distance <= 25') }
scope :inactive_distant, -> { where(active: false).where('distance >= 200') }
# A union of the aforementioned scopes
scope :active_near_and_inactive_distant, -> { union_scope(active_nearby, inactive_distant) }
end
I also have encountered this problem, and now my go-to strategy is to generate SQL (by hand or using to_sql on an existing scope) and then stick it in the from clause. I can't guarantee it's any more efficient than your accepted method, but it's relatively easy on the eyes and gives you a normal ARel object back.
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id})
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id})
Post.from("(#{watched_news_posts.to_sql} UNION #{watched_topic_posts.to_sql}) AS posts")
You can do this with two different models as well, but you need to make sure they both "look the same" inside the UNION -- you can use select on both queries to make sure they will produce the same columns.
topics = Topic.select('user_id AS author_id, description AS body, created_at')
comments = Comment.select('author_id, body, created_at')
Comment.from("(#{comments.to_sql} UNION #{topics.to_sql}) AS comments")
Based on Olives' answer, I did come up with another solution to this problem. It feels a little bit like a hack, but it returns an instance of ActiveRelation, which is what I was after in the first place.
Post.where('posts.id IN
(
SELECT post_topic_relationships.post_id FROM post_topic_relationships
INNER JOIN "watched" ON "watched"."watched_item_id" = "post_topic_relationships"."topic_id" AND "watched"."watched_item_type" = "Topic" WHERE "watched"."user_id" = ?
)
OR posts.id IN
(
SELECT "posts"."id" FROM "posts" INNER JOIN "news" ON "news"."id" = "posts"."news_id"
INNER JOIN "watched" ON "watched"."watched_item_id" = "news"."id" AND "watched"."watched_item_type" = "News" WHERE "watched"."user_id" = ?
)', id, id)
I'd still appreciate it if anybody has any suggestions to optimize this or improve the performance, because it's essentially executing three queries and feels a little redundant.
You could also use Brian Hempel's active_record_union gem that extends ActiveRecord with an union method for scopes.
Your query would be like this:
Post.joins(:news => :watched).
where(:watched => {:user_id => id}).
union(Post.joins(:post_topic_relationships => {:topic => :watched}
.where(:watched => {:user_id => id}))
Hopefully this will be eventually merged into ActiveRecord some day.
Could you use an OR instead of a UNION?
Then you could do something like:
Post.joins(:news => :watched, :post_topic_relationships => {:topic => :watched})
.where("watched.user_id = :id OR topic_watched.user_id = :id", :id => id)
(Since you are joins the watched table twice I'm not too sure what the names of the tables will be for the query)
Since there are a lot of joins, it might also be quite heavy on the database, but it might be able to be optimized.
How about...
def union(scope1, scope2)
ids = scope1.pluck(:id) + scope2.pluck(:id)
where(id: ids.uniq)
end
Arguably, this improves readability, but not necessarily performance:
def my_posts
Post.where <<-SQL, self.id, self.id
posts.id IN
(SELECT post_topic_relationships.post_id FROM post_topic_relationships
INNER JOIN watched ON watched.watched_item_id = post_topic_relationships.topic_id
AND watched.watched_item_type = "Topic"
AND watched.user_id = ?
UNION
SELECT posts.id FROM posts
INNER JOIN news ON news.id = posts.news_id
INNER JOIN watched ON watched.watched_item_id = news.id
AND watched.watched_item_type = "News"
AND watched.user_id = ?)
SQL
end
This method returns an ActiveRecord::Relation, so you could call it like this:
my_posts.order("watched_item_type, post.id DESC")
There is an active_record_union gem.
Might be helpful
https://github.com/brianhempel/active_record_union
With ActiveRecordUnion, we can do:
the current user's (draft) posts and all published posts from anyone
current_user.posts.union(Post.published)
Which is equivalent to the following SQL:
SELECT "posts".* FROM (
SELECT "posts".* FROM "posts" WHERE "posts"."user_id" = 1
UNION
SELECT "posts".* FROM "posts" WHERE (published_at < '2014-07-19 16:04:21.918366')
) posts
In a similar case I summed two arrays and used Kaminari:paginate_array(). Very nice and working solution. I was unable to use where(), because I need to sum two results with different order() on the same table.
Heres how I joined SQL queries using UNION on my own ruby on rails application.
You can use the below as inspiration on your own code.
class Preference < ApplicationRecord
scope :for, ->(object) { where(preferenceable: object) }
end
Below is the UNION where i joined the scopes together.
def zone_preferences
zone = Zone.find params[:zone_id]
zone_sql = Preference.for(zone).to_sql
region_sql = Preference.for(zone.region).to_sql
operator_sql = Preference.for(Operator.current).to_sql
Preference.from("(#{zone_sql} UNION #{region_sql} UNION #{operator_sql}) AS preferences")
end
Less problems and easier to follow:
def union_scope(*scopes)
scopes[1..-1].inject(where(id: scopes.first)) { |all, scope| all.or(where(id: scope)) }
end
So in the end:
union_scope(watched_news_posts, watched_topic_posts)
gem 'active_record_extended'
Also has a set of union helpers among many others.
I would just run the two queries you need and combine the arrays of records that are returned:
#posts = watched_news_posts + watched_topics_posts
Or, at the least test it out. Do you think the array combination in ruby will be far too slow? Looking at the suggested queries to get around the problem, I'm not convinced that there will be that significant of a performance difference.
Elliot Nelson answered good, except the case where some of the relations are empty. I would do something like that:
def union_2_relations(relation1,relation2)
sql = ""
if relation1.any? && relation2.any?
sql = "(#{relation1.to_sql}) UNION (#{relation2.to_sql}) as #{relation1.klass.table_name}"
elsif relation1.any?
sql = relation1.to_sql
elsif relation2.any?
sql = relation2.to_sql
end
relation1.klass.from(sql)
end
When we add UNION to the scopes, it breaks at time due to order_by clause added before the UNION.
So I changed it in a way to give it a UNION effect.
module UnionScope
def self.included(base)
base.send(:extend, ClassMethods)
end
module ClassMethods
def union_scope(*scopes)
id_column = "#{table_name}.id"
sub_query = scopes.map { |s| s.pluck(:id) }.flatten
where("#{id_column} IN (?)", sub_query)
end
end
end
And then use it like this in any model
class Model
include UnionScope
scope :union_of_scopeA_scopeB, -> { union_scope(scopeA, scopeB) }
end
Tim's answer is great. It uses the ids of the scopes in the WHERE clause. As shosti reports, this method is problematic in terms of performance because all ids need to be generated during query execution. This is why, I prefer joeyk16 answer. Here a generalized module:
module ActiveRecord::UnionScope
def self.included(base)
base.send :extend, ClassMethods
end
module ClassMethods
def self.union(*scopes)
self.from("(#{scopes.map(&:to_sql).join(' UNION ')}) AS #{self.table_name}")
end
end
end
If you don't want to use SQL syntax inside your code, here's solution with arel
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id}).arel
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id}).arel
results = Arel::Nodes::Union.new(watched_news_posts, watched_topic_posts)
from(Post.arel_table.create_table_alias(results, :posts))
I need to delete a lot of records at once and I need to do so based on a condition in another model that is related by a "belongs_to" relationship. I know I can loop through each checking for the condition, but this takes forever with my large record set because for each "belongs_to" it makes a separate query.
Here is an example. I have a "Product" model that "belongs_to" an "Artist" and lets say that artist has a property "is_disabled".
If I want to delete all products that belong to disabled artists, I would like to be able to do something like:
Product.delete_all(:joins => :artist, :conditions => ["artists.is_disabled = ?", true])
Is this possible? I have done this directly in SQL before, but not sure if it is possible to do through rails.
The problem is that delete_all discards all the join information (and rightly so). What you want to do is capture that as an inner select.
If you're using Rails 3 you can create a scope that will give you what you want:
class Product < ActiveRecord::Base
scope :with_disabled_artist, lambda {
where("product_id IN (#{select("product_id").joins(:artist).where("artist.is_disabled = TRUE").to_sql})")
}
end
You query call then becomes
Product.with_disabled_artist.delete_all
You can also use the same query inline but that's not very elegant (or self-documenting):
Product.where("product_id IN (#{Product.select("product_id").joins(:artist).where("artist.is_disabled = TRUE").to_sql})").delete_all
In Rails 4 (I tested on 4.2) you can almost do how OP originally wanted
Application.joins(:vacancy).where(vacancies: {status: 'draft'}).delete_all
will give
DELETE FROM `applications` WHERE `applications`.`id` IN (SELECT id FROM (SELECT `applications`.`id` FROM `applications` INNER JOIN `vacancies` ON `vacancies`.`id` = `applications`.`vacancy_id` WHERE `vacancies`.`status` = 'draft') __active_record_temp)
If you are using Rails 2 you can't do the above. An alternative is to use a joins clause in a find method and call delete on each item.
TellerLocationWidget.find(:all, :joins => [:widget, :teller_location],
:conditions => {:widgets => {:alt_id => params['alt_id']},
:retailer_locations => {:id => #teller_location.id}}).each do |loc|
loc.delete
end
I'm looking for help with Ruby optimization regarding loading of associations on demand.
This is simplified example. I have 3 models: Post, Comment, User. References are: Post has many comments and Comment has reference to User (:author). Now when I go to the post page, I expect to see post body + all comments (and their respective authors names). This requires following 2 queries:
select * from Post -- to get post data (1 row)
select * from Comment inner join User -- to get comment + usernames (N rows)
In the code I have:
Post.find(params[:id], :include => { :comments => [:author] }
But it doesn't work as expected: as I see in the back end, there're still N+1 hits (some of them are cached though). How can I optimize that?
UPD
After some investigation, it looks like code was correct, but it doesn't work as expected in case I have named belongs_to in a Comment model. Once I changed from :author to :user, it worked as expected.
In my project I have a similar relationship to your Post, Comment, and User models. I only see three actual sql queries.
Post.find(1, :include => { :comments => [:author] })
From the debug log it shows these three queries
SELECT * FROM `posts` WHERE (`posts`.`id` = 1)
SELECT `comments`.* FROM `comments` WHERE (`comments`.`post_id` = 1)
SELECT * FROM `authors` WHERE (`authors`.`id` IN (4,8,15,16,23,42))
If you are happy with 2/3 queries, you can try:
#post = Post.find params[:id]
#comments = Comments.find_by_post_id(params[:id], :include => [:author])
or
#comments = #post.comments(:include => [:author])
Edit: Have you tried with:
Post.find(params[:id], :include => { :comments => :author }