Yii eager loading — Fatal error: Out of memory - memory

I have a problem with Yii eager loading.
I open user profile page and use:
$model=User::model()->with('routes', 'likes', 'comments', 'questions', 'cityname')->findByPk($id);
Relations is:
public function relations()
{
return array(
'routes'=>array(self::HAS_MANY, 'Route', 'author_id', 'order'=>'routes.id DESC'),
'questions'=>array(self::HAS_MANY, 'Question', 'author_id', 'order'=>'questions.id DESC'),
'comments'=>array(self::HAS_MANY, 'Comment', 'author_id', 'order'=>'comments.id DESC',),
'likes'=>array(self::HAS_MANY, 'Like', 'author_id', 'order'=>'likes.id DESC'),
'cityname'=>array(self::BELONGS_TO, 'City', 'city'),
);
}
When i have around 70 (or more) comments in Comment table, i have error:
Fatal error: Out of memory (allocated 348651520) (tried to allocate 78 bytes) in /home/milk/kolyasya.ru/diplomyii/framework/db/CDbCommand.php on line 516
The interesting part of this problem, is if i comment any element of with(), for example:
$model=User::model()->with('routes', 'likes', 'comments', /* 'questions' */, 'cityname')->findByPk($id);
then all works as it should.
I checked all relations in all models and set ini_set('memory_limit', '512M'), but i can't find a source of the problem.
Maybe I need to use lazy loading?

You're suffering from exploding number of row combinations. Take a look at this question, it describes the same problem on a smaller scale. Basically, you are running a huge query with multiple one-to-many joins, similar to this:
SELECT ... FROM `User` `t`
LEFT JOIN `Route` routes ON t.id = routes.author_id
LEFT JOIN `Question` questions ON t.id = questions.author_id
LEFT JOIN `Comment` comments ON t.id = comments.author_id
LEFT JOIN `Like` likes ON t.id = likes.author_id
LEFT JOIN `City` city ON t.city = city.id
WHERE t.id = :id
ORDER BY routes.id DESC, questions.id DESC, comments.id DESC, likes.id DESC
You can take this query, modify it to SELECT COUNT(*) and run it in phpMyAdmin to see how many rows it returns. It will be equal to the number of routes multiplied by the number of questions multiplied by the number of comments multiplied by the number of likes created by this user.
In this situation, it would be a lot more efficient to fetch each HAS_MANY relation in a separate query. Yii can do that:
$model=User::model()
->with(array(
'routes' => array('together' => false),
'likes' => array('together' => false),
'comments' => array('together' => false),
'questions' => array('together' => false),
'cityname' => array(),
))
->findByPk($id);
If you do so, Yii will instead produce multiple SQL queries with less memory usage, similar to the following:
SELECT ... FROM `User` `t`
LEFT JOIN `City` `city` ON `t`.`city` = `city`.`id`
WHERE `t`.`id` = :id;
SELECT ... FROM `Route` `routes`
WHERE `author_id` = :id
ORDER by `routes`.`id` DESC;
SELECT ... FROM `Question` `questions`
WHERE `author_id` = :id
ORDER BY `questions`.`id` DESC;
SELECT ... FROM `Comment` `comments`
WHERE `author_id` = :id
ORDER BY `comments`.`id` DESC;
SELECT ... FROM `Like` `likes`
WHERE `author_id` = :id
ORDER BY `likes`.`id` DESC;
The results will be aggregated and returned to your code just like before.

Related

Complex :includes for Eager Loading

I'm trying to create a pretty complex eager load; I’d like to modify the second statement, not the first. I need to create a join on the second statement that includes a column from another table.
Everything I try modifies the first statement and leaves the second as is. If there’s another way to accomplish the same task without N+1 queries, I'm open.
This:
Conversation.joins(:phones)
.where('phones.id' => 2)
.order('last_message_at DESC')
.includes(:messages)
Generates:
SELECT "conversations".* FROM "conversations"
INNER JOIN "conversations_phones"
ON "conversations_phones"."conversation_id" = "conversations"."id"
INNER JOIN "phones"
ON "phones"."id" = "conversations_phones"."phone_id"
WHERE "phones"."id" = 2 ORDER BY last_message_at DESC
SELECT "messages".* FROM "messages"
WHERE "messages"."conversation_id" IN (10, 11) ORDER BY created_at ASC
Makes sense, but not where I want to be.
I can write the needed second statement with something like this:
Message.joins(:message_tags)
.select('messages.*, message_tags.status as read')
.group('messages.id, message_tags.status')
.order('messages.id')
.where(:message_tags => { :user_id => current_user.id })
.where(:messages => { :conversation_id => [10, 11] })
Which correctly generates:
SELECT messages.*, message_tags.status as read FROM "messages"
INNER JOIN "message_tags" ON "message_tags"."message_id" = "messages"."id"
WHERE "message_tags"."user_id" = 2 AND "messages"."conversation_id" IN (10, 11)
GROUP BY messages.id, message_tags.status
ORDER BY messages.id
Basically, I want the more complex messages select to replace the simpler one so I can call #conversations.first.messages.first.read without creating a new query.
Sounds like you need to add some conditions to an association:
has_many :messages, -> { select('messages.*, message_tags.status as read')) }
and then something like:
Conversation.joins(:phones)
.where('phones.id' => 2)
.order('last_message_at DESC')
.includes(:messages => :message_tags)
.where(:message_tags => { :user_id => current_user.id })

Problem with named_scope causes error in will_paginate - how to include group_by in count?

[rails 2.3.12] named_scope:
named_scope :order_by_price, lambda {{:joins => :variants, :group => "products.id", :order => "MAX(price)"}}
console:
1.
> Product.order_by_price.size
=> 21
2.
> p = Product.order_by_price
> p.size
=> 4
sql queries:
1.
SELECT count(*) AS count_all FROM `products` INNER JOIN `variants` ON variants.product_id = products.id
2.
SELECT `products`.* FROM `products` INNER JOIN `variants` ON variants.product_id = products.id GROUP BY products.id ORDER BY MAX(price)
I use will_paginate for pagination. In this case total_entries value is 21 and number of pages is based on this, although there are only 4 products...
Any ideas how can I get this to work correctly?
EDIT
In general I have to include group_by when calling Product.count... how?
No answers, but I found a solution. Maybe it will be useful for someone else too. I just had to redefine count, selecting distinct product_id:
def self.count(*args)
super(args, {:select => "(products.id)", :distinct => true})
end

ActiveRecord Count to count rows returned by group by in Rails

I looked around and couldn't find any answers to this. All answers involved counts that did not use a GROUP BY.
Background:
I have a paginator that will take options for an ActiveRecord.find. It adds a :limit and :offset option and performs the query. What I also need to do is count the total number of records (less the limit), but sometimes the query contains a :group option and ActiveRecord.count tries to return all rows returned by the GROUP BY along with each of their counts. I'm doing this in Rails 2.3.5.
What I want is for ActiveRecord.count to return the number of rows returned by the GROUP BY.
Here is some sample code that demonstrates one instance of this (used for finding all tags and ordering them by the number of posts with that tag):
options = { :select => 'tags.*, COUNT(*) AS post_count',
:joins => 'INNER JOIN posts_tags', #Join table for 'posts' and 'tags'
:group => 'tags.id',
:order => 'post_count DESC' }
#count = Tag.count(options)
options = options.merge { :offset => (page - 1) * per_page, :limit => per_page }
#items = Tag.find(options)
With the :select option, the Tag.count generates the following SQL:
SELECT count(tags.*, COUNT(*) AS post_count) AS count_tags_all_count_all_as_post_count, tags.id AS tags_id FROM `tags` INNER JOIN posts_tags GROUP BY tags.id ORDER BY COUNT(*) DESC
As you can see it merely wrapped a COUNT() around the 'tags.*, COUNT(*)', and MySQL complains about the COUNT within a COUNT.
Without the :select option, it generates this SQL:
SELECT count(*) AS count_all, tags.id AS tags_id FROM `tags` INNER JOIN posts_tags GROUP BY tags.id ORDER BY COUNT(*)
which returns the whole GROUP BY result set and not the number of rows.
Is there a way around this or will I have to hack up the paginator to account for queries with GROUP BYs (and how would I go about doing that)?
Seems like you'd need to handle the grouped queries separately. Doing a count without a group returns an integer, while counting with a group returns a hash:
Tag.count
SQL (0.2ms) SELECT COUNT(*) FROM "tags"
=> 37
Tag.count(:group=>"tags.id")
SQL (0.2ms) SELECT COUNT(*) AS count_all, tags.id AS tags_id FROM "tags"
GROUP BY tags.id
=> {1=>37}
If you're using Rails 4 or 5 you can do the following as well.
Tag.group(:id).count
The workaround for my situation seems to be to replace the :group => 'tags.id' with :select => 'DISTINCT tags.id' in the options hash before executing the count.
count_options = options.clone
count_options.delete(:order)
if options[:group]
group_by = count_options[:group]
count_options.delete(:group)
count_options[:select] = "DISTINCT #{group_by}"
end
#item_count = #type.count(count_options)
Another (hacky) solution:
selection = Tag.where(...).group(...)
count = Tag.connection.select_value "select count(*) from (" + selection.to_sql + ") as x"
If I understand your question correctly, then it should work if you don't use Tag.count at all. Specifying 'COUNT(*) AS post_count' in your select hash should be enough. For example:
#tag = Tag.first(options)
#tag.post_count
As you can see, the post_count value from the query is accessible from the #tag instance. And if you want to get all tags, then perhaps something like this:
#tags = Tag.all(options)
#tags.each do |tag|
puts "Tag name: #{tag.name} posts: #{tag.post_count}"
end
Update:
Count can be called with which attribute to count and also a parameter :distinct
options = { :select => 'tags.*, COUNT(*) AS post_count',
:joins => 'INNER JOIN posts_tags', #Join table for 'posts' and 'tags'
:group => 'tags.id',
:order => 'post_count DESC',
:offset => (page - 1) * per_page,
:limit => per_page }
#count = Tag.count(:id, :distinct => true, :joins => options[:joins])
#items = Tag.find(options)

named_scope + average is causing the table to be specified more then once in the sql query run on postgresql

I have a named scopes like so...
named_scope :gender, lambda { |gender| { :joins => {:survey_session => :profile }, :conditions => { :survey_sessions => { :profiles => { :gender => gender } } } } }
and when I call it everything works fine.
I also have this average method I call...
Answer.average(:rating, :include => {:survey_session => :profile}, :group => "profiles.career")
which also works fine if I call it like that.
However if I were to call it like so...
Answer.gender('m').average(:rating, :include => {:survey_session => :profile}, :group => "profiles.career")
I get...
ActiveRecord::StatementInvalid: PGError: ERROR: table name "profiles" specified more than once
: SELECT avg("answers".rating) AS avg_rating, profiles.career AS profiles_career FROM "answers" LEFT OUTER JOIN "survey_sessions" survey_sessions_answers ON "survey_sessions_answers".id = "answers".survey_session_id LEFT OUTER JOIN "profiles" ON "profiles".id = "survey_sessions_answers".profile_id INNER JOIN "survey_sessions" ON "survey_sessions".id = "answers".survey_session_id INNER JOIN "profiles" ON "profiles".id = "survey_sessions".profile_id WHERE ("profiles"."gender" = E'm') GROUP BY profiles.career
Which is a little hard to read but says I'm including the table profiles twice.
If I were to just remove the include from average it works but it isn't really practical because average is actually being called inside a method which gets passed the scoped. So there is some times gender or average might get called with out each other and if either was missing the profile include it wouldn't work.
So either I need to know how to fix this apparent bug in Rails or figure out a way to know what scopes were applied to a ActiveRecord::NamedScope::Scope object so that I could check to see if they have been applied and if not add the include for average.
Looks like ActiveRecord is generating some bad SQL:
SELECT avg("answers".rating) AS avg_rating,
profiles.career AS profiles_career
FROM "answers"
LEFT OUTER JOIN "survey_sessions" survey_sessions_answers
ON "survey_sessions_answers".id = "answers".survey_session_id
LEFT OUTER JOIN "profiles"
ON "profiles".id = "survey_sessions_answers".profile_id
INNER JOIN "survey_sessions"
ON "survey_sessions".id = "answers".survey_session_id
INNER JOIN "profiles"
ON "profiles".id = "survey_sessions".profile_id
WHERE ("profiles"."gender" = E'm')
GROUP BY profiles.career
Presumably it's generated the left joins as part of getting the projected property, and the inner joins as part of getting the criteria: this wouldn't be invalid (just inefficient) if it assigned aliases to those tables, but it doesn't. Is there a way to specify an alias name from your app?

Rails scalar query

I need to display a UI element (e.g. a star or checkmark) for employees that are 'favorites' of the current user (another employee).
The Employee model has the following relationship defined to support this:
has_and_belongs_to_many :favorites, :class_name => "Employee", :join_table => "favorites",
:association_foreign_key => "favorite_id", :foreign_key => "employee_id"
The favorites has two fields: employee_id, favorite_id.
If I were to write SQL, the following query would give me the results that I want:
SELECT id, account,
IF(
(
SELECT favorite_id
FROM favorites
WHERE favorite_id=p.id
AND employee_id = ?
) IS NULL, FALSE, TRUE) isFavorite
FROM employees
Where the '?' would be replaced by the session[:user_id].
How do I represent the isFavorite scalar query in Rails?
Another approach would use a query like this:
SELECT id, account, IF(favorite_id IS NULL, FALSE, TRUE) isFavorite
FROM employees e
LEFT OUTER JOIN favorites f ON e.id=f.favorite_id
AND employee_id = ?
Again, the '?' is replaced by the session[:user_id] value.
I've had some success writing this in Rails:
ee=Employee.find(:all, :joins=>"LEFT OUTER JOIN favorites ON employees.id=favorites.favorite_id AND favorites.employee_id=1", :select=>"employees.*,favorites.favorite_id")
Unfortunately, when I try to make this query 'dynamic' by replacing the '1' with a '?', I get errors.
ee=Employee.find(:all, :joins=>["LEFT OUTER JOIN favorites ON employees.id=favorites.favorite_id AND favorites.employee_id=?",1], :select=>"employees.*,favorites.favorite_id")
Obviously, I have the syntax wrong, but can :joins expressions be 'dynamic'? Is this a case for a Lambda expression?
I do hope to add other filters to this query and use it with will_paginate and acts_as_taggable_on, if that makes a difference.
edit
errors from trying to make :joins dynamic:
ActiveRecord::ConfigurationError: Association named 'LEFT OUTER JOIN favorites ON employees.id=favorites.favorite_id AND favorites.employee_id=?' was not found; perhaps you misspelled it?
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/associations.rb:1906:in `build'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/associations.rb:1911:in `build'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/associations.rb:1910:in `each'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/associations.rb:1910:in `build'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/associations.rb:1830:in `initialize'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1789:in `new'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1789:in `add_joins!'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1686:in `construct_finder_sql'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1548:in `find_every'
from /Users/craibuc/.gem/ruby/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:615:in `find'
Try this:
ee=Employee.all(
:select=>"employees.*,favorites.favorite_id",
:joins=>"LEFT OUTER JOIN favorites AS favorites
ON employees.id=favorites.favorite_id AND
favorites.employee_id = #{session[:user_id]}")
Or to be exact:
joins = Employee.send(:sanitize_sql_array,
["LEFT OUTER JOIN favorites AS favorites
ON employees.id=favorites.favorite_id AND
favorites.employee_id = ? ", session[:user_id]
])
ee=Employee.find(:all,
:select=>"employees.*,favorites.favorite_id",
:joins=> joins )
Second approach addresses the SQL injection issues.
Edit 1
To test these calls in irb do the following:
Simulate the session object by creating hash:
>> session = {:user_id => "1" }
session = {:user_id => "1" }
=> {:user_id=>"1"}
Execute the finder:
>> ee=Employee.find(:all,
:select=>"employees.*,favorites.favorite_id",
:joins=>"LEFT OUTER JOIN favorites AS favorites
ON employees.id=favorites.favorite_id AND
favorites.employee_id = #{session[:user_id]}")
I imagine both ways are possible, but normally, I'd stick the condition in the WHERE clause (:conditions):
ee = Employee.find(:all,
:select => 'employees.*, favorites.favorite_id',
:conditions => ['favorites.employee_id = ?', 1],
:joins => 'LEFT OUTER JOIN favorites ON employees.id = favorites.favorite_id'
)
:joins expects either a raw string or a symbol (association name), or an array of associations. So you can't have dynamic conditions there.
See parameters section here.

Resources