How can I optimize and combine these inefficient database queries? - ruby-on-rails

Question

Here's are my DB queries:
User has_many UserFollow (UserFollow is a relationship between User models).
User has_many Photos.
Photo has_many PhotoFollow relationships (PhotoFollow is a relationship between User and Photo models).
#user_list = Array.new
user_followers = UserFollow.where("user_1_id = ?", current_user.id).includes(:follower)
user_followers.each do |f|
#user_list << f.user
end
photos = Photo.where("user_id = ?", current_user.id).includes(:follow_relationships => [:photo])
photos.each do |p|
p.follow_relationships.each do |f|
#user_list << f.user if !#user_list.include? f.user
end
end
if #user_list.size < 150
users = User.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).includes(:solutions).limit(150 - #user_list.size)
users.each do |u|
#user_list << u if !#user_list.include? u
end
end
All of this takes a ridiculous amount of time obviously. Using includes helps, but I am wondering if there is some way to more efficiently do this set of operations.
Thanks,
Ringo

Answer 1

Append your associations first.
class User < ActiveRecord::Base
has_many :follows_to_user, :class_name => 'UserFollow', :foreign_key => 'follower_id'
has_many :follows_to_photo, :class_name => 'PhotoFollow', :foreign_key => 'user_id' # if you don't have it now
end
Now, first two queries you can make much more elegant within just one SQL query, returning AR::Relation scope.
#user_list = User.includes(:follows_to_user => {}, :follows_to_photo => {:photo => {}}).where(["user_follows.user_1_id = :user_id OR photos.user_id = :user_id", :user_id => current_user.id])
About 150... [updated]
Of course you'd better to implement that logic appending the previous SQL statement with conditions and UNION statement (using SQL syntax only), that should return AR::Relation and will be a little faster. But you can stay lazy and leave it in ruby, although it will return Array:
if (count = #user_list.count) && count < 150 # run COUNT just once and store value into local variable
#user_list |= User.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).includes(:solutions).limit(150 - count)
end

Answer 2

Looking at your code, you're planning on getting a list of users to your #user_list. You can build up the list of user ids first so we don't create unnecessary AR objects
First code
#user_list = Array.new
user_followers = UserFollow.where("user_1_id = ?", current_user.id).includes(:follower)
user_followers.each do |f|
#user_list << f.user
end
can be changed to
# assuming you have a user_id column on user_follows table
user_ids = User.joins(:user_follows).where(user_follows: { user_1_id: current_user.id })
.uniq.pluck('user_follows.user_id')
Second code
photos = Photo.where("user_id = ?", current_user.id).includes(:follow_relationships =>[:photo])
photos.each do |p|
p.follow_relationships.each do |f|
#user_list << f.user if !#user_list.include? f.user
end
end
can be changed to
user_ids += Photo.where(user_id: current_user.id).joins(follow_relationships: :photo)
.uniq.pluck('follow_relationships.user_id')
Third code
if #user_list.size < 150
users = User.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).includes(:solutions).limit(150 - #user_list.size)
users.each do |u|
#user_list << u if !#user_list.include? u
end
end
can be changed to
user_ids += users = User.where(verified: true)
.where('first_name IS NOT NULL AND last_name IS NOT NULL')
.where('id NOT IN (?)', user_ids)
.limit(150 - user_ids.size).pluck(:id)
Then you can just fetch all the users using user_ids
#user_list = User.where(id: user_ids)

Answer 3

There has to be a better way then my answer, but why don't you include :user since you're loading them when iterating over your queries?
#user_list = Array.new
user_followers = UserFollow.includes(:user).where("user_1_id = ?", current_user.id)
# why did you include followers?
user_followers.each do |f|
#user_list << f.user
end
photos = Photo.includes(follow_relationships: { photo: :user }).where("user_id = ?", current_user.id)
photos.each do |p|
p.follow_relationships.each do |f|
#user_list << f.user unless #user_list.include? f.user
end
end
if #user_list.size < 150
users = User.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).limit(150 - #user_list.size)
# why did you include solutions?
users.each do |u|
#user_list << u unless #user_list.include? u
end
end
Maybe this is faster, I'm not sure:
#follower_ids = UserFollow.where("user_1_id = ?", current_user.id).pluck(:user_1_id).uniq
#photo_ids = Photo.joins(follow_relationships: :photo)
#photo_ids = #photo_ids.where("user_id = ? and user_id not in (?)", current_user.id, #follower_ids)
#photo_ids = #photo_ids.pluck(:user_id).uniq
#followers = User.where("id in (?)", #follower_ids)
#photo_users = User.where("id in (?) and not in (?)", #photo_ids, #follower_ids)
#array_size = (#follower_ids + #photo_ids).size
if #array_size < 150
#users = User.where("verified = ? and first_name is not null and last_name is not null", true)
#users = #users.where("id not in (?)", #photo_ids + #follower_ids).limit(150 - #array_size)
else
#users = []
end
#final_array = #followers + #photo_users + #users
I haven't tested if this works, or if it's faster. It has more database queries but less iterations.
Update
What if you added another column to the users model that gets updated with a value from 1 to 3, depending on whether they have followers, photos, or nothing.
Then all you would need to do is:
# in User model
def self.valid_users
where("verified = ? and first_name is not null and last_name is not null", true)
end
#users = User.valid_users.order("sortable ASC").limit(150)

Answer 4

This is just a guess.
#user_list = current_user.followers.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).includes(:solutions, :photos => {:follow_relationships}).limit(150)
Over all I think its all wrong due to the complexity alone. If you cant read what its doing clearly then you should start over again.

Answer 5

Have you set up table indexes for your tables?
If you haven't yet, set them up for all of your foreign keys and for columns that you need to include in your conditions. In your DB migration script (with your correct table and column names, of course). They will speed up your queries, especially if you have a large dataset:
add_index :user_follows, :follower_id
add_index :user_follows, :followed_id
add_index :photos, :user_id
add_index :photo_follow_relationships, :photo_id
add_index :photo_follow_relationships, :follower_id
add_index :users, :verified
add_index :users, :first_name
add_index :users, :last_name
Also, some comments and recommendations:
# ... [SNIP] Query and add to user list.
user_followers = [] # You are not actually using the UserFollow records. Unload
# them from memory. Otherwise, they will be stored until we
# leave the block.
# There is no need to fetch other Photo data here, and there is no need to load
# :photo for FollowRelationship. But there is a need to load :user.
photos = Photo.where(:user_id => current_user.id).select('photos.id').
includes(:follow_relationships => [:user])
# ... [SNIP] Add to user list.
photos = [] # Unload photos.
# ... [SNIP] Your condition.
# There is no need to load :solutions for the users here.
users = User.where("verified = ? and first_name IS NOT NULL and last_name IS NOT NULL", true).limit(150 - #user_list.size)
# ... [SNIP] Add to user list.
It would, of course, be nicer if you refactor the code too, as in some of mind blank's recommendations. You can also manke use of has_many :through associations too to clean up your controller.

Answer 6

Add the following relationships to your models
class User
has_many :user_follows
has_many :inverse_user_follows, :class_name=>'UserFollow', :foreign_key=>:follower_id
# followers for user
has_many :followers, :through => :user_follows
# users followed by user
has_many :followed_users, :through => :inverse_user_follows, :source => :user
# photos created by user
has_many :photos
has_many :photo_user_follows, :through => :photos, :source => :user_follows
# followers for user's photo
has_many :photo_followers, :through => :photo_user_follows, :source => :follower
has_many :photo_follows
# photos followed by user
has_many :followed_photos, :source => :photo, :through => :photo_follows
end
class UserFollow
# index user_id and follower_id columns
belongs_to :user
belongs_to :follower, :class_name => "User"
end
Photo related models
class Photo
# index user_id column
belongs_to :user
has_many :photo_follows
has_many :followers, :through => :photo_follows
end
class PhotoFollow
# index photo_id and follower_id columns
belongs_to :photo
belongs_to :follower, :class_name => "User"
end
Now you can get the users who are current users followers OR current user's photo followers OR active users..
user_ids = current_user.follower_ids | current_user.photo_follower_ids
User.where("ids IN (?) OR
( verified = ? and first_name IS NOT NULL and last_name IS NOT NULL )",
user_ids, true).limit(150)