Rails + ActiveRecord - fighting with "N+1" issue - ruby-on-rails

I have these models:
car:
class Car < ActiveRecord::Base
has_many :car_services, dependent: :destroy
has_many :services, through: :car_services
end
car_service
class CarService < ActiveRecord::Base
belongs_to :car
belongs_to :service
validates_uniqueness_of :service_id, scope: :car_id
end
truck
class Truck < ActiveRecord::Base
has_many :truck_services, dependent: :destroy
has_many :services, through: :truck_services
end
truck_service
class TruckService < ActiveRecord::Base
belongs_to :truck
belongs_to :service
end
service
class Service < ActiveRecord::Base
has_many :car_services
has_many :cars, through: :car_services
has_many :truck_services
has_many :trucks, through: :truck_services
end
In controllers' actions (there are separated actions for cars and for trucks) I query the models like this:
#cars = Car.find_by_sql("SELECT ...")
...
#trucks = Truck.find_by_sql("SELECT ...")
and then in the respective views:
<% #cars.each do |result| %>
<li data-services="<% result.services.each do |service| %><%= service.name %>,<% end %>">...</li>
...
<% end %>
This procedure works very well for cars, it's very fast.
Different action for trucks, but the same procedure:
<% #trucks.each do |result| %>
<li data-services="<% result.services.each do |service| %><%= service.name %>,<% end %>">...</li>
...
<% end %>
Now, this procedure is very slow. When I tried to debug why (because as I mentioned, the procedure is same for both models, but for cars it's working very fast even though that in the car model is 600k records and in the truck "only" 300k), I spotted in the console log following information:
...
Service Load (159.1ms) SELECT `services`.* FROM `services` INNER JOIN `truck_services` ON `services`.`id` = `truck_services`.`service_id` WHERE `truck_services`.`truck_id` = 35769
Service Load (166.8ms) SELECT `services`.* FROM `services` INNER JOIN `truck_services` ON `services`.`id` = `truck_services`.`service_id` WHERE `truck_services`.`truck_id` = 45681
Service Load (151.4ms) SELECT `services`.* FROM `services` INNER JOIN `truck_services` ON `services`.`id` = `truck_services`.`service_id` WHERE `truck_services`.`truck_id` = 50974
...
How is this possible? Why I don't see these lines also for cars and I see them for truck? Do I miss something? I was thinking about missing indexes in the MySQL tables, but there aren't used any.
I am fighting with this issue the whole Saturday, but I have no solution for this...
I would be very grateful for an advice how to fix this issue.
Thank you very much guys.

I will start with an advice: post your sql queries for others to understand exactly what you are doing and to be able to help you
To eliminate N+1 you should use includes or preload
In your controllers do this:
#cars = Car.includes(:services).all
respectively,
#trucks = Truck.includes(:services).all
You can see that I preferred a much cleaner syntax to find_by_sql.
When you run:
#cars = Car.includes(:services).all
this will trigger 3 queries:
SELECT "cars".* FROM "cars";
SELECT "car_services".* FROM "car_services" WHERE "car_services".car_id IN (?, ?, ? ...);
SELECT "services".* FROM "services" WHERE "services".car_service_id IN (?, ?, ? ...);
If you remove N+1 your page will load faster but you should check that all your keys have indexes defined, too.
The above its just an example. You should really post your complete queries for us to understand exactly what you are doing. And those models Car and Truck, they look very similar, maybe it will be a better solution to use Single Table Inheritance.
You could rewrite this:
<% #trucks.each do |result| %>
<li data-services="<% result.services.each do |service| %><%= service.name %>,<% end %>">...</li>
<% end %>
as this
<% #trucks.each do |truck| %>
<%= content_tag :li, '...', data: {services: truck.services.map(&:name).join(', ')} %>
<% end %>
Hope this will help you somehow.

Related

How to override table column value with column value from second table if it exist?

I'm building e-commerce application in which you can set user specific prices for products. If price for specific product for specific user is not set, it will show default product price.
It works fine, but I'm looking for more efficient solution since I'm not happy with current one.
Tables:
users
products (all product info + regular_price)
prices (user_id, product_id, user_price)
Models:
class User < ApplicationRecord
has_many :prices
end
class Product < ApplicationRecord
has_many :prices
validates :name, presence: true
def self.with_user_prices(current_user)
Product.joins(
Product.sanitize_sql_array(['LEFT OUTER JOIN prices ON prices.user_id = ?
AND products.id = prices.product_id', current_user])
).select('products.*, prices.user_price')
end
end
class Price < ApplicationRecord
belongs_to :product
belongs_to :user
end
How I get all products with user specific prices in controller:
#products = Product.with_user_prices(current_user)
How I display them in view:
<% #products.each do |product| %>
<%= product.user_price ? product.user_price : product.regular_price %>
<% end %>
As you see I'm currently joining prices table and then in view I display user_price (prices table) if it exists, otherwise regular_price (products table).
I would love to solve everything in a single query keeping only one price column with appropriate value according to the current_user
You can make use of SQL COALESCE function:
class Product < ApplicationRecord
# ...
def self.with_user_prices(user)
includes(:prices).where(prices: { user_id: user.id }).select(
'products.*, COALESCE(prices.user_price, products.regular_price) as price'
)
end
end
Then you can use it simply by:
<%= product.price %>
Note that I simplified Product.with_user_prices method a little bit by using includes, which is gonna generate SQL LEFT JOIN query since there's condition on prices.
New Answer:
please do not mark this answer as correct because I just basically extended Marek's code and yours into mine, as after several attempts I came to what you've already done anyway, but I'm just putting it here in case it helps anyone:
app/models/product.rb
class Product < ApplicationRecord
def self.with_user_prices(user)
joins(
sanitize_sql_array([
"LEFT OUTER JOIN prices on prices.product_id = products.id AND prices.user_id = ?", user.id
])
).select(
'products.*',
'COALESCE(prices.user_price, products.regular_price) AS price_for_user'
)
end
end
controller:
#products = Product.with_user_prices(current_user)
view:
<% #products.each do |product| %>
<%= product.price_for_user %>
<% end %>
Old Answer (Inefficient Code):
Untested but can you try the following? (Not sure if this is more or less efficient than your approach)
app/models/product.rb
class Product < ApplicationRecord
has_many :prices
def price_for_user(user)
prices.includes(:user).where(
users: { id: user.id }
).first&.user_price || regular_price
end
end
controller:
# will perform LEFT OUTER JOIN (to eager load both `prices` and `prices -> user`) preventing N+1 queries
#products = Product.eager_load(prices: :user)
view:
<% #products.each do |product| %>
<%= product.price_for_user(current_user) %>
<% end %>

Rails display name attribute from different table along with count from join table

I have the following tables: Products, Categories, ProductCategory.
I'm trying to display in a branch structure something like:
Accessories - 5,000
Men's Watches - 2,000
Tag Huer - 1,000
Samsung - 1,000
Women's Watches - 3,000
The issue I'm getting into is that my query is slowing down the application. I have the following query:
def category_level
Connector::Category.includes(:products).group_by(&:parent_category_id)
end
Then in my views I have the following:
<ul class="list-style-upper">
<% category_level[nil].each do |root| %>
<%= render 'products/submenu/category_item', category: root %>
<% end %>
</ul>
Which loads into:
<li class="list-one">
<%= category.name %><p><%= category.products.count %></p>
<% if category_level[category.id].present? %>
<ul class="list-style-upper-sub">
<% category_level[category.id].each do |child| %>
<%= render 'products/submenu/category_item', category: child %>
<% end %>
</ul>
<% end %>
</li>
It displays but takes a long time due to it hitting the Products table and looping through all the products. So to make it easier I thought I'd just hit the ProductCategory page to get a count with the following:
def level_up
#category_counts = Connector::ProductCategory.group(:category_id).count
end
This will actually just display the following:
{54 => 11, 29 => 14, 51 => 19, 10 => 3202}
Although yes 10 would represent Accessories I'd rather see Accessories - 3,202.
Any advice on cleaning this up to pull in the attribute of name?
Provided that your models look something like:
class Category < ApplicationRecord
has_many :product_categories
has_many :products, through: :product_categories
end
class ProductCategory < ApplicationRecord
belongs_to :product
belongs_to :category
end
class Product < ApplicationRecord
has_many :product_categories
has_many :categories, through: :product_categories
end
You can get a count of the associated items by joining and selecting a count of the joined table:
Category.left_outer_joins(:product_categories)
.select('categories.*, count(product_categories.*) AS product_count')
.group('categories.id')
.left_outer_joins was added in Rails 5. For earlier versions use:
.join('LEFT OUTER JOIN product_categories ON product_categories.category_id = category.id')
You can also use .joins(:product_categories) if you don't care about categories without any products.
The count will be available as .product_count on each record.
This can also be achieved by adding a counter cache:
class ProductCategory < ApplicationRecord
belongs_to :product
belongs_to :category, counter_cache: true
end
class AddProductCategoriesCountToCategories < ActiveRecord::Migration[5.2]
def change
add_column :categories, :product_categories_count, :integer, default: 0
end
end
This will store a count in categories.product_categories_count. Counter caches are best used when you have more read than write operations.

Efficient way to count associated objects in Rails 4

I am looking for a way to show a count of how many images there are for a category but obtained through a has_many association. I have been reading a little on counter_cache but as yet no joy on an implementation
class Category < ActiveRecord::Base
has_many :image_categories
has_many :images, through: :image_categories
end
class ImageCategory < ActiveRecord::Base
# Holds image_id and category_id to allow multiple categories to be saved per image, as opposed to storing an array of objects in one DB column
belongs_to :image
belongs_to :category
end
class Image < ActiveRecord::Base
# Categories
has_many :image_categories, dependent: :destroy
has_many :categories, through: :image_categories
end
Controller
#categories = Category.all
View
<% #categories.each do |c| %>
<li>
<%= link_to '#', data: { :filter => '.' + c.name.delete(' ') } do %>
<%= c.name %> (<%= #count here %>)
<% end %>
</li>
<% end %>
A couple important things to consider with counter_cache:
Certain Rails methods can update the database while bypassing callbacks (for instance update_column, update_all, increment, decrement, delete_all, etc.) and can cause inconsistent values for a counter cache. Same applies to any database changes outside of Rails.
Creating/deleting a child model always requires updating the parent. To ensure consistency of the counter cache Rails uses an additional DB transaction during this update. This usually isn't a problem but can cause database deadlocks if your child model is created/deleted frequently, or if the parent model is updated frequently. (http://building.wanelo.com/2014/06/20/counter-cache-a-story-of-counting.html)
These problems will be exacerbated since you're using a counter cache across a join table.
If you want to do an efficient dynamic count, that's always up to date, then you can use a custom select with a grouped join:
#categories = Category.select("categories.*, COUNT(DISTINCT images.id) AS images_count").joins(:images).group("categories.id")
<% #categories.find_each do |c| %>
<li>
<%= link_to '#', data: { :filter => '.' + c.name.delete(' ') } do %>
<%= c.name %> (<%= c.images_count # <- dynamic count column %>)
<% end %>
</li>
<% end %>
The cost of this grouped join should be very small provided your foreign keys are indexed, and I'd strongly consider taking this approach if you need images_count to always be consistent with the true value, or if images are frequently being created or destroyed. This approach may also be easier to maintain in the long run.
Since you are looking for an efficient way, i would suggest using counter_cache
Here is how your models should look like:
class Category < ActiveRecord::Base
has_many :image_categories
has_many :images, through: :image_categories
end
class ImageCategory < ActiveRecord::Base
# Holds image_id and category_id to allow multiple categories to be saved per image, as opposed to storing an array of objects in one DB column
belongs_to :image, counter_cache: :category_count
belongs_to :category, counter_cache: :image_count
end
class Image < ActiveRecord::Base
# Categories
has_many :image_categories, dependent: :destroy
has_many :categories, through: :image_categories
end
You'll need to add image_count field to your categories table and category_count in images table.
Once you are done adding the counters and fields, you'd need to reset the counters so that the fields are updated with the correct count values for the records already present in your db.
Category.find_each { |category| Category.reset_counters(category.id, :images) }
Image.find_each { |image| Image.reset_counters(image.id, :categories) }

Rails, how to avoid the "N + 1" queries for the totals (count, size, counter_cache) in associations?

I have a these models:
class Children < ActiveRecord::Base
has_many :tickets
has_many :movies, through: :tickets
end
class Movie < ActiveRecord::Base
has_many :tickets
has_many :childrens, through: :tickets
belongs_to :cinema
end
class Ticket < ActiveRecord::Base
belongs_to :movie, counter_cache: true
belongs_to :children
end
class Cinema < ActiveRecord::Base
has_many :movies, dependent: :destroy
has_many :childrens, through: :movies
end
What I need now is in the page of "Cinemas" I wanna print the sum (count, size?) of the childrens just for the movies of that cinemas, so I wrote this:
in the cinemas_controller.rb:
#childrens = #cinema.childrens.uniq
in the cinemas/show.html.erb:
<% #childrens.each do |children| %><%= children.movies.size %><% end %>
but obviously I have bullet gem that alert me for Counter_cache and I don't know where to put this counter_cache because of different id for the movie.
And also without the counter_cache what I have is not what I want because I want a count for how many childrens in that cinema taking them from the tickets from many days in that cinema.
How to?
UPDATE
If in my view I use this code:
<% #childrens.each do |children| %>
<%= children.movies.where(cinema_id: #cinema.id).size %>
<% end %>
gem bullet don't say me anything and every works correctly.
But I have a question: this way of querying the database is more heavy because of the code in the views?
This might help you.
#childrens_count = #cinema.childrens.joins(:movies).group("movies.children_id").count.to_a
You can use includes to load all associations ahead of time. For example:
#childrens = #cinema.childrens.includes(:movies).uniq
This will load all of the children's movies in the controller, preventing the view from needing access to the database in your loop.
You might agree, that the number of movies belongs to a child equals the number of tickets they bought.
That's why you could just cache the number of tickets and show it on the cinemas#show.
You can even create a method to make it more clear.
class Children < ActiveRecord::Base
has_many :tickets
has_many :movies, through: :tickets
def movies_count
tickets.size
end
end
class Ticket < ActiveRecord::Base
belongs_to :movie, counter_cache: true
belongs_to :children, counter_cache: true
end
class Movie < ActiveRecord::Base
belongs_to :cinema
has_many :tickets
has_many :childrens, through: :tickets
end
class Cinema < ActiveRecord::Base
has_many :movies, dependent: :destroy
has_many :childrens, through: :movies
end
And then:
<% #childrens.each do |children| %><%= children.tickets.size %><% end %>
Or
<% #childrens.each do |children| %><%= children.movies_count %><% end %>
But if you want to show the number of tickets for every movie, you definitely need to consider the following:
#movies = #cinema.movies
Then:
<% #movies.each do |movie| %><%= movie.tickets.size %><% end %>
Since you have belongs_to :movie, counter_cache: true, tickets.size won't make a count query.
And don't forget to add tickets_count column. More about counter_cache...
P.S. Just a note, according to conventions we name a model as Child and an association as Children.
Actually is much more simpler than the remaining solutions
You can use lazy loading:
In your controller:
def index
# or you just add your where conditions here
#childrens = Children.includes(:movies).all
end
In your view index.hml.erb:
<% #childrens.each do |children| %>
<%= children.movies.size %>
<% end %>
The code above won't make any extra query if you use size but if you use count you will face the select count(*) n + 1 queries
I wrote a little ActiveRecord plugin some time ago but haven't had the chance to publish a gem, so I just created a gist:
https://gist.github.com/apauly/38f3e88d8f35b6bcf323
Example:
# The following code will run only two queries - no matter how many childrens there are:
# 1. Fetch the childrens
# 2. Single query to fetch all movie counts
#cinema.childrens.preload_counts(:movies).each do |cinema|
puts cinema.movies.count
end
To explain a bit more:
There already are similar solutions out there (e.g. https://github.com/smathieu/preload_counts) but I didn't like their interface/DSL. I was looking for something (syntactically) similar to active records preload (http://apidock.com/rails/ActiveRecord/QueryMethods/preload) method, that's why I created my own solution.
To avoid 'normal' N+1 query issues, I always use preload instead of joins because it runs a single, seperate query and doesn't modify my original query which would possibly break if the query itself is already quite complex.
In You case You could use something like this:
class Ticket < ActiveRecord::Base
belongs_to :movie, counter_cache: true
belongs_to :children
end
class Movie < ActiveRecord::Base
has_many :tickets
has_many :childrens, through: :tickets
belongs_to :cinema
end
class Children < ActiveRecord::Base
has_many :tickets
has_many :movies, through: :tickets
end
class Cinema < ActiveRecord::Base
has_many :movies, dependent: :destroy
has_many :childrens, through: :movies
end
#cinema = Cinema.find(params[:id])
#childrens = Children.eager_load(:tickets, :movies).where(movies: {cinema_id: #cinema.id}, tickets: {cinema_id: #cinema.id})
<% #childrens.each do |children| %>
<%= children.movies.count %>
<% end %>
Your approach using counter_cache is in right direction.
But to take full advantage of it, let's use children.movies as example, you need to add tickets_count column to children table firstly.
execute rails g migration addTicketsCountToChildren tickets_count:integer,
then rake db:migrate
now every ticket creating will increase tickets_count in its owner(children) by 1 automatically.
then you can use
<% #childrens.each do |children| %>
<%= children.movies.size %>
<% end %>
without getting any warning.
if you want to get children count by movie, you need to add childrens_count to movie table:
rails g migration addChildrensCountToMovies childrens_count:integer
then rake db:migrate
ref:
http://yerb.net/blog/2014/03/13/three-easy-steps-to-using-counter-caches-in-rails/
please feel free to ask if there is any concern.
Based on sarav answer if you have a lot of things(requests) to count you can do:
in controller:
#childrens_count = #cinema.childrens.joins(:movies).group("childrens.id").count.to_h
in view:
<% #childrens.each do |children| %>
<%= #childrens_count[children.id] %>
<% end %>
This will prevent a lot of sql requests if you train to count associated records

Rails - List separated by commas through a join table

I have three tables, one of which is a join table between the other two tables.
Jobs: id
Counties: id
Countyizations: job_ib, county_id
I want to create a list of counties a specific job has associations with. I'm trying to use something like:
<%= #counties.map { |county| county.id }.join(", ") %>
But this obviously is not using the countyizations table. How can I change the above code to accomplish what I need? Also, I'd like to list the counties alphabetically in ASC order.
P.S.
I suppose I should have added how I'm linking my tables in my models.
Jobs: has_many :countyizations & has_many :counties, :through => :countyizations
Counties: has_many :countyizations & has_many :jobs, :through => :countyizations
Countyizations: belongs_to :county & belongs_to :job
For a given job.id you can use this this return all the counties filtered by the given job.
<%= #counties.order('name asc').includes(:jobs).where('jobs.id = ?', job.id) %>
Replace job.id based on your requirement, you could set a #job instance variable in the controller and use in the view instead.
Or even better move this code to controller action:
# controller
def show
job_name = ...
#counties = ...
#county_jobs = #counties.order('name asc').includes(:jobs).where(jobs.name = ?', job_name)
end
Then in your view, to show all the counties that have the searched job:
<%= #counties.map(&:id).join.(',') %>
I am not sure if I understand you correctly. Is the following what you want?
class Job < ActiveRecord::Base
has_many :countries, :through => :countyizations
end
class County < ActiveRecord::Base
has_many :jobs, :through => :countyizations
end
<%= #job.counties.sort{|a, b| a.name <=> b.name}.map{ |county| county.name }.join(", ") %>
I think use "has_many_and_belongs_to" instead "of has_many" may work also.

Resources