I have a "Loan" model in Rails I'm trying to build. There's a corresponding "Payment" model. The balance on the loan is the original amount of the loan minus the sum of all the payments. Calculating the balance is easy, but I'm trying to calculate the balance on lots of loans while avoiding an N+1 query, while at the same time making the "balance" a property of the "Loan" model.
When I call the index method of the Loans controller, I can run a custom select query, which allows me to return a "balance" property through a straight SQL query.
class LoansController < ApplicationController
def index
#loans = Loan
.joins("LEFT JOIN payments on payments.loan_id = loan.id")
.group("loans.id")
.select("loans.*, loans.amount - SUM(payments.amount) as balance")
end
def index_002
#loans = Loan.includes(:payments)
end
def index_003
#loans = Loan.includes(:payments)
end
end
class Loan < ActiveRecord::Base
has_many :payments
def balance=(value)
# I'd like balance to load automatically in the Loan model.
raise NotImplementedError.new("Balance of a loan cannot be set directly.")
end
def balance_002
# No N+1 query, but iterating through each payment in Ruby
# is grossly inefficient as well
amount - payments.map(:amount).inject(0, :+)
end
def balance_003
# Even with the "includes" in the controller, this is N+1
amount - (payments.sum(:amount) || 0)
end
end
Now my question is how to do this all the time with my Loan model. Normally ActiveRecord loads one or more models using the following query:
SELECT * FROM loans
--where clause optional
WHERE id IN (?)
Is there any way to override the Loan model so that it loads the following query:
SELECT
loans.*, loans.amount - SUM(payments.amount) as balance
FROM
loans
LEFT JOIN
payments ON payments.loan_id = loans.id
GROUP BY
loans.id
This way "balance" is a property of the model and only has to be declared in one place, but we also avoid the inefficiencies of N+1 queries.
I like to use database views for this, so that rails thinks it's talking to a regular database table (ensuring things like eager loading work normally) when in fact there are aggregations or complex joins going on. In your case, I might define a second loan_balances view:
create view loan_balances as (
select loans.id as loan_id, loans.amount - sum(payments.amount) as balance
from loans
left outer join payments on payments.loan_id = loans.id
group by 1
)
Then just do regular rails association stuff:
class LoanBalance < ActiveRecord::Base
belongs_to :loan, inverse_of: :loan_balance
end
class Loan < ActiveRecord::Base
has_one :loan_balance, inverse_of: :loan
delegate :balance, to: :loan_balance, prefix: false
end
This way in actions where you want the balance you can eager load it with includes(:loan_balance), but you won't get into thorny issues from violating rails conventions in all the standard CRUD stuff surrounding Loan itself.
It looks like I finally answered my own question. Here it is. I overrode the default scope.
class Loan < ActiveRecord::Base
validates :funded_amount, presence: true, numericality: {greater_than: 0}
has_many :payments, dependent: :destroy, inverse_of: :loan
default_scope {
joins("LEFT JOIN payments as p ON p.loan_id = loans.id")
.group("loans.id").select("loans.*, sum(p.amount) as paid")
}
def balance
funded_amount - (paid || 0)
end
end
Related
I have a model Invoice which has_many Payments and a model Payment that belongs_to Invoice.
We export Invoice data monthly in batches, and we need each Invoice's last Payment.
In our view we are currently doing Invoice.payments.last once for each Invoice that we are exporting, and I was asked to prevent N+1 queries.
I don't understand if I should add this query in the controller or in the Invoice model, or if it should be a has_one :last_payment association or a scope.
Any help would be appreciated.
If the number of payments per invoice is relatively small you can just include/eager_load/preload the association:
invoices = Invoice.includes(:payments)
invoices.each do |i|
puts i.payments.last.amount # no n+1 query
end
However this will load all the associated records into memory at once. This can cause performance problems.
One a very performant read optimization would be to add foreign key column to the invoices table and a belongs_to association which you can use when eager loading:
class AddLatestPaymentToInvoices < ActiveRecord::Migration[6.0]
def change
add_reference :invoices, :latest_payment, null: false, foreign_key: { to_table: :payments }
end
end
class Invoice < ApplicationRecord
has_many :payments, after_add: :set_latest_invoice!
belongs_to :latest_payment,
class_name: 'Payment'
private
def set_latest_payment(payment)
update_columns(latest_payment_id: payment.id)
end
end
invoices = Invoice.includes(:latest_payment)
invoices.each do |i|
puts i.latest_payment.amount # no n+1 query
end
The cost is an additional UPDATE query per record inserted. It can be optimized by using a DB trigger instead of an association callback.
I have been racking my brain all day and can't get this to work. I am very new to ruby and rails so my apologies for any silly errors.
My problem is I am joining 3 tables together to get a #students object. This works but if I call for example #student.name then 'name' doesn't exist.
Below is my code:
Controller
note I have tried using .includes and .join and the same problem happens.
class MyprojectController < ApplicationController
def show
#project = Project.find(params[:id])
#dateformat = '%b %e - %H:%M'
#user = Student.includes("INNER JOIN researchers ON students.researcher_id = researchers.id
INNER JOIN users ON researchers.user_id = users.id").where('user_id = ?', current_user.id)
end
User Model
class User < ApplicationRecord
include EpiCas::DeviseHelper
has_many :event_registrations
has_many :events, through: :event_registrations
belongs_to :project
has_many :researchers
#has_many :students, :through => :researchers
#has_many :supervisors, :through => :researchers
# def self.authenticate(username)
# where(username: username).first
# end
end
Researcher Model
class Researcher < ApplicationRecord
#belongs_to :project
belongs_to :user
has_many :supervisor
has_many :students
end
Student Model
class Student < ApplicationRecord
#Must have the following
validates :name, :email, :surname, :email, :supervisor, :registration_number, presence: true
#ensures unique email addresses
validates :email, uniqueness: true
#assosiations
belongs_to :researcher
end
So every student has a researcher_id and every researcher has a user_id. So the joins should go student->researcher->user and then I want to be able to use all the attributes from all tables in an #user object.
I tried using Student.join(:researcher, :user) but that tried to do a join from the Student table to the researchers table and then tried to join the user table by using a user_id from the student table (but of the user_id is in the researcher table). So i have just done the query myself.
All the data seems to be there but as 'raw attributes'.
Any help would be greatly appreciated!
-James
Rather than try and join things into one return (like you would in sql) use includes so that you can access all your data in fewer queries but you still have access to your data in objects. The point of using an ORM like ActiveRecord is to be able to access your data using objects. The downside of using an ORM is that sometimes it's not as efficient at getting you the exact data you want, because the data is pushing into objects. Using includes provides a sort of middle ground where you can access the data you require in objects and you don't necessarily have to run queries for each association.
Try something like (depending on how you're getting your user id -- I'm assuming from project):
#user = User.includes(researcher: :student).find(project.user_id)
And then you can access things through the normal rails associations:
researcher = #user.researcher
student = researcher.student
I hope that helps and best of luck!
Imagine 3 models: User, Club and UserClub.
class User < ActiveRecord::Base
has_many :clubs,
through: :user_clubs
has_many :user_clubs
end
class Club < ActiveRecord::Base
has_many :users,
through: :user_clubs
has_many :user_clubs
end
class UserClub < ActiveRecord::Base
belongs_to :user
belongs_to :club
end
Very typical join table stuff.
Now, imagine a scenario where you want to retrieve one user's clubs and the amount of users that are in each club.
In a controller, retrieving a users clubs is simple:
def index
#clubs = current_user.clubs
#do whatever you will with them
end
The second part puzzles me though as I don't know how to do it as efficiently as possible.
Sure, I could do something like this:
def index
#clubs = current_user.clubs
#no_of_users_per_club = Hash.new(0)
#clubs.each do |club|
#no_of_users_per_club[club.id] = UserClub.where(club_id: club.id).count
end
#Do whatever you would do after
end
Is there a better way to do this? It would be a tad redundant, but ultimately, maybe the best solution is to simply store that integer as an attribute of each network, so that when a user joins a club, I increment it by one and when a user leaves a club, I decrease it by one?
UPDATE: The selected answer below shows a very cool way to do it and an even cooler way to limit the results to just your clubs.
#no_of_user_per_club_of_mine = UserClub.
joins("INNER JOIN user_clubs AS uc ON user_clubs.club_id = uc.club_id").
where("uc.user_id = ?" , current_user.id).
group("user_clubs.club_id").
count("user_clubs.user_id")
You can use group to retrieve the count of users in each club directly from the join model 'TableClub'.
#no_of_users_per_club = UserClub.group(:club_id).count(:user_id)
# => {1=>1, 2=>5, 3=>8}
To get the number of users in each club, where these clubs are joined by current_user:
#no_of_user_per_club_of_mine = UserClub.joins("INNER JOIN user_clubs AS uc ON user_clubs.club_id = uc.club_id").where("uc.user_id = ?" , current_user.id).group("user_clubs.club_id").count("user_clubs.user_id")
#count = current_user.user_clubs.group_by{|o|o.club_id}.map{|k,v|[k, v.first.club.users.length]}
This will return a array of array which contain club_id at first index and count of users related to this club at second index
#count.length will return total clubs related to current user
EDIT - Using 'includes' generates a SQL 'IN' clause. When using Oracle this has a 1,000 item limit. It will not work for my company. Are there any other solutions out there?
Is it possible to eager load an association on an association?
For example, let's say I have an Academy class, and an academy has many students. Each student belongs_to student_level
class Academy < ActiveRecord::Base
has_many :students
end
class Student < ActiveRecord::Base
belongs_to :academy
belongs_to :student_level
end
class StudentLevel < ActiveRecord::Base
has_many :students
end
Is it possible to tailor the association in Academy so that when I load the students, I ALWAYS load the student_level with the student?
In other words, I would like the following section of code to produce one or two queries total, not one query for every student:
#academy.students.each do |student|
puts "#{student.name} - #{student.student_level.level_name}"
end
I know I can do this if I change students from an association to a method, but I don't want to do that as I won't be able to reference students as an association in my other queries. I also know that I can do this in SQL in the following manner, but I want to know if there's a way to do this without finder_sql on my association, because now I need to update my finder_sql anytime my default scope changes, and this won't preload the association:
SELECT students.*, student_levels.* FROM students
LEFT JOIN student_levels ON students.student_level_id = student_levels.id
WHERE students.academy_id = ACADEMY_ID_HERE
Have you tried using includes to eager load the data?
class Academy < ActiveRecord::Base
has_many :students
# you can probably come up with better method name
def students_with_levels
# not sure if includes works off associations, see alternate below if it does not
self.students.includes(:student_level)
end
def alternate
Student.where("academy_id = ?", self.id).includes(:student_level)
end
end
see also: http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
should result in 3 queries
the initial find on Academy
the query for a collection of Student objects
the query for all of those Students StudentLevel objects
Additions:
# the same can be done outside of methods
#academy.students.includes(:student_level).each do |student|
puts "#{student.name} - #{student.student_level.level_name}"
end
Student.where("academy_id = ?", #academy.id).includes(:student_level).each do |student|
puts "#{student.name} - #{student.student_level.level_name}"
end
ActiveRelation queries are also chainable
#academy.students_with_levels.where("name ILIKE ?", "Smi%").each do # ...
Sort of related a nice article on encapsulation of ActiveRecord queries (methods) - http://ablogaboutcode.com/2012/03/22/respect-the-active-record/
I think I need something akin to a rails eager loaded query with a limit on it but I am having trouble finding a solution for that.
For the sake of simplicity, let us say that there will never be more than 30 Persons in the system (so Person.all is a small dataset) but each person will have upwards of 2000 comments (so Person.include(:comments) would be a large data set).
Parent association
class Person < ActiveRecord::Base
has_many :comments
end
Child association
class Comment < ActiveRecord::Base
belongs_to :person
end
I need to query for a list of Persons and include their comments, but I only need 5 of them.
I would like to do something like this:
Limited parent association
class Person < ActiveRecord::Base
has_many :comments
has_many :sample_of_comments, \
:class_name => 'Comment', :limit => 5
end
Controller
class PersonController < ApplicationController
def index
#persons = Person.include(:sample_of_comments)
end
end
Unfortunately, this article states: "If you eager load an association with a specified :limit option, it will be ignored, returning all the associated objects"
Is there any good way around this? Or am I doomed to chose between eager loading 1000s of unneeded ActiveRecord objects and an N+1 query? Also note that this is a simplified example. In the real world, I will have other associations with Person, in the same index action with the same issue as comments. (photos, articles, etc).
Regardless of what "that article" said, the issue is in SQL you can't narrow down the second sql query (of eager loading) the way you want in this scenario, purely by using a standard LIMIT
You can, however, add a new column and perform a WHERE clause instead
Change your second association to Person has_many :sample_of_comments, conditions: { is_sample: true }
Add a is_sample column to comments table
Add a Comment#before_create hook that assigns is_sample = person.sample_of_comments.count < 5