I have problems with query performance with Postgresql and Rails counting related models while retrieving data.
class MasterModel
# few fields, like name, description and such
has_and_belongs_to_many :business_models, class: 'BusinessModel'
end
class BusinessModel
# Lots of important information, many fields
has_and_belongs_to_many :master_models, class: 'MasterModel'
end
The use case in question being that business_model can be related to any amount of master_model so typically you should have a small amount of master_model, a great amount of business_model and a even bigger amount of many to many relationships.
When showing master_model index page, you can visualize its information and a delete button only enabled when there are no relationships, hence the reason why its important to count the relationship in its representation.
So I tried some ways to achieve this:
Includes relationship is incredibly slow in ActiveRecord but not in query time. At least it has no N+1.
MasterModel.includes(:business_models).limit(50).offset(0).each do |master|
master.business_models.size
end
No includes. We have N+1 but is incredibly fast as long as model pagination is reasonable.
MasterModel.limit(50).offset(0).each do |master|
master.business_models.size
end
Given that I only need to know if relationships exists or not I tried a select with exists. Single query and fast.
MasterModel.select(
:id,
:name,
:description.
'NOT EXISTS (
SELECT many.master_id
FROM many
WHERE many.master_id = master.id
) AS removable'
).limit(50).offset(0).each do |master|
master.business_models.removable
end
In the end, I chose the 3rd choice but I am not totally convinced. What would be the Rails way? Am I doing something wrong in the other cases?
If you would used has_many through association you would be able to use counter_cache but HABTM doesn't support counter_cache so that you can implement your own counter_cache
First of all you need to add new integer column to the master_models table called business_models_count
add_column :master_models, :business_models_count, :integer, default: 0
And add next code to your model MasterModel
class MasterModel
has_and_belongs_to_many :business_models, class: 'BusinessModel', before_add: :inc_business_models_count, before_remove: :dec_business_models_count
private
def inc_business_models_count(*)
self.increment!(:business_models_count)
end
def dec_business_models_count(*)
self.decrement!(:business_models_count)
end
end
And write some rake task which goes through MasterModel records and update counter for existing records.
It can be done like this:
MasterModel.find_each do |master|
master.increment!(:business_models_count, master.business_models.size)
end
And after that you will be able to get business_models_count of each MasterModel instance without N+1
MasterModel.limit(50).offset(0).each do |master|
master.business_models_count
end
Related
We have two models that belong to company: customer and vendor invoices. Currently they have their own index pages; generating a collection for pagination/sorting on these pages is as easy as current_company.customer_invoices.
class CustomerInvoice < ApplicationRecord
belongs_to :company
end
class VendorInvoice < ApplicationRecord
belongs_to :company
end
class Company < ApplicationRecord
has_many :customer_invoices, -> { order(due_date: :desc) }
has_many :vendor_invoices, -> { order(due_date: :desc) }
end
We now need to make a shared index page that will paginate and sort both kinds of invoices. STI seems like the obvious solution, but since they are functionally VERY different, and have minimal intersection in their schema columns, it strikes us as a bad use case. Is there any other option besides loading all of the records and sorting/paginating them in memory?
class Company
def invoices
(customer_invoices + vendor_invoices)
end
end
class InvoicesController < ApplicationController
def index
#invoices = current_company
.invoices
.sort_by(&:due_date)
.page(params[:whaveter_page])
end
end
As the number of invoices grows this will have super terrible memory performance :( .
To give you a best answer, one would need to know about the structure of your database, use cases etc.
But here are two approaches that you can consider:
Create separate table that will store data for both models. Such table should only contain columns required for filtering, sorting and data displayed on the combined list. and of course references to original rows in respectable tables. With such table it is pretty straightforward, you just query this table. Major disadvantage is that now you need to write to two tables when creating or updating invoice. But generating combined index will be super-fast. Technically you can move all the indexes to this table only and do other database optimizations.
You can use UNION statement and combine results from both tables into one. You need to select similar columns from both tables with same types. It will be slower than the first solution, but the advantage is that you don't have to maintain additional table. It will be created on demand. Writing union statement with active record may be a little bit challenging
I've got a Match model and a Team model.
I want to count how many goals a Team scores during the league (so I have to sum all the scores of that team, in both home_matches and away_matches).
How can I do that? What columns should I put into the matches and teams database tables?
I'd assume your Match model looks something like this:
belongs_to :home_team, class_name:"Team"
belongs_to :away_team, class_name:"Team"
attr_accessible :home_goal_count, :away_goal_count
If so, you could add a method to extract the number of goals:
def goal_count
home_matches.sum(:home_goal_count) + away_matches.sum(:away_goal_count)
end
Since this could be expensive (especially if you do it often), you might just cache this value into the team model and use an after_save hook on the Match model (and, if matches ever get deleted, then an after_destroy hook as well):
after_save :update_team_goals
def update_team_goals
home_team.update_attribute(:goal_count_cache, home_team.goal_count)
away_team.update_attribute(:goal_count_cache, away_team.goal_count)
end
Since you want to do this for leagues, you probably want to add a belongs_to :league on the Match model, a league parameter to the goal_count method (and its query), and a goal_count_cache_league column if you want to cache the value (only cache the most recently changed with my suggested implementation, but tweak as needed).
You dont put that in any table. Theres a rule for databases: Dont ever store data in your database that could be calculated from other fields.
You can calcuate that easyly using this function:
def total_goals
self.home_matches.collect(&:home_goals).inject(&:+)+self.away_matches.collect(&:away_goals).inject(&:+)
end
that should do it for you. If you want the mathes filtered for a league you can use a scope for that.
I am feeling a bit slow when it comes to rails and the Active Record associations... I have two tables.
Table = Rings
Table = Variations with foreign_key => "ring_id".
class Ring < ActiveRecord::Base
has_many :variations
end
class Variation < ActiveRecord::Base
belongs_to :ring
end
So in my "index/list" view i want to display all the rings, and both the variations, and i was thinking it would be possible to do this through one SQL query... however, i have tried the join and the include methods and i think i am just not understanding how they work properly.
So my question is, how would i write a query in my controller, that would pull my "title" and "value" column values from the "variations" and combine them into one simple object for easy looping? Or do i have to loop through all rings and look up the variation values during the loop?
thanks
In your controller:
#rings = Ring.includes(:variations).all
In index.html.erb:
#rings.each do |ring|
...
ring.variations.each do |variation|
...
end
end
The includes portion of the query will prevent Rails from repeatedly querying the database as you loop through and render your rings and variations in the view.
You need to use the includes method: Ring.inclues(:variations). Then the variation will be loaded along with the rings in a single SQL query.
For more info: http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
Live site: http://iatidata.heroku.com
Github: https://github.com/markbrough/IATI-Data
Based on aid information released through the IATI Registry: iatiregistry.org
I'm a bit of a Rails n00b so sorry if this is a really stupid question.
There are two key Models in this app:
Activity - which contains details
such as recipient country, funding
organisation
Transaction - which contains details such as how much money (value) was committed or disbursed (transaction_type), when, to whom, etc.
All Transactions nest under an Activity. Each Activity has multiple Transactions. They are connected together by activity_id. has_many :transactions and belongs_to :activity are defined in the Activity and Transaction Models respectively.
So: all of this works great when I'm trying to get details of transactions for a single activity - either when looking at a single activity (activity->show) or looping through activities on the all activities page (activity->index). I just call
#activities.each do |activity|
activity.transactions.each do |transaction|
transaction.value # do something like display it
end
end
But what I now really want to do is to get the sum of all transactions for all activities (subject to :conditions for the activity).
What's the best way to do this? I guess I could do something like:
#totalvalue = 0
#activities.each do |activity|
activity.transactions.each do |transaction|
#totalvalue = #totalvalue + transaction.value
end
end
... but that doesn't seem very clean and making the server do unnecessary work. I figure it might be something to do with the model...?! sum() is another option maybe?
This has partly come about because I want to show the total amount going to each country for the nice bubbles on the front page :)
Thanks very much for any help!
Update:
Thanks for all the responses! So, this works now:
#thiscountry_activities.each do |a|
#thiscountry_value = #thiscountry_value + a.transactions.sum(:value)
end
But this doesn't work:
#thiscountry_value = #thiscountry_activities.transactions.sum(:value)
It gives this error:
undefined method `transactions' for #<Array:0xb5670038>
Looks like I have some sort of association problem. This is how the models are set up:
class Transaction < ActiveRecord::Base
belongs_to :activity
end
class Activity < ActiveRecord::Base
has_and_belongs_to_many :policy_markers
has_and_belongs_to_many :sectors
has_many :transactions
end
I think this is probably quite a simple problem, but I can't work out what's going on. The two models are connected together via id (in Activity) and activity_id (in Transactions).
Thanks again!
Use Active Record's awesome sum method, available for classes:
Transaction.sum(:value)
Or, like you want, associations:
activity.transactions.sum(:value)
Let the database do the work:
#total_value = Transaction.sum(:value)
This gives the total for all transactions. If you have some activities already loaded, you can filter them this way:
#total_value = Transaction.where(:activity_id => #activities.map(&:id)).sum(:value)
You can do it with one query:
#total_value = Transaction.joins(:activity).where("activities.name" => 'foo').sum(:value)
My code was getting pretty messy summing up virtual attributes. So I wrote this little method to do it for me. You just pass in a collection and a method name as a string or symbol and you get back a total. I hope someone finds this useful.
def vsum collection, v_attr # Totals the virtual attributes of a collection
total = 0
collection.each { |collect| total += collect.method(v_attr).call }
return total
end
# Example use
total_credits = vsum(Account.transactions, :credit)
Of course you don't need this if :credit is a table column. You are better off using the built in ActiveRecord method above. In my case i have a :quantity column that when positive is a :credit and negative is a :debit. Since :debit and :credit are not table columns they can't be summed using ActiveRecord.
As I understood, you would like to have the sum of all values of the transaction table. You can use SQL for that. I think it will be faster than doing it the Ruby way.
select sum(value) as transaction_value_sum from transaction;
You could do
#total_value = activity.transactions.sum(:value)
http://ar.rubyonrails.org/classes/ActiveRecord/Calculations/ClassMethods.html
It occurred to me that if I have a has_many join, where the foreign model does not have a belongs_to, and so the join is one way, then I don't actually need a foreign key.
We could have a column, category_ids, which stores a marshaled Array of IDs which we can pass to find.
So here is an untested example:
class page < AR
def categories
Category.find(self.category_ids)
end
def categories<<(category)
# get id and append to category_ids
save!
end
def category_ids
#cat_ids ||= Marshal.load(read_attribute(:category_ids)) rescue []
end
def category_ids=(ids)
#cat_ids = ids
write_attribute(:category_ids, ids)
end
end
page.category_ids => [1,4,12,3]
page.categories => Array of Category
Is there accepted pattern for this already? Is it common or just not worth the effort?
Wouldn't performance suffer here when you are marshalling / unmarshalling?
I personally don't think this is worth the effort and what you are trying to do doesn't seem clear.
Actually this looks like a many-to-many mapping rather than a many-to-one, as there is no code that prevents a category from belonging to more than one page, surely you want something like:
create table categories_pages (
category_id integer not null references categories(id),
page_id integer not null references pages(id),
primary_key(category_id, page_id)
);
with either a has and belongs to many on both sides or has_many :through on both sides (depending on whether you want to store more stuff).
I agree with Omar that this doesn't seem to be worth the effort.
Your database no longer reflects your data model and isn't going to provide any help in enforcing this relationship. Also, you now have to keep your marshalled id array in sync with the Category table and enforce uniqueness across Pages if you want to restrict the relationship to has_many.
But I guess most importantly, what is the benefit of this approach? It is going to increase complexity and increase the amount of code you have to write.