Related
I have been eager loading ActiveStorage attachments as follows:
Journey.includes(created_by_user: [profile_picture_attachment: :blob])
We have been using variants and every since we upgraded to Rails 6.1 and enabled tracking Active Storage variants in database, we notice n+1 queries in the logs because of a loop as follows:
Journey.includes(created_by_user: [profile_picture_attachment: :blob]).each do |j|
j.created_by_user.profile_picture.variant(resize_to_fill: [32, 32]).processed
end
Journey Load (0.8ms) SELECT "journeys".* FROM "journeys"
User Load (0.7ms) SELECT "users".* FROM "users" WHERE "users"."id" = $1 [["id", 607]]
ActiveStorage::Attachment Load (0.6ms) SELECT "active_storage_attachments".* FROM "active_storage_attachments" WHERE "active_storage_attachments"."record_type" = $1 AND "active_storage_attachments"."name" = $2 AND "active_storage_attachments"."record_id" = $3 [["record_type", "User"], ["name", "profile_picture"], ["record_id", 607]]
ActiveStorage::Blob Load (0.6ms) SELECT "active_storage_blobs".* FROM "active_storage_blobs" WHERE "active_storage_blobs"."id" = $1 [["id", 144]]
ActiveStorage::VariantRecord Load (0.5ms) SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1 AND "active_storage_variant_records"."variation_digest" = $2 LIMIT $3 [["blob_id", 144], ["variation_digest", "k9S9jJS87DbFgXD1sW9j5XkOr1c="], ["LIMIT", 1]]
ActiveStorage::VariantRecord Load (0.5ms) SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1 AND "active_storage_variant_records"."variation_digest" = $2 LIMIT $3 [["blob_id", 144], ["variation_digest", "k9S9jJS87DbFgXD1sW9j5XkOr1c="], ["LIMIT", 1]]
ActiveStorage::VariantRecord Load (0.5ms) SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1 AND "active_storage_variant_records"."variation_digest" = $2 LIMIT $3 [["blob_id", 144], ["variation_digest", "k9S9jJS87DbFgXD1sW9j5XkOr1c="], ["LIMIT", 1]]
ActiveStorage::VariantRecord Load (0.5ms) SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1 AND "active_storage_variant_records"."variation_digest" = $2 LIMIT $3 [["blob_id", 144], ["variation_digest", "k9S9jJS87DbFgXD1sW9j5XkOr1c="], ["LIMIT", 1]]
ActiveStorage::VariantRecord Load (0.5ms) SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1 AND "active_storage_variant_records"."variation_digest" = $2 LIMIT $3 [["blob_id", 144], ["variation_digest", "k9S9jJS87DbFgXD1sW9j5XkOr1c="], ["LIMIT", 1]]
I tried to eager load with the following but it doesn't seem to work:
Journey.includes(created_by_user: [profile_picture_attachment: { blob: :variant_records }])
Has anyone tried eager loading the tracked variant records to share your ideas?
Turns out there is a pull request addressing exactly this. For anyone looking for a solution, please follow the merge request.
https://github.com/rails/rails/pull/37901
It eventually winds its way to a secondary PR, which allows eager loading the stored variants. It's now merged into future versions of Rails, here was the code they used to eager load variants:
https://github.com/rails/rails/pull/40842/files
I'm building a GraphQL API with Ruby on Rails and the graphql gem. Now I have some n:m relations, like Projects have many Users and Users have many Projects. My models are like:
# /app/models/project.rb
has_many :project_assignments
has_many :project_managers, through: :project_assignments, source: :user
# /app/models/project_assignment.rb
belongs_to :project
belongs_to :user
# /app/models/user.rb
has_many :project_assignments
has_many :projects, through: :project_assignments
Now I want to query all projects and their corresponding project managers with a query like this:
query {
projects {
edges {
node {
id
projectManagers {
edges {
node {
id
}
}
}
}
}
}
}
And my resolvers are basically like Project.all and each Project calls project.projectManagers, which results in hundreds of queries:
Project Load (3.3ms) SELECT "projects".* FROM "projects"
User Load (1.5ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 2]]
User Load (0.7ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 3]]
User Load (1.1ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 4]]
User Load (1.0ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 5]]
User Load (0.8ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 6]]
...
I've added the bullet gem, but there is no warning about a missing eager loading. In fact, If I use Project.all.includes(:project_managers), I get the query that I want (SELECT "project_assignments".* FROM "project_assignments" WHERE "project_assignments"."project_id" IN ($1, $2, $3, $4, $5, $6, $7, $8, ...)), but the User queries fires anyhow:
User Load (0.5ms) SELECT "users".* FROM "users" WHERE "users"."id" IN ($1, $2, $3) [["id", 3], ["id", 2], ["id", 1]]
User Load (0.8ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 2]]
User Load (0.9ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 3]]
User Load (0.8ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 4]]
User Load (0.7ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 5]]
User Load (0.7ms) SELECT "users".* FROM "users" INNER JOIN "project_assignments" ON "users"."id" = "project_assignments"."user_id" WHERE "project_assignments"."project_id" = $1 [["project_id", 6]]
...
Is there anything that I can do the preload the users?
I tried eager_load also, but the result is basically the same (tried another example with an normal has_many association (no :through):
SQL (1.3ms) SELECT "projects"."id" AS t0_r0, "projects"."title" AS t0_r1, "projects"."number" AS t0_r2, "projects"."description" AS t0_r3, "projects"."deadline" AS t0_r4, "projects"."archived" AS t0_r5, "projects"."customer_id" AS t0_r6, "projects"."rate_type" AS t0_r7, "projects"."daily_rate" AS t0_r8, "projects"."service_rates" AS t0_r9, "projects"."budget_type" AS t0_r10, "projects"."budget_rate" AS t0_r11, "projects"."created_at" AS t0_r12, "projects"."updated_at" AS t0_r13, "projects"."status" AS t0_r14, "projects"."slug" AS t0_r15, "project_labels"."id" AS t1_r0, "project_labels"."title" AS t1_r1, "project_labels"."description" AS t1_r2, "project_labels"."color" AS t1_r3, "project_labels"."project_id" AS t1_r4, "project_labels"."created_at" AS t1_r5, "project_labels"."updated_at" AS t1_r6 FROM "projects" LEFT OUTER JOIN "project_labels" ON "project_labels"."project_id" = "projects"."id" WHERE "projects"."id" IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) [["id", 1], ["id", 2], ["id", 3], ["id", 4], ["id", 5], ["id", 6], ["id", 7], ["id", 8], ["id", 9], ["id", 10]]
ProjectLabel Load (0.3ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 10]]
ProjectLabel Load (0.5ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 2]]
ProjectLabel Load (0.7ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 5]]
ProjectLabel Load (0.5ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 8]]
ProjectLabel Load (0.5ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 6]]
ProjectLabel Load (0.6ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 4]]
ProjectLabel Load (0.4ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 1]]
ProjectLabel Load (0.3ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 3]]
ProjectLabel Load (0.3ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 9]]
ProjectLabel Load (0.5ms) SELECT "project_labels".* FROM "project_labels" WHERE "project_labels"."project_id" = $1 [["project_id", 7]]
You can do all sorts of things with has_many including includes which should help with N+1 queries
has_many :project_assignments, -> { includes(:projects) }
Project.all.includes(:project_managers)
# Project.includes(:project_managers) # shorthand of above
... would include project_managers automatically in the SQL depending on the query.
To also include project_assignment.user, then just merge them:
Project.includes(:project_managers, project_assignments: :user)
# probably below is equivalent of above (but not sure)
# just because `project_managers` association is also going "through" :project_assignments
Project.includes(project_assignments: :user)
See "Loading nested relationships" here in the docs
I got a problem with ActiveStorage, Currently I have has_on_attachedand has_many_attached relation on my Model
Every time I call my model it loads the relation as below:
ActiveStorage::Attachment Load (4.2ms) SELECT "active_storage_attachments".* FROM "active_storage_attachments" WHERE "active_storage_attachments"."record_id" = $1 AND "active_storage_attachments"."record_type" = $2 AND "active_storage_attachments"."name" = $3 LIMIT $4 [["record_id", 4934], ["record_type", "User"], ["name", "profile_picture"], ["LIMIT", 1]]
ActiveStorage::Attachment Exists (0.9ms) SELECT 1 AS one FROM "active_storage_attachments" WHERE "active_storage_attachments"."record_id" = $1 AND "active_storage_attachments"."record_type" = $2 AND "active_storage_attachments"."name" = $3 LIMIT $4 [["record_id", 4934], ["record_type", "User"], ["name", "home_pictures"], ["LIMIT", 1]]
How can I disable this behaviour by default ?
You can use Model.with_attached_images.find(:id) to avoid the N+1
https://github.com/rails/rails/tree/master/activestorage#examples
I'm not sure if you can disable eager loading the attachments, but the above should help clean up a little bit.
I met a N+1 issue in this situation:
Library has many Programs. Now I want to get all the programs located a certain country, so I have a code:
country = "US"
programs = #libraries.includes(:programs).map do |library|
library.programs.where(country: country)
end
But now there is N+1 problem:
Program Load (0.8ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 15], ["country", "US"]]
Program Load (0.4ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 73], ["country", "US"]]
Program Load (0.5ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 27], ["country", "US"]]
Program Load (0.3ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 177], ["country", "US"]]
Program Load (0.3ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 38], ["country", "US"]]
Program Load (0.4ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 51], ["country", "US"]]
Program Load (0.6ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 18], ["country", "US"]]
Program Load (0.3ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 20], ["country", "US"]]
Program Load (0.5ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 42], ["country", "US"]]
Program Load (0.5ms) SELECT "programs".* FROM "programs" WHERE "programs"."library_id" = $1 AND "programs"."country" = $2 [["library_id", 39], ["country", "US"]]
Update:
My purpose is not to just filter the programs, but to use it. For example:
programs = #libraries.includes(:programs).each do |library|
if library.programs.where(country: country).size < 5
puts "US programs are less than 5 so you can still add"
end
end
Does anyone know how to solve the N+1 problem?
You can chain the where query to the includes like below
programs = #libraries.includes(:programs).where(programs: {country: country})
which should solve the N+1 problem.
See specifying-conditions-on-eager-loaded-associations
Update #1:
You can simply do it like this
programs = #libraries.includes(:programs).where(programs: {country: country}).size < 5 #returns true or false
if programs #true
puts "US programs are less than 5 so you can still add"
else #false
#your code
end
Update #2:
This should do
programs_size = #libraries.includes(:programs).where(programs: {country: country}).map { |library| library.programs.size }
Which would perform only one query and returns the size of each library.programs matching that condition as array something like below
=> [5, 4, 7, 4, 6, 2, 1]
Now you can iterate over the programs_size array and perform the logic
programs_size.each do |ps|
if ps < 5 #true
puts "US programs are less than 5 so you can still add"
else #false
#your code
end
end
I have an object called message which belongs to carrier, company, country
I allowing for a bulk insert of users via a CSV - what i want to do before is make sure every row is valid before i import it (so that i can inform the user prior to the beginning the import)
So i have created a method that loops through all new data and does Message.new(PARAMS_IN_HERE) and then call .valid? on it, which is fine and achieves the desired results.
However, when i look in the logs i see loads of queries like this
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.3ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Company Load (0.2ms) SELECT "companies".* FROM "companies" WHERE "companies"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Country Load (0.2ms) SELECT "countries".* FROM "countries" WHERE "countries"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Carrier Load (0.2ms) SELECT "carriers".* FROM "carriers" WHERE "carriers"."id" = $1 LIMIT $2 [["id", 1], ["LIMIT", 1]]
Which is obviously quite wasteful as one its doing the same query multiple times. Is there a way to make rails cache the value/preload when it needs to look for/ or just stop it happening?
This is what my message class look like
class Message < ApplicationRecord
belongs_to :company
belongs_to :carrier
belongs_to :country
before_validation :set_default_details, on: :create
private
def set_default_details
if self.user.present?
self.carrier_id = self.user.company.tariff.carrier_id
self.country_id = self.user.country_id
self.company_id = self.user.company_id
end
end
end
Your validation references self.user but that's not defined in the model snippet you posted.
If your validations refer to to association objects, like self.company, give those objects to the constructor: Message.new(company: company) rather than Message.new(company_id: company.id). If necessary look up all dependent objects ahead of time - this can be done with a single query - and store id -> object mapping in a Hash.
Similarly you can have validations reference fields like company_id but it's probably better to use association objects everywhere.