I have a rails API that currently has quite a few N+1 queries that I'd like to reduce.
As you can see it's going through quite a few loops before returning the data.
The relationships are as follows:
Company Model
class Company < ApplicationRecord
has_many :jobs, dependent: :destroy
has_many :contacts, dependent: :destroy
has_many :listings
end
Job Model
class Job < ApplicationRecord
belongs_to :company
has_many :listings
has_and_belongs_to_many :technologies
has_and_belongs_to_many :tools
scope :category, -> ( category ) { where category: category }
end
Listing Modal
class Listing < ApplicationRecord
belongs_to :job, dependent: :destroy
belongs_to :company, dependent: :destroy
scope :is_active, -> ( active ) { where is_active: active }
end
Job Serializer
class SimpleJobSerializer < ActiveModel::Serializer
attributes :id,
:title,
:company_name,
attribute :technology_list, if: :technologies_exist
attribute :tool_list, if: :tools_exist
def technology_list
custom_technologies = []
object.technologies.each do |technology|
custom_technology = { label: technology.label, icon: technology.icon }
custom_technologies.push(custom_technology)
end
return custom_technologies
end
def tool_list
custom_tools = []
object.tools.each do |tool|
custom_tool = { label: tool.label, icon: tool.icon }
custom_tools.push(custom_tool)
end
return custom_tools
end
def tools_exist
return object.tools.any?
end
def technologies_exist
return object.technologies.any?
end
def company_name
object.company.name
end
end
Current query in controller
Job.eager_load(:listings).order("listings.live_date DESC").where(category: "developer", listings: { is_active: true }).first(90)
I've tried to use eager_load to join the listings to the Jobs to make the request more efficient but i'm unsure how to handle this when some of the n+1 queries are coming from inside the serializer as it tries to look at tools and technologies.
Any help would be much appreciated!
You might was well eager load tools and technologies since you know that the serializer is going to use them:
Job.eager_load(:listings, :tools, :technologies)
.order("listings.live_date DESC")
.where(category: "developer", listings: { is_active: true })
.first(90)
After that you really need to refactor that serializer. #each should only be used when you are only interested in the side effects of the iteration and not the return value. Use #map, #each_with_object, #inject etc. These calls can be optimized. return is implicit in ruby so you only explicitly return if you are bailing early.
class SimpleJobSerializer < ActiveModel::Serializer
# ...
def tool_list
object.tools.map { |t| { label: tool.label, icon: tool.icon } }
end
# ...
end
Try nested preload:
Job.preload(:technologies, :tools, company: :listings).order(...).where(...)
Related
I'm trying to control the nested attributes in case of duplications, do find the row and use it instead of creating a new one, it works fine lower nested level which is the meals.
however if I use it the commented code in the plan.rb ( you can check it below ) makes the meals blank, as if I'm not passing any meals inside my request, any idea about this?
Plan.rb
class Plan < ApplicationRecord
has_and_belongs_to_many :meals
has_and_belongs_to_many :days
has_one_attached :image, dependent: :destroy
validate :acceptable_image
accepts_nested_attributes_for :days, reject_if: ->(object) { object[:number].blank? }
#! this is causing meals to not save
# # before_validation :find_days
# def find_days
# self.days = self.days.map do |object|
# Day.where(number: object.number).first_or_initialize
# end
# end
#!
end
Day.rb
class Day < ApplicationRecord
has_and_belongs_to_many :meals
has_and_belongs_to_many :plans
accepts_nested_attributes_for :meals, reject_if: ->(object) { object[:name].blank? }
before_validation :find_meals
def find_meals
self.meals = self.meals.map do |object|
Meal.where(name: object.name).first_or_initialize
end
end
end
Meal.rb
class Meal < ApplicationRecord
has_and_belongs_to_many :plans
has_and_belongs_to_many :days
end
This is how I permit my params
def plan_params
params.require(:plan).permit(:name, :monthly_price, :image_url, days_attributes: [:number, meals_attributes: [:name, :calories, :protein, :fat, :carbohydrates, :categorie]])
end
I'm sorry for making this long, but I wanted to give as many details as possible.
Since you're mapping the self.days association, Day.where(number: object.number).first_or_initialize replaces the days array with Day objects without any meal attributes.
You need to do something like this instead, inside the map block:
day = Day.where(number: object.number).first_or_initialize
day.attributes = object.attributes
# or similar, to copy the nested attributes provided by the request
day
I want to be able to find unpopulated hives, but don't find any solution.
Can you help me please ?
The goal is to be able to do Hive.unpopulated
The main problem is the most_recent, butins ok for me to work with a raw SQL, but I don't find the right query.
Here are my classes :
class Hive < ApplicationRecord
has_many :moves, dependent: :destroy
has_many :yards, through: :moves
has_many :populations, -> { where(:most_recent => true) }
has_many :colonies, through: :populations
validates :name, uniqueness: true
def hive_with_colony
"#{name} (colony #{if self.colonies.count > 0 then self.colonies.last.id end})"
end
def self.populated
Hive.joins(:populations)
end
def self.unpopulated
end
end
class Population < ApplicationRecord
belongs_to :hive
belongs_to :colony
after_create :mark_most_recent
before_create :mark_end
class Colony < ApplicationRecord
has_many :populations, -> { where(:most_recent => true) }
has_many :hives, through: :populations
has_many :visits
has_many :varroas
has_many :most_recents_populations, -> { where(:most_recent => true) }, :class_name => 'Population'
scope :last_population_completed, -> { joins(:populations).where('populations.most_recent=?', true)}
I think you can do a simple query to select Hives which are not in populated list, so:
def self.unpopulated
where.not(id: populated.select(:id))
end
Another option is a LEFT OUTER JOIN and picking the lines that have no population id set on the right side.
def self.unpopulated
left_outer_joins(:populations).where(populations: { id: nil })
end
It depends on your data if Thanh's version (which compares a potentially huge list of ids) or this version (which makes a sightly more complex join but doesn't need to compare against a list of ids) is more performant.
I'm trying to build a Rails API with the following JSON structure:
{ team: "team_name",
players: players: [
{
name: "player_one",
contracts: [
{
start_date: '7/1/2017',
seasons: [
{
year: 2017,
salary: 1000000
}
]
}
]
},
{
name: "player_two"
}
]
}
My models are set up as follows:
class Team < ApplicationRecord
has_many :players
end
class Player < ApplicationRecord
belongs_to :team
has_many :contracts
end
class Contract < ApplicationRecord
belongs_to :player
has_many :seasons
end
class Season < ApplicationRecord
belongs_to :contract
end
I'm currently using the following code, but I'd like to clean this up using ActiveModel Serializers (Or other clean solutions)
def show
#team = Team.find(params[:id])
render json: #team.as_json(
except: [:id, :created_at, :updated_at],
include: {
players: {
except: [:created_at, :updated_at, :team_id],
include: {
contracts: {
except: [:id, :created_at, :updated_at, :player_id],
include: {
seasons: {
except: [:id, :contract_id, :created_at, :updated_at]
}
}
}
}
}
}
)
end
Additionally, the array items are showing in descending order by ID, I'm hoping to reverse that. I'm also hoping to order the players by the first contract, first season, salary.
Active Model Serializers is a nice gem to fit into your needs. Assuming you would use 0-10-stable branch (latest: 0.10.7), you can go in this way:
Serializers (Doc)
# app/serializers/team_serializer.rb or app/serializers/api/v1/team_serializer.rb (if your controllers are in app/controllers/api/v1/ subdirectory)
class TeamSerializer < ActiveModel::Serializer
attributes :name # list all desired attributes here
has_many :players
end
# app/serializers/player_serializer.rb
class PlayerSerializer < ActiveModel::Serializer
attributes :name # list all desired attributes here
has_many :contracts
end
# app/serializers/contract_serializer.rb
class ContractSerializer < ActiveModel::Serializer
attributes :start_date # list all desired attributes here
has_many :seasons
end
# app/serializers/season_serializer.rb
class SeasonSerializer < ActiveModel::Serializer
attributes :year, :salary # list all desired attributes here
end
TeamController
def show
#team = Team
.preload(players: [contracts: :seasons])
.find(params[:id])
render json: #team,
include: ['players.contracts.seasons'],
adapter: :attributes
end
Note 1: Using preload (or includes) will help you eager-load multiple associations, hence saving you from N + 1 problem.
Note 2: You may wish to read the details of the include option in render method from Doc
I think your approach is right. Looks clean as it could be.
You can use also named_scopes on your Model.
See docs here.
I've a model that has a nested model of skills. Its a common has_many example. Elastic search is indexing the skills as an array of strings.
My question is, I am attempting to match on those skills by way of two different inputs.
Required skills and bonus skills.
So if I have two query terms one for required and one for bonus, I want to query the skills attribute with required input, if none found, query with the bonus input.
I'm using elasticsearch-rails gem. Didn't think I needed to post any code as this is more theory.
UPDATE
class Profile
has_many :skills
...
end
class Skill
belongs_to :profile
end
Mappings
settings index: { number_of_shards: 1, number_of_replicas: 0 } do
...
mapping dynamic: 'false' do
indexes :skills, analyzer: 'keyword'
end
...
end
Overriden as_json
def as_indexed_json(options={})
hash = self.as_json(
include: {location: { methods: [:coordinates], only: [:coordinates] },
locations_of_interest: { methods: [:coordinates], only: [:coordinates]}
})
hash['skills'] = self.skills.map(&:name)
hash['interests'] = self.interests.map(&:name)
hash
end
I guess in essence i'm looking to perform the reverse of a multi_match on multiple fields and boosting one but instead searching one field with multiple inputs (required and bonus) and depending no the results of required search with bonus input. Does this makes things more clear?
This is my query so far, first attempt.
if options[:required_skills].present? && options[:bonus_skills].present?
bool do
must do
term skills: options[:required_skills]
end
should do
term skills: options[:bonus_skills]
end
end
end
class SkillContainer < ActiveRecord::Base
has_many :skill_links, dependent: :destroy, inverse_of: :skill_container
has_many :skills, through: :skill_links
has_many
end
##################################
create_table :skill_link do |t|
t.references :skill
t.references :skill_container
t.boolean :required
t.boolean :bonus
end
##################################
class SkillLink
belongs_to :skill_container
belongs_to :skill
scope :required, -> {
where(required: true)
}
scope :bonus, -> {
where(bonus: true)
}
end
class Skill < ActiveRecord::Base
has_many :skill_links, dependent: :destroy, inverse_of: :skill
has_many :skills, through: :skill_links
end
#required skills from any skill container
SkillContainer.last.skills.merge(SkillLink.required)
#bonus skills from any skill container
SkillContainer.last.skills.merge(SkillLink.bonus)
scopes can be combined with your elastic search
I read this interesting article about Using Polymorphism to Make a Better Activity Feed in Rails.
We end up with something like
class Activity < ActiveRecord::Base
belongs_to :subject, polymorphic: true
end
Now, if two of those subjects are for example:
class Event < ActiveRecord::Base
has_many :guests
after_create :create_activities
has_one :activity, as: :subject, dependent: :destroy
end
class Image < ActiveRecord::Base
has_many :tags
after_create :create_activities
has_one :activity, as: :subject, dependent: :destroy
end
With create_activities defined as
def create_activities
Activity.create(subject: self)
end
And with guests and tags defined as:
class Guest < ActiveRecord::Base
belongs_to :event
end
class Tag < ActiveRecord::Base
belongs_to :image
end
If we query the last 20 activities logged, we can do:
Activity.order(created_at: :desc).limit(20)
We have a first N+1 query issue that we can solve with:
Activity.includes(:subject).order(created_at: :desc).limit(20)
But then, when we call guests or tags, we have another N+1 query problem.
What's the proper way to solve that in order to be able to use pagination ?
Edit 2: I'm now using rails 4.2 and eager loading polymorphism is now a feature :)
Edit: This seemed to work in the console, but for some reason, my suggestion of use with the partials below still generates N+1 Query Stack warnings with the bullet gem. I need to investigate...
Ok, I found the solution ([edit] or did I ?), but it assumes that you know all subjects types.
class Activity < ActiveRecord::Base
belongs_to :subject, polymorphic: true
belongs_to :event, -> { includes(:activities).where(activities: { subject_type: 'Event' }) }, foreign_key: :subject_id
belongs_to :image, -> { includes(:activities).where(activities: { subject_type: 'Image' }) }, foreign_key: :subject_id
end
And now you can do
Activity.includes(:part, event: :guests, image: :tags).order(created_at: :desc).limit(10)
But for eager loading to work, you must use for example
activity.event.guests.first
and not
activity.part.guests.first
So you can probably define a method to use instead of subject
def eager_loaded_subject
public_send(subject.class.to_s.underscore)
end
So now you can have a view with
render partial: :subject, collection: activity
A partial with
# _activity.html.erb
render :partial => 'activities/' + activity.subject_type.underscore, object: activity.eager_loaded_subject
And two (dummy) partials
# _event.html.erb
<p><%= event.guests.map(&:name).join(', ') %></p>
# _image.html.erb
<p><%= image.tags.first.map(&:name).join(', ') %></p>
This will hopefully be fixed in rails 5.0. There is already an issue and a pull request for it.
https://github.com/rails/rails/pull/17479
https://github.com/rails/rails/issues/8005
I have forked rails and applied the patch to 4.2-stable and it works for me. Feel free to use my fork, even though I cannot guarantee to sync with upstream on a regular basis.
https://github.com/ttosch/rails/tree/4-2-stable
You can use ActiveRecord::Associations::Preloader to preload guests and tags linked, respectively, to each of the event and image objects that are associated as a subject with the collection of activities.
class ActivitiesController < ApplicationController
def index
activities = current_user.activities.page(:page)
#activities = Activities::PreloadForIndex.new(activities).run
end
end
class Activities::PreloadForIndex
def initialize(activities)
#activities = activities
end
def run
preload_for event(activities), subject: :guests
preload_for image(activities), subject: :tags
activities
end
private
def preload_for(activities, associations)
ActiveRecord::Associations::Preloader.new.preload(activities, associations)
end
def event(activities)
activities.select &:event?
end
def image(activities)
activities.select &:image?
end
end
image_activities = Activity.where(:subject_type => 'Image').includes(:subject => :tags).order(created_at: :desc).limit(20)
event_activities = Activity.where(:subject_type => 'Event').includes(:subject => :guests).order(created_at: :desc).limit(20)
activities = (image_activities + event_activities).sort_by(&:created_at).reverse.first(20)
I would suggest adding the polymorphic association to your Event and Guest models.
polymorphic doc
class Event < ActiveRecord::Base
has_many :guests
has_many :subjects
after_create :create_activities
end
class Image < ActiveRecord::Base
has_many :tags
has_many :subjects
after_create :create_activities
end
and then try doing
Activity.includes(:subject => [:event, :guest]).order(created_at: :desc).limit(20)
Does this generate a valid SQL query or does it fail because events can't be JOINed with tags and images can't be JOINed with guests?
class Activity < ActiveRecord::Base
self.per_page = 10
def self.feed
includes(subject: [:guests, :tags]).order(created_at: :desc)
end
end
# in the controller
Activity.feed.paginate(page: params[:page])
This would use will_paginate.