I'm working on a project that uses ActiveAdmin for its administration backend.
I have two models, a Book model which has_many Products. When I try to access the products index view in ActiveAdmin, it seems to try to load the full books table into memory (there are about 1.5 million books in my database). CPU usage goes up to 100% and memory usage spikes to gigabytes.
Turning on mysql logging confirms that this is what happens when this view is called:
17 Query SELECT `books`.* FROM `books`
As far as I can tell this happens before any attempt to load the products.
To figure out this issue I stripped the models down to their bare bones:
class Product < ActiveRecord::Base
belongs_to :book
end
class Book < ActiveRecord::Base
has_many :products
end
I also reduced the AA definition to its most basic form:
ActiveAdmin.register Product do
end
Is this normal for ActiveAdmin? It doesn't seem like desirable behavior.
For anyone dealing with this same issue, I finally traced it to the automatically generated sidebar in ActiveAdmin. This includes a search field that includes a select box for all associated records.
If you have an associated table with over a million records like I do AA will happily attempt to insert the entire table into the select box.
The answer was to include some custom filters in the AA definition for products like so:
ActiveAdmin.register Product do
filter :title
end
That way the association won't be included (unless you specify it yourself.)
A better approach now is to use remove_filter for the particular attribute or relationship:
Or you can also remove a filter and still preserve the default
filters:
preserve_default_filters!
remove_filter :id
https://activeadmin.info/3-index-pages.html
Related
Trying to avoid n+1 query
I'm working on a web based double entry accounting application that has the following basic models;
ruby
class Account < ApplicationRecord
has_many :splits
has_many :entries, through: :splits
end
class Entry < ApplicationRecord
has_many :splits, -> {order(:account_id)}, dependent: :destroy, inverse_of: :entry
attribute :amount, :integer
attribute :reconciled
end
class Split < ApplicationRecord
belongs_to :entry, inverse_of: :splits
belongs_to :account
attribute :debit, :integer
attribute :credit, :integer
attribute :transfer, :string
end
This is a fairly classic Accounting model, at least it is patterned after GnuCash, but it leads to somewhat complex queries. (From ancient history this is pretty much a 3rd normal form structure!)
First Account is a hierarchal tree structure (an Account belongs to a parent (except ROOT) and my have many children, children may also have many children, which I call a family). Most of these relations are covered in the Account model and optimized as much as you can a recursive structure.
An Account has many Entries(transactions) and entries must have at least two Splits that the sum of the Amount attribute(or Debits/Credits) must equal 0.
The primary use of this structure is to produce Ledgers, which is just a list of Entries and their associated Splits usually filtered by a date range. This is fairly simple if the account has no Family/Children
ruby
# self = a single Account
entries = self.entries.where(post_date:#bom..#eom).includes(:splits).order(:post_date,:numb)
It get more complex if you want a ledger of an account that has many children (I want a Ledger of all Current Assets)
ruby
def self.scoped_acct_range(family,range)
# family is a single account_id or array of account_ids
Entry.where(post_date:range).joins(:splits).
where(splits: {account_id:family}).
order(:post_date,:numb).distinct
end
While this works, I guess I have an n+1 query because if I use includes instead of joins I won't get all the splits for an Entry, only those in the family - I want all splits. That means it reloads(queries) the splits in the view. Also distinct is needed because a split could reference an account multiple time.
My question is there a better way to handle this three model query?
I threw together a few hacks, one going backwards from splits:
ruby
def self.scoped_split_acct_range(family,range)
# family is a single account_id or array of account_ids
# get filtered Entry ids
entry_ids = Split.where(account_id:family).
joins(:entry).
where(entries:{post_date:range}).
pluck(:entry_id).uniq
# use ids to get entries and eager loaded splits
Entry.where(id:eids).includes(:splits).order(:post_date,:numb)
end
This also works and by the ms reported in log, may even be faster. Normal use of either would be looking at 50 or so Entries for a month, but then you can filter a years worth of transactions - but you get what you asked for. For normal use, an ledger for a month is about 70ms, Even a quarter is around 100ms.
I've used a few attributes in both Splits and Accounts that got rid a few view level queries. Transfer is basically concatenated Account names going up the tree.
Again, just looking to see if I'm missing something and there is a better way.
Using a nested select is the proper option IMO.
You can optimize your code with the nested select to use the following:
entry_ids = Entry.where(post_date: range)
.joins(:splits)
.where(post_date: range, splits: { account_id: family })
.select('entries.id')
.distinct
Entry.where(id: entry_ids).includes(:splits).order(:post_date,:numb)
This will generate a single SQL statement with a nested select, instead of having 2 SQL queries: 1 to get the Entry ids and pass it to Rails and 1 other query to select entries based on those ids.
The following gem, developed by an ex-colleague, can help you deal with this kind of stuff: https://github.com/MaxLap/activerecord_where_assoc
In your case, it would enable you to do the following:
Entry.where_assoc_exists(:splits, account_id: 123)
.where(post_date: range)
.includes(:splits)
.order(:post_date, :numb)
Which does the same thing as I suggested but behind the scene.
I'm seeking brainstorming input for a Rails design issue I've run across.
I have simple Book reviews feature. There's a Book class, a User class, and a UserBook class (a.k.a., reviews and ratings).
class User < ActiveRecord::Base
has_many :user_books
end
# (book_id, user_id, review data...)
class UserBook < ActiveRecord::Base
belongs_to :user
belongs_to :book
end
In the corresponding book controller for the "show" book action, I need to load the book data along with the set of book reviews. I also need to find out whether the current user (if there is one) has contributed to those reviews.
I'm currently running two queries, Book.where(...) and UserBook.where(...), and placing the results into two separate objects passed on to the view. Now, while I could run a third query to find whether the user is among those reviews (on UserBook), I'd prefer to pull that from the #reviews result set. But do I do that in the controller, or in the view?
Also worth noting is that in the view I have to draw Add vs Update review buttons accordingly, with their corresponding ajax URLs. So I'd prefer to know it before I start looping through a result set.
If I detect this in the controller though, I'll need three instance variables passed in, which I understand is considered distasteful in Rails land. Not sure how to avoid this.
Suggestions appreciated.
This smells like a case for has_many through, which is designed for cases where you want to access the data of a third table through an intermediate table (in this case, UserBook)
Great explanation of has_many :through here
Might look something like this:
class User < ActiveRecord::Base
has_many :user_books
has_many :users, through: :books
end
Then you can simply call
#user = User.find(x)
#user.user_books` # (perhaps aliased as `User.find(x).reviews`)
and
#user.books
to get a list of all books associated with the User.
This way, you can gain access to all of the information you need for a particular user with a single #user instance variable.
PS - You'll want to take a look at the concept of Eager loading, which will prevent you from making extraneous database calls while fetching all of this information.
I have a Rails app that keeps count of User, Comments for many Movie_Categories. The models look like this:
class User < ActiveRecord::Base
has_many :comments
end
class Movie < ActiveRecord::Base
belongs_to :movie_category
has_many :comments
end
class MovieCategory < ActiveRecord::Base
has_many :movies
end
class Comment < ActiveRecord::Base
belongs_to :movie
belongs_to :user
end
I can of course find User's Comments count by MovieCategory doing something like this for each MovieCategory:
#user.comment.where("movie_category_id =?", movie_category_id)
However, it puts a lot of load on the server, if I need to make this call too frequently, so I was thinking about doing the calculation once per hour (in a background job) for all Users for all Movie_Categories and then storing the counts in the User table, each Movie Category with it's own column. That way, I don't have to run the calculations for each user as frequently and can just access the counts from the User table.
The thing I'm wondering is if there is a more dry way to do this since I'm not sure when my Movie_Categories will stop growing (and with each time comes a new table field). I also thought about caching the User show views (where these counts appear), but even so, if I don't have these columns in the User table then it seems like each time a new User page is loaded (or cached expired) it will have to run through calculating all of this for the User comment counts again.
Is there a better approach for the issue that I'm facing with not putting too much burden on the server?
Given your comments about being in development at the moment, I would say don't worry about it until you have to worry about it! However, if you want to plan ahead my suggestions would be to go with fragment caching and indexes on the foreign keys.
If your site grows to the size you're talking about, running migrations to add additional fields to your users table could take significant time.
I note you've referred to caching in your question so assume you're broadly familiar with it but given a view such as:
<ul>
<li>Action Movies: 23 Comments</li>
<li>Comedy Movies: 14 Comments</li>
</ul>
You would wrap this in a cache block:
<% cache "user-#{user.id}-comments", #user.comments.last.created_at.to_i do %>
...
<% end %>
This will cache the fragment displaying the counts and auto expire it each time a new comment is posted by that user. You could really get into granular detail by caching each <li> and expiring only when a comment is posted in that category but it might be overkill at an early stage.
For the index on the foreign keys, you can add this in a migration using the syntax:
add_index :comments, :movie_category_id
I don't think the query you run is that bad but you never know until you hit production and scale quite what effect it will have.
You have a set of related models created through a scaffold e.g. a house, which has many rooms, which each have many windows, which each has a selection of locks.
These resources are already full of data i.e. someone has entered all the information, such as: a room called 'kitchen' has various windows associated with it and these windows each have five different locks associated with them.
Someone comes along and says:
Can you create a form that lets someone create a new project where they can select the different rooms, windows and then specify the locks that they would like for that project? (these are already in the system, nothing new to add, just the associations to a new project)
This sounds like a nested form but I have wasted a lot of time trying to solve this - there are many levels of nesting, which make this tricky. Any suggestions?
session based solution
With such deeply nested models select box on the front end wouldn't be enough...
Assuming this, you may want to create a current_house who's id live in a session (just like current_user works).
Once you have your current_house add different items by navigating to your list of items view and clicking on the add_to link :
# house_controller.rb
def add_to
current_house.polymorphic_items << Kitchen.find(params[:id])
redirect_to :back
end
But there are many approaches to this session based solution which sort of implements a cart/order system. You may want to add a current_item to add stuff in each leaf of your tree aka room of your house.
E.G after clicking on the kitchen you just added :
before_filter :set_current_item
def add_to
current_item.windows << Window.find(id)
end
current_item beeing polymorphic : a living room, a bathroom etc.
But how you implement that precisely depends on your Domain Model....
As a rule of thumb regarding nested forms I'd follow rails guidance for routes : don't go deeper than one level or you'll end up in a mess.
Yes this is a nested form. Railscasts nested forms is a great place to start.
If everything is already in the system you probably just want select boxes so they can select what they want. Also check out the .build method. If you have multiple levels of nesting you can also manually set the association by passing in the foreign key yourself.
I think you can model this with a single level of nested attributes, given the models below (based on Windows/Locks pre-existing and a room just needing to mix and match them into a set of windows with given locks):
class House < ActiveRecord::Base
has_many :rooms
end
class Room < ActiveRecord::Base
belongs_to :house
has_many :window_configs
end
class WindowConfig < ActiveRecord::Base
belongs_to :room
belongs_to :window
belongs_to :lock
end
class Lock < ActiveRecord::Base
has_many :window_configs
end
class Window < ActiveRecord::Base
has_many :window_configs
end
... based on that model setup, you could have a single house form that you dynamically add child 'room' definitions to that each have a name and a collection of window_configs which have two select boxes for each one (choose a window definition and then a lock definition). Because you're dynamically adding multiple rooms with multiple windows, you'd need some JS to populate new form elements, but it could all live in a single nested form.
form_for :house do |form|
# Dynamically add a Room form for each room you want with js
form.fields_for :room do |room_attributes|
room_attributes.text_field :name
# Dynamically add window_config forms on Room w/ JS
room_attributes.fields_for :window_config do |window_attributes|
window_attributes.select :window_id, Window.all
window_attributes.select :lock_id, Lock.all
models:
#StatusMessage model
class StatusMessage < ActiveRecord::Base
belongs_to :users
default_scope :order => "created_at DESC"
end
#User Model
class User < ActiveRecord::Base
has_many :status_messages
end
In controller I want to join these two tables and get fields from both table. for example I want email field from User and status field from StatusMessage. When I use :
#status = User.joins(:status_messages)
Or
#status = User.includes(:status_messages)
It gives me only the user table data.
How can I implement this requirement?
You need to use includes here. It preloads data so you won't have another SQL query when you do #user.status_messages.
And yes you can't really see the effect: you need to check your logs.
First of all, I don't think it is possible (and reasonable) what you want to do. The reason for that is that the relation between User and StatusMessage is 1:n, that means that each user could have 0 to n status messages. How should these multitudes of attributes be included in your user model?
I think that the method joints in class ActiceRecord has a different meaning, it is part of the query interface. See the question LEFT OUTER joins in Rails 3
There are similar questions on the net, here is what I have found that matches most:
Ruby on Rails: How to join two tables: Includes (translated for your example) in the user a primary_status_message, which is then materialized in the query for the user. But it is held in one attribute, and to access the attributes of the status_message, you have to do something like that: #user.primary_status_message.status
When you use #status = User.includes(:status_messages) then rails eagerley loads the data of all the tables.
My point is when you use this User.includes(:status_messages) it will loads the data of status_messages also but shows only users table data then if you want first user status_messages then you have to #user.first.status_messages