I'm getting into a project in Ruby on Rails currently, which I don't really have any experience in. Now, we're running into some performance problems that I think are related to executing too many queries.
We have a Service model that we set in our controller as follows:
#services = Service.includes(:points).with_points
The with_points scope is defined in the service as such:
scope :with_points, -> { joins(:points).where.not(points: []).distinct }
(I think the where clause is not needed here, but that's probably not relevant to the question.)
Then in the view, we're using the fetched services, with linked points, as follows:
<% #services.each do |s| %>
<div class="col-xs-12 col-sm-6 col-lg-4 serviceDiv" data-rating="<%= s.service_ratings %>" >
<% if s.service_ratings == "A" %>
<% grade = "rating-a" %>
<!-- etc. -->
Now, as far as I've seen when researching around, this is a relatively normal pattern when trying to list all rows from a table. However, when I look at the logs, it looks like a separate query is executed for every service?
# This query is what I'd expect:
SQL (1.6ms) SELECT DISTINCT "services"."id" # etc, etc
# But then we also get one of these for every service
Service Exists (1.5ms) SELECT 1 AS one FROM "services" WHERE "services"."name" = $1 AND ("services"."id" != $2) LIMIT $3
# And quite a few of these for every service:
CACHE Service Exists (0.0ms) SELECT 1 AS one FROM "services" WHERE "services"."name" = $1 AND ("services"."id" != $2) LIMIT $3
Now, my hunch is that those "Service Exists" lines are bad news. What are they, and where are they coming from? Is there anything else that's relevant that I'm missing here?
After #MarekLipka's pointers in the comments on this question, I managed to find the source of the problem. Not sure how many people will run into this setup in the future, but I'll share it just in case.
The clue was in our accessing s.service_ratings which was not, in fact, a column in the database. ActiveRecord Query Trace pointed to the source of all those queries being a reference to s.service_ratings in the view, which was a red flag.
service_ratings was actually a method (assuming I'm using the correct terminology here) on the Service model that, apart from returning a value based on a calculation on several of the model's properties, also called self.update_attributes to actually store that value in the database. This meant that every time we retrieved this model's data in order to display it, we also ran another query to save that value to the database - often redundantly.
In other words, the solution for us now is to either run the calculation at some other point of time and store it in the database once, or recalculate it every time we need it and not store it at all.
Related
I have a Rails application where there is a request to create elements on a table and another one, waiting, that reads if such element was created or not.
Right now, I check the live Data and the waiting process never sees any new record for the table.
Any ideas how to force a reconnection or anything that will update the model reference?
The basic idea:
One user, through a POST, is creating a record (that i can see directly on the database).
Another piece of code, that was running before the requests, is waiting for that record, but does not find it.
I'd appreciate any insights.
The waiting request is probably using the ActiveRecord cache. Rails caches queries for the duration of a request, so if you run the exact same query multiple times in the same request, it will only hit the database the first time.
You can simulate this in the console:
Entity.cache { Entity.first; Entity.first }
In the logs, you'll see something like this (notice the second request says CACHE):
[2018-12-06T10:51:02.436 DEBUG (1476) #] Entity Load (4.9ms) SELECT "entities".* FROM "entities" LIMIT 1
[2018-12-06T10:51:02.450 DEBUG (1476) #] CACHE (0.0ms) SELECT "entities".* FROM "entities" LIMIT 1
To bypass the cache, you can use:
Entity.uncached { Entity.where(key: #key)&.last }
Using uncached will disable the cache inside the block, even if the enclosing scope is running inside a cached block.
It seems that, for a running Rails application, if a running piece of code is looking for anything that has been updated by a new request, after the first began its execution, the updated data would not show, using active_record models.
The solution: Just run a raw query.
sql = "SELECT * FROM entities WHERE key = '"+key+"' ORDER BY ID DESC LIMIT 1;"
records_array = ActiveRecord::Base.connection.execute(sql).values
I know it's a huge oversimplification and that there is probably an underlying problem, but it did solved it.
I'm trying to utilize .exists?() to return true or false, pretty simple. It would be nice to do the one-liner scope like so:
scope :any_alternates, lambda{|apikey| Track.exists?(:track_id => apikey)}
Or even using this scope syntax:
scope :any_alternates, ->(apikey) {Track.exists?(:track_id => apikey)}
But for some reason, the above scopes will return all rows in my db table when there's not a match. It works how it should when it finds a match however, but breaks if none...
I'm forced to create a method, which (to my knowledge) should be doing the same thing in the above scope:
def self.any_alternates(apikey)
return Track.exists?(:track_id => apikey)
end
Any idea why .exists?() isn't working inside of my scope?
After some testing...
If there is no match, then the scope will return all rows in the DB... (I updated above to mention that). I checked the generated query on both the scope and method to see if there's a difference, but they're the same:
SELECT 1 AS one FROM `tracks` WHERE `tracks`.`track_id` = '_btbd_uUmQT8hYUK3SrJ9Q' LIMIT 1
Update:
even though I'm searching on a column called track_id, this column is not setup as a relationship to another model. I know this is confusing, but that's how this table got setup (for good reason, beyond this issue so not worth touching on here)
Are you passing in nil? That would cause all records to be returned. You can drop the Track in the scope, like this:
scope :any_alternates, lambda{|apikey| exists?(:track_id => apikey)}
Here is what happens when you pass in nil:
> Track.any_alternates(nil).count
Track Exists (1.7ms) SELECT 1 AS one FROM "track" WHERE "tracks"."track_id" IS NULL LIMIT 1
instead of passing in a value:
> Track.any_alternates('X4DBA36gbtqgWl4F1')
Track Exists (0.4ms) SELECT 1 AS one FROM "tracks" WHERE "tracks"."track_id" = $1 LIMIT 1 [["track_id", "X4DBA36gbtqgWl4F1"]]
Are you sure that you have the right search? A Track having a track_id implies that there is a relation between Track and another track model which would be interesting.
Also the scope syntax is off. A scope is just a query of the table, so traditionally it is only things associated with the current model.
scope :any_alternates, ->(api_key) { |api_key| where(track_id: api_key }
I'm a rails newbie and am building an app. My current problem is trying to find the average time since the last purchase for each customer of an online store using the app, where we have data about their orders and their customers. The problem is that I'm getting an error that says "undefined method `src_created_at' for nil:NilClass." Right now I'm trying to do this only for customers that have purchased once, and leaving aside those that purchased multiple times.
Here's my code:
#customers = Customer.where(:identity_id => #identity.id)
#single_order_customers = #customers.where("orders_count = ?", 1)
days_since_array = []
#single_order_customers.each do |s|
one_order = Order.where(:identity_id => #identity.id, :src_customer_id => s.src_id)
the_date = one_order[0].src_created_at
purchase_date = the_date.to_date
days_between = (Date.today - purchase_date)
days_since_array << days_between
end
days = days_since_array.inject(:+)
#adslp = days / days_since_array.count
Thanks in advance. I can provide what customer and order data looks like if necessary. Any advice would help, even though I know this question is somewhat vague. I've tried some kind if and unless statements validating presence or nil values and they're not working.
Edit:
Here's what's run in the console:
Order Load (123.0ms) SELECT "orders".* FROM "orders" WHERE "orders"."identity_id" = 2 AND "orders"."src_customer_id" = '114863554'
NoMethodError: undefined method `src_created_at' for nil:NilClass
(The above is several orders successfully run and then breaking on the last I've shown.)
Last point: when I try, specifically, to find nil values for this purpose I don't find any.
Order.where(:identity_id => 2, :src_created_at => nil)
Order Load (209.6ms) SELECT "orders".* FROM "orders" WHERE "orders"."identity_id" = 2 AND "orders"."src_created_at" IS NULL
=> []
For your last point where you tried to find nil values, you are getting an empty array [] because your query returned an empty set (no records matched your query). Trying to access any index on an empty array in rails won't throw an IndexOutOfBoundsException, but will instead just return nil, which is where you are getting your nil from. You are expecting one_order[0] to be an order item, but it is instead nil.
The solution to your problem would be to make sure the Order you are searching for already exists in the database. You might want to check how your orders are being created, e.g. if you use Order.create(params[:order]), do you have any validations that are failing, etc. You can check the orders you have through the rails console; just run Order.all. There should be an Order that has an identity_id of 2 and src_created_at of nil for the last query you wrote to return an actual order in the set.
A quick fix for now would be to remove the extra query to set one_order and just get it from the customer:
...
#single_order_customers.each do |s|
one_order = s.orders.first
the_date = one_order.src_created_at
...
The extra query does not seem to be necessary. You have already filtered #customers by identity_id, and src_customer_id will definitely match s.src_id if you get orders by calling s.orders. Relationships should be set up correctly in the models so that a customer has_many orders. You mentioned that you are just starting with rails; I would highly recommend reading a tutorial for rails first to save yourself headaches like these in the future :)
New to Ruby and Rails so forgive me if my terminology is off a bit.
I am working on optimizing some inherited code and watching the logs I am seeing queries repeat themselves due to lines like this in a .rabl file:
node(:program) { |user| (!user.programs.first.nil?) ? user.programs.first.name : '' }
user and program are both active record objects
Moving to the rails console, I can replicate the problem, but I can also get the expected behavior, which is only one query:
>u = User.find(1234)
User Load (0.3ms) SELECT `users`.* FROM `users` WHERE [...]
> (!u.programs.first.nil?) ? u.programs.first.name : ''
Program Load (0.2ms) SELECT `programs`.* FROM `programs` [...]
Program Load (0.3ms) SELECT `programs`.* FROM `programs` [...]
=> "Output"
Note that repeating the ternary statement in the console will always give me 2 queries.
I can get the expected behavior like so:
> newu = User.find(12345)
User Load (3.8ms) SELECT `users`.* FROM `users` WHERE [...]
> newu.programs
Program Load (0.6ms) SELECT `programs`.* FROM `programs` [...]
> (!newu.programs.first.nil?) ? newu.programs.first.name : ''
=> "Output"
Repeating the ternary statement now won't requery at all.
So the question is: why does calling newu.programs change the behavior? Shouldn't calling u.programs.first.nil? also act to load all the program records in the same way?
With an association, first is not sugar for [0].
If the association is loaded, then it just returns the first element of the array. If it is not loaded, it makes a database query to load just that one element. It can't stick that in the association cache (at least not without being smarter), so the next query to first does the query again (this will use the query cache if turned on)
What Rails is assuming is that if the association is big, and you are only using one element of it then it would be silly to load the whole thing. This can be a little annoying when this isn't the case and you are just using the one item, but you're using it repeatedly.
To avoid this you can either assign the item to a local variable so that you do genuinely only call first once, or do
newu.programs[0]
which will load the whole association (once only) and return the first element.
Rails does the same thing with include?. Instead of loading the whole collection, it will run a query that tests whether a specific item is in the collection (unless the collection is loaded)
I'm trying to figure out where a whole pile of extra queries are being generated by my rails app. I need some ideas on how to tackle it. Or, if someone can give me some hints, I'd be grateful.
I get these:
SQL (1.0ms) SELECT name
FROM sqlite_master
WHERE type = 'table' AND NOT name = 'sqlite_sequence'
SQL (0.8ms) SELECT name
FROM sqlite_master
WHERE type = 'table' AND NOT name = 'sqlite_sequence'
SQL (0.8ms) SELECT name
FROM sqlite_master
WHERE type = 'table' AND NOT name = 'sqlite_sequence'
repeated over and over on every request to the DB (as much as 70 times for a single request)
I tried installing a plugin that traced the source of the queries, but it really didn't help at all. I'm using the hobofields gem, dunno if that is what's doing it but I'm somewhat wedded to it at the moment
Any tips on hunting down the source of these extra queries?
Have a look at ActiveRecord gem in connection_adapters/sqlite_adapter.rb on line 173 (I'm using ActiveRecord 3.0.7) you have a function called tables that generates the exact query you posted:
SELECT name
FROM sqlite_master
WHERE type = 'table' AND NOT name = 'sqlite_sequence'
In development mode this function gets called for every table from your database, on every request.
In production this function gets called only once, when the server starts, and the result is cached.
In development for each request rails looks into your database to see what columns each table has so it can generate methods on your models like "find_by_name". Since in production is unlikely that your database changes, rails does this lookup only on server start.
It's very difficult to tell this without look into the Code.
but i am sure you write your query in a certain loop
for i in 0..70
SqliteMaster.find(:all, :conditions=>["type=? and not name=?", 'table', 'sqlite_sequesnce'])
end
So my advice is that to check all the methods which gets called after requesting certain method and look whether the query called in a loop.
I've just seen this appearing in my logs when I do a search with the metasearch gem but ONLY in development mode.
I believe that it is caused by the plugin acts-as-taggable-on.
It will check if the table or the cache column exists.