So I've been using MongoDB (with the Mongoid Ruby Gem) for a while now, and as our app has grown I've noticed requests taking longer and longer as my data has grown, here is what a typical request for my app looks like, but it takes about 500ms, just for the DB stuff.
Nothing special here just some controller stuff:
Started GET "/cities/san-francisco?date_range=past_week" for 127.0.0.1 at 2011-11-15 11:13:04 -0800
Processing by CitiesController#show as HTML
Parameters: {"date_range"=>"past_week", "id"=>"san-francisco"}
Then the queries run, but what I don't understand is that for every query that runs it performs a MONGODB dashboard_development['system.namespaces'].find({}) before actually running it! Why?
MONGODB dashboard_development['system.namespaces'].find({})
MONGODB dashboard_development['users'].find({:_id=>BSON::ObjectId('4e80e0090f6d2e306f000001')})
MONGODB dashboard_development['system.namespaces'].find({})
MONGODB dashboard_development['cities'].find({:slug=>"san-francisco"})
MONGODB dashboard_development['system.namespaces'].find({})
MONGODB dashboard_development['accounts'].find({:_id=>BSON::ObjectId('4e80e0090f6d2e306f000002')})
MONGODB dashboard_development['system.namespaces'].find({})
MONGODB dashboard_development['neighborhoods'].find({"city_id"=>BSON::ObjectId('4e80e00a0f6d2e306f000005')})
Then the views get rendered, they are pretty slow too... but that is a seperate problem all together, I'll address that at another time.
Rendered cities/_title_and_scope.html.erb (109.3ms)
Rendered application/_dropdown.html.erb (0.1ms)
Rendered application/_date_range_selector.html.erb (6.2ms)
Rendered cities/show.html.erb within layouts/application (122.7ms)
Rendered application/_user_dropdown.html.erb (0.9ms)
Rendered application/_main_navigation.html.erb (5.8ms)
So minus the views the request took about 500ms, thats too long for a really simple query, additionally the app is going to grow and that time is going to grow as well. Also this example is faster than the requests usually take, sometimes it takes 1000ms or more!
Completed 200 OK in 628ms (Views: 144.9ms)
Additionally I wanted to ask what fields are most appropriate for indexes? Maybe this is my problem, as I'm not really using them at all. Any help understanding this would be really really appreciated. Thanks!
You need to use indexes -- otherwise, your mongo queries are executing what is best described as a full table scan. It is loading the entirety of your collection's json documents into memory and then evaluating each one to determine if it should be including in the response.
Strings, Date, Numbers can all be used as index -- the trick is, have an index on each attribute you are doing a "where" on.
You can turn off table-scans in your mongo config to help find table scans and destroy them!
Related
I have just configured database replication in a Rails 6.0 application. It works smoothly, but I am trying to confirm that the right database/role is used for the queries I am expecting it to (long story short: manually calling ActiveRecord::Base.connected_to for business logic reasons).
I Googled and Stack-Overflow-ed that question, but can't seem to find an answer.
Is it possible to have the SQL queries debug entries display the database name and role used?
2021-08-06T13:26:02.186960+00:00 app[web.1]: D, [2021-08-06T13:26:02.186858 #46] DEBUG -- : [c132c761-c02c-4690-a8b0-f8c29d999xxx] Shop Load (1.6ms) SELECT "shops".* FROM "shops" WHERE "shops"."slug" = 'some slug' LIMIT 1
If not, what other ways are at my disposal to assert that my replica/follower is being hit instead of the leader/master database?
If it is of any help to answer my question: I am running this on Heroku.
Thank you in advance!
I have a Rails application where there is a request to create elements on a table and another one, waiting, that reads if such element was created or not.
Right now, I check the live Data and the waiting process never sees any new record for the table.
Any ideas how to force a reconnection or anything that will update the model reference?
The basic idea:
One user, through a POST, is creating a record (that i can see directly on the database).
Another piece of code, that was running before the requests, is waiting for that record, but does not find it.
I'd appreciate any insights.
The waiting request is probably using the ActiveRecord cache. Rails caches queries for the duration of a request, so if you run the exact same query multiple times in the same request, it will only hit the database the first time.
You can simulate this in the console:
Entity.cache { Entity.first; Entity.first }
In the logs, you'll see something like this (notice the second request says CACHE):
[2018-12-06T10:51:02.436 DEBUG (1476) #] Entity Load (4.9ms) SELECT "entities".* FROM "entities" LIMIT 1
[2018-12-06T10:51:02.450 DEBUG (1476) #] CACHE (0.0ms) SELECT "entities".* FROM "entities" LIMIT 1
To bypass the cache, you can use:
Entity.uncached { Entity.where(key: #key)&.last }
Using uncached will disable the cache inside the block, even if the enclosing scope is running inside a cached block.
It seems that, for a running Rails application, if a running piece of code is looking for anything that has been updated by a new request, after the first began its execution, the updated data would not show, using active_record models.
The solution: Just run a raw query.
sql = "SELECT * FROM entities WHERE key = '"+key+"' ORDER BY ID DESC LIMIT 1;"
records_array = ActiveRecord::Base.connection.execute(sql).values
I know it's a huge oversimplification and that there is probably an underlying problem, but it did solved it.
I have an AngularJS app and there's one page in my application, only one, that is taking 2 minutes to load. It is loading a bit of data, but the data itself is only 700KB and I benchmarked the entire rails action starting from the beginning until right before the render and it only takes 15-20 seconds. But when I look at the actual network call, or I put a timer before the angular http post call and then one in the success, they both show the call taking almost 2 minutes. I can't figure out what's going on between the render and the success on angular that would be causing this extreme time difference. Does anyone know how I could further debug this or possibly know what could be causing this?
The rails action just does a couple big database calls, all optimized, then does some work on the data, then the data (which is already JSONified with to_json) is rendered out.
Rails action ends with Completed 200 OK in 20458ms (Views: 913.8ms | ActiveRecord: 139.6ms)
Edit: If I put a limit on my data it's almost instant, so it definitely has to do with the data. But I'm not sure what could be causing the minute and a half disconnect between when the rails action finished and the http post success begins.
Edit2: An ajax call takes an equal amount of time. So there must be an issue with how the data is being parses on the front end, not sure the best way to do this. Since there's an obvious issue between the render and the page getting it.
Turns out the issue was somehow the extremely complex hash my old coworker wrote. I think the whole thing was pretty unnecessary so I deleted all 90 lines of code where he built the hash from scratch and instead 3 lines.
I now have the two activerecord queries with the proper includes, and then wrote one render state on those activerecord objects with as_json with the proper include and only parameters and the only thing now loads in 25 seconds on development. I can only imagine it'll be faster in production/staging. I don't know why the hashes were so hard to render as json, but calling as_json on the active record objects within the render statement completely fixed my issue.
I have a controller that returns JSON or XML from a fairly complex relational query with some controller logic as well.
I've tuned on the DB side by refining my query and making sure my indexes are correct for my query.
In my log I see items like this:
Completed in 740ms (View: 1, DB: 50)
So if I understand correctly this means the view took 1 second to render and the DB query was 50ms. Is all the remaining time in the controller? I've tried bypassing my controller logic and just leaving my to_json and to_xml in there and it is just as slow. As a point of reference my average returned JSON result set is 168k.
Are there other steps that go into the Completed in time? Does it include time until last byte for the network transfer?
Update: I wrapped various parts of my controller in benchmarking blocks:
self.class.benchmark("Active Record Find") do
#my query here
end
What I found was that even though the log line says DB: 50 my active record find is taking almost all of the remaining time. So now I'm confused as to what that DB number means and why the benchmark line will say ~ 600ms but the DB: time will be ~50.
Thanks
Your DB number is time actually spent in the database, but not loading ActiveRecord objects.
So if you're loading 168,000 ruby active_record objects to render then as JSON, this would explain your 550 ms (or more!)
If these times are observed in the development environment, the additional time is probably because the application classes are not being cached. The application files are being reloaded on every request.
As suggested in this answer, try setting config.cache_classes = true in environment/development.rb and restarting your server to see what effect this has on your response times. Be sure to change this back to config.cache_classes = false and restart your server once you're done.
I am working on an "optimization" on my application and I am trying to understand the output that rails (version 2.2.2) gives at the end of the render.
Here is the "old" way:
Rendered user/_old_log (25.7ms)
Completed in 466ms (View: 195, DB: 8) | 200 OK
And the "new" way:
Rendered user/_new_log (48.6ms)
Completed in 337ms (View: 192, DB: 33) | 200 OK
These queries were exactly the same, the difference is the old way is parsing log files while the new way is querying the database log table.
The actual speed of the page is not the issue (the user understands that this is a slow request) ... but I would like the page to respond as quickly as possible even though it is a "slow" page.
So, my question is, what do the numbers represent/mean? In other words, which way was the faster method and why?
This:
Rendered user/_old_log (25.7ms)
is the time to render just the _old_log partial template, and comes from an ActiveSupport::Notification getting processed by ActionView::LogSubscriber
This:
Completed 200 OK in 466ms
Is the http status returned, as well as the total time for the entire request. It comes from ActionController::LogSubscriber.
Also, note those parenthetical items at the end:
(Views: 124.6ms | ActiveRecord: 10.8ms)
Those are the total times for rendering the entire view (partials & everything) and all database requests, respectively, and come from ActionController::LogSubscriber as well.
Jordan's answer is correct. To paraphrase, the first number is the time the page took to load. The second is how long the view took to generate. The last number is how long it took for your database to handle all queries you sent to it.
You can also get an estimation of how long your Controller and Model code took by subtracting the last two numbers from the first number, but a better way would be to use the Benchmark.measure method (http://www.ruby-doc.org/stdlib/libdoc/benchmark/rdoc/classes/Benchmark.html).
Your new way appears to have improved because code in the Controller/Model completes faster.
Your new way is spending less time overall but more time rendering the template.