I have a Rails 5 app with a model called Sensor Registry.
It currently has about 160,000 records, but I am experiencing extremely low loading times when trying to display this data.
The application is running on a dual core Intel(R) Xeon(R) CPU E5-2670 v3 # 2.30GHz and 2GB of RAM.
The server logs show the following:
Started GET "/sensor_registries" for 187.220.30.180 at 2017-01-10 23:43:41 +0000
Cannot render console from 187.220.30.180! Allowed networks: 127.0.0.1, ::1, 127.0.0.0/127.255.255.255
ActiveRecord::SchemaMigration Load (1.2ms) SELECT "schema_migrations".* FROM "schema_migrations"
Processing by SensorRegistriesController#index as HTML
Rendering sensor_registries/index.html.erb within layouts/application
SensorRegistry Load (604.0ms) SELECT "sensor_registries".* FROM "sensor_registries"
Sensor Load (0.6ms) SELECT "sensors".* FROM "sensors" WHERE "sensors"."id" IN (49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 65, 61, 63, 64, 62)
Rendered sensor_registries/index.html.erb within layouts/application (54663.9ms)
Completed 200 OK in 55468ms (Views: 54827.7ms | ActiveRecord: 611.5ms)
I got rid of the N+1 problem, but I'm wondering if there is something more I can do about the database queries.
Anyway the problem seems to be at the moment of rendering the page, it takes about 54 seconds to process it.
Is there a way to optimize CPU usage?
What would be the best solution for speeding up the process and showing the data to the user fast enough?
This certainly doesn't look like a hardware problem, but a problem with implementation. This is impossible to answer perfectly without knowing more about your data structure and architecture, but a few thoughts that may help you track down the problem:
1) How big is the rendered page? Is it possible that the sheer size of the data is what's causing slow rendering times? If this is the problem and the rendered page is just too big, think about paginating the results.
2) How much memory is the ruby process using at any time? When you say, it "has about 160,000 records", I assume you're talking about the sensor_registries table, and I assume the subsequent sensors query with the sensors.id in (...) bit is constructed with some data from the sensor_registries table. If this entire table is loaded up in memory before any further work is done, is it possible that the ruby process is just out of memory?
3) Also, does the entire table really need to be loaded up all at once? You might want to take a look at http://apidock.com/rails/ActiveRecord/Batches/find_in_batches. That method is great for breaking up work that needs to be done on a large table.
4) Even better–and there's really not a better way to put this–rethink your architecture. Loading up a whole table in memory as part of a synchronous request (even a small one) is almost always a no-no. Can you come up with a sql query to get all your needed records without loading the whole table?
5) If you absolutely need the whole table, what about caching the result after a single load? If you expect the result of a query to be the same across requests with identical parameters, could you construct a cache key out of the params, and use that to store the result? At least that way, all requests after the first one will be fast(er).
User following methods:
User pagination to display limit number of records on page and load more records on click button.
User page caching to put the data into cache and load the heavy page in less time.
Load js files after the page load and use compressed js files.
Reduce the amount of records you show at the same time using pagination or something (5-10 at most). That way your queries and rendering will be reduces by a lot. I prefer the will_paginate gem for this but there are many more.
In your views, reduce the amount of logic you are using to render each sensor and create separate views for single sensors (show). In the single sensor views you can add more logic.
Use the new Relic gem to monitor your app and see which request take the most amount of time. Maybe you have slow external resources like api calls, etc which you can perform via Ajax and not on the server: https://docs.newrelic.com/docs/agents/ruby-agent/installation-configuration/ruby-agent-installation
Those are the basics, once you are done with that, move on to eager loading and caching.
For eager loading read this: http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
For caching read DHH's blog post: https://signalvnoise.com/posts/3113-how-key-based-cache-expiration-works
Related
Lets assume i have hundred thousand users
Simple Example,
user = User.where(id: 1..10000)
User Load (30.8ms) SELECT `users`.* FROM `users` WHERE (`users`.`id` BETWEEN 1 AND 10000)
in here, i want to slice more like this,
user.where(id: 100..1000)
User Load (2.9ms) SELECT `users`.* FROM `users` WHERE (`users`.`id` BETWEEN 1 AND 10000) AND (`users`.`id` BETWEEN 100 AND 1000)
Why is active record hitting db twice? it already has result that has bigger data. why does it have to hit db, not just reuse and slice ActiveRecord::Relation?
Is there any good solution for this?
ActiveRecord keeps track of queries and is able to cache certain duplicate requests, but in this case it's not that immediate for a library to understand that the second one is a subset of the first.
Moreover, there are several reasons why a generic library such as ActiveRecord may not want to implement a caching logic like that one. Caching a large data set in a very large application may result into several Mb of memory, and processes may reach the memory limit of the machine fairly quickly because the garbage collector would not be able to recollect the memory.
Long story short, it's a very bad idea to implement such feature in a generic ORM library.
If you want to implement it in your own code, you are free to do it.
ActiveRecord is hitting the db twice because you are running it in the console. This invokes the query on each line through .inspect. If this was run within a block of code, invocation would be delayed till you actually access user.
Instead of making two iteration pass it in single:
User.where("id between ? and ?", 100,1000)
It will reduce the db hitting, hope its a answer for your question
I have been working in Rails (I mean serious working) for last 1.5 years now. Coming from .Net background and database/OLAP development, there are many things I like about Rails but there are few things about it that just don't make sense to me. I just need some clarification for one such issue.
I have been working on an educational institute's admission process, which is just a small part of much bigger application. Now, for administrator, we needed to display list of all applied/enrolled students (which may range from 1000 to 10,000), and also give a way to export them as excel file. For now, we are just focusing on exporting in CSV format.
My questions are:
Is Rails meant to display so many records at the same time?
Is will_paginate only way to paginate records in Rails? From what I understand, it still fetches all the records from DB, and then selectively displays relevant records. Back in .Net/PHP/JSP, we used to create stored procedure and from there we selectively returns relevant records. Since, using stored procedure being a known issue in Rails, what other options do we have?
Same issue with exporting this data. I benchmarked the process i.e. receiving request at the server, execution of the query and response return. The ActiveRecord creation was taking a helluva time. Why was that? There were only like 1000 records, and the page showed connection timeout at the user. I mean, if connection times-out while working on for 1000 records, then why use Rails or it means Rails are not meant for such applications. I have previously worked with TB's of data, and never had this issue.
I never understood ORM techniques at the core. Say, we have a table users, and are associated with multiple other tables, but for displaying records, we need data from only tables users and its associated table admissions, then does it actually create objects for all its associated tables. I know, the data will be fetched only if we use the association, but does it create all the objects before-hand?
I hope, these questions are not independent and do qualify as per the guidelines of SF.
Thank you.
EDIT: Any help? I re-checked and benchmarked again, for 1000 records, where in we are joining 4-5 different tables (1000 users, 2-3 one-to-one association, and 2-3 one-to-many associations), it is creating more than 15000 objects. This is for eager loading. As for lazy loading, it will be 1000 user query plus some 20+ queries). What are other possible options for such problems and applications? I know, I am kinda bumping the question to come to top again!
Rails can handle databases with TBs of data.
Is will_paginate only way to paginate records in Rails?
There are many other gems like "kaminari".
it fetches all records from the db..
NO. It doesnt work that way. For example take the following query,Users.all.page(1).per(10)
User.all wont fire a db query, it will return a proxy object. And you call page(1) and per(10) on the proxy(ActiveRecord::Relation). When you try to access the data from the proxy object, it will execute a db query. Active record will accumulate all conditions and paramaters you pass and will execute a sql query when required.
Go to rails console and type u= User.all; "f"; ( the second statement: "f", is to prevent rails console from calling to_s on the proxy to display the result.)
It wont fire any query. Now try u[0], it will fire a query.
ActiveRecord creation was taking a helluva time
1000 records shouldn't take much time.
Check the number of sql queries fired from the db. Look for signs of
n+1 problem and fix them by eager loading.
Check the serialization of the records to csv format for any cpu or memory intensive operation.
Use a profiler and track down the function that is consuming most of the time.
I have a service that generates a large map through multiple iterations and calculations from multiple tables. My problem is I cannot use pagination offset to slice the data because the data is coming from multiple tables and different modifications happen on the data. To display this on the screen; I have to send the map with 10-20,000 records to the view and that is problematic with this large dataset.
At this time I have on-page pagination but this is very slow and inefficient.
One thing I thought is to dump it on a table and query it each time but then I have to deal with concurrent users.
My question is what is the best approach to display this list when I cannot use database slicing (offset, max)?
I am using
grails 1.0.3
datatables and jquery
Maybe SlickGrid! is an option for you. One of there examples works with 50000 rows and it seems to be fast.
Christian
I end up writing the result of the map in a table and use the data slicing on that table for pagination. It takes some time to save the data but at least I don't have to worry about the performance with the large data. I use time-stamp to differentiate between requests. each requests will be saved and retrieved with its time stamp.
The problem with your typical rails pagination gem is that it does 2 queries: one for the page you're on and one for the total count. When you don't care about how many pages there are (e.g. in an endless scroll), that 2nd query is unnecessary (just add 1 to your LIMIT clause in the 1st query and you know if there are more or not).
Is there a gem that'll do pagination without the 2nd query? The 2nd query is expensive when applying non-indexed filters in my WHERE clause on large datasets and indexing all my various filters is unacceptable because I need my inserts to be fast.
Thanks!
Figured it out. When using the will_paginate gem, you can supply your own total_entries option to AR:Base.paginate. This makes it so the 2nd query doesn't run.
This works for sufficiently large datasets where you only care about recent entries.
This isn't necessarily acceptable if you actually expect to hit the end of your list because if the list size is divisible by per_page you're going to query an empty set on your last query. With endless scroll, this is fine. With a manual "load more" button, you'll be displaying "load more" at the very end when there are no more items to load.
The standard approach, as you've identified, is to fetch N+1 records when you need N and if you get more than N records in the response, there is at least one additional page of results you can display.
The only reason you'd want to do an explicit COUNT(*) call is if you need to know specifically how many more records you will need to fetch. On some engines this can take a good chunk of time to compute so it is best avoided especially if the value is never directly used.
Since this is so simple, you really don't need a plugin to do it. Plugins like will_paginate is more concerned with the number of pages available so it does the count operation.
I have an optimization question. This is not my situation but for the purposes of the question this example fits better:
Let's say you have an app that displays your monthly specials. That information rarely changes - once a month. It display 5 specials, but each special needs to a display info like this:
Product.name
Product.manufacturer.name
Store.shipping_address.address_line_1
Store.shipping_address.address_line_1
Store.shipping_address.city
Store.shipping_address.state
Store.shipping_address.zip
Store.phone_number.area_code
Store.phone_number.phone_number
With properly setup associations, you'll end up doing 5 queries to a db. ( 1 to pull all the specials, 1 for address, 1 for phone, 1 for product, 1 for manufacturer).
In this example - shipping_address and phone_number, are methods setup in Store model, that do lookups on polymorphic models of Address and Phone number that have a 1 to many relationship to a model Category ( hold info like shipping vs billing , or phone vs. fax)
Here is the problem, let's say you have 5 specials, and they may be in various stores, so you need to do 5 queries on each page to display this info, including the front page.
What would be the best way to optimize this / speed it up?
The bottom line is that you do need to retrieve this data at some point--there's no getting around that. But, you can minimize the number of queries using eager loading and caching.
Here's the official guide to eager loading:
http://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations
Eager loading will minimize the number of queries, but you're still hitting the database. So, to avoid running those queries on each request, I would definitely cache the data at some level. Rails offers various caching strategies. Fragment caching might be an excellent solution here. Here are more details on caching:
http://guides.rubyonrails.org/caching_with_rails.html