I am using sqlite3 database in my project. for that I can retrive the data from the Database using following query "select * from tablename"..
But I want to take the hundred sequence records from the database, like If I scroll the UITableView based on the I want to take 100 100 records.
I have tried the following things,
SELECT * FROM mytable ORDER BY record_date DESC LIMIT 100; - It retrives only 100 records.When I scroll the table i want to fetch the next 100 records and show it.
Is it possible to do it
Please Guide me.
You could simply use the OFFSET clause, but this would still force the database to compute all the records that you're skipping over, so it would become inefficient for a larger table.
What you should do is to save the last record_date value of the previous page, and continue with the following ones:
SELECT *
FROM MyTable
WHERE record_date < ?
ORDER BY record_date DESC
LIMIT 100
See https://www.sqlite.org/cvstrac/wiki?p=ScrollingCursor for details.
Related
I've rewritten this question as my previous explanation was causing confusion.
In the SQL world, you have an initial record set that you apply a query to. The output of this query is the result set. Generally, the initial record set is an entire table of records and the result set is the records from the initial record set that match the query ruleset.
I have a use case where I need my application to occasionally operate on only a subset of records in a table. If a table has 10,000 records in it, I'd like my application to behave like only the first 1,000 records exist. These should be the same 1,000 records each time. In other words, I want the initial record set to be the first 1,000 devices in a table (when ordered by primary key), and the result set the resulting records from these first 1,000 devices.
Some solutions have been proposed, and it's revealed that my initial description was not very clear. To be more explicit, I am not trying to implement pagination. I'm also not trying to limit the number of results I receive (which .limit(1,000) would indeed achieve).
Thanks!
This is the line in your question that I don't understand:
This causes issues though with both of the calls, as limit limits the results of the query, not the database rows that the query is performed on.
This is not a Rails thing, this is a SQL thing.
Device.limit(n) runs SELECT * FROM device LIMIT n
Limit always returns a subset of the queried result set.
Would first(n) accomplish what you want? It will both order the result set ascending by the PK and limit the number of results returned.
SQL Statements can be chained together. So if you have your subset, you can then perform additional queries with it.
my_subset = Device.where(family: "Phone")
# SQL: SELECT * FROM Device WHERE `family` = "Phone"
my_results = my_subset.where(style: "Touchscreen")
# SQL: SELECT * FROM Device WHERE `family` = "Phone" AND `style` = "Touchscreen"
Which can also be written as:
my_results = Device.where(family: "Phone").where(style: "Touchscreen")
my_results = Device.where(family: "Phone", style: "Touchscreen")
# SQL: SELECT * FROM Device WHERE `family` = "Phone" AND `style` = "Touchscreen"
From your question, if you'd like to select the first 1,000 rows (ordered by primary key, pkey) and then query against that, you'll need to do:
my_results = Device.find_by_sql("SELECT *
FROM (SELECT * FROM devices ORDER BY pkey ASC LIMIT 1000)
WHERE `more_searching` = 'happens here'")
You could specifically ask for a set of IDs:
Device.where(id: (1..4).to_a)
That will construct a WHERE clause like:
WHERE id IN (1,2,3,4)
I have written a Rails 4 app that accepts and plots sensor data. Sometimes there are 10 points per hour (but this number is not fixed). I'm plotting the data and doing a simple query of Points.all to get all the data points.
In order to reduce the query size, I would like to only return one record per hour. It doesn't matter which record is returned. The first record each hour using the created_at field would be fine.
How do I construct a query to do this?
You can get first one, but maybe average value is better. All you need to do is to group it by hour. I am not 100% about sqlite syntax but something in this sense:
connection.execute("SELECT AVG(READING_VALUE) FROM POINTS GROUP BY STRFTIME('%Y%m%d%H0', CREATED_AT)")
Inspired from this answer, here is an alternative which retrieves the latest record in that hour (if you don't want to average):
Point.from(
Point.select("max(unix_timestamp(created_at)) as max_timestamp")
.group("HOUR(created_at)") # subquery
)
.joins("INNER JOIN points ON subquery.max_timestamp = unix_timestamp(created_at)")
This will result in the following query:
SELECT `points`.*
FROM (
SELECT max(unix_timestamp(created_at)) as max_timestamp
FROM `points`
GROUP BY HOUR(created_at)
) subquery
INNER JOIN points ON subquery.max_timestamp = unix_timestamp(created_at)
You can also use MIN instead to get the first record of the hour, if you like, as well.
Let's say I have an ActiveRecord query which has 1 million records as result and I'm using WillPaginate to (of course) paginate this result.
So, my doubt is:
All these 1 million records are stored somehow in memory or the will_paginate query gets executed again to retrieve more and more records ?
I hope I'm being clear :)
No, it won't repeatedly grab all 1 million records. The queries look something like this:
SELECT ... FROM ... LIMIT X OFFSET Y
...where X is the maximum per page, and Y is the number of records to skip from the top of the stack, as determined by X * (current_page - 1).
Will_paginate will run separate queries to populate the data and will not fetch all the data at once. You can check how will paginate works in the terminal also. Lets say you are using rails default webrick server. Then in your terminal you will see the sql queries made by will_paginate like so:
Processing by InvoicesController#index as HTML
Invoice Load (0.1ms) SELECT "invoices".* FROM "invoices" ORDER BY id DESC LIMIT 5 OFFSET 0
I have a table with paginated data and this is the way I select data for each page:
#visitors = EventsVisitor
.select('visitors.*, events_visitors.checked_in, events_visitors.checkin_date, events_visitors.source, events_visitors.id AS ticket_id')
.joins(:visitor)
.order(order)
.where(:event_id => params[:event_id])
.where(filter_search)
.where(mode)
.limit(limit)
.offset(offset)
Also to build table pagination I need to know total count of records. Currently my solution for this is very rough:
total = EventsVisitor
.select('count(*) as count, events_visitors.*')
.joins(:visitor)
.order(order)
.where(:event_id => params[:event_id])
.where(filter_search)
.where(mode)
.first()
.count
So my question is as follows - What is the optimal ruby way to select limited data for the current page and total count of records?
I noticed that if I do #visitors.count - additional sql query will be generated:
SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM `events_visitors` INNER JOIN `visitors` ON `visitors`.`id` = `events_visitors`.`visitor_id` WHERE `events_visitors`.`event_id` = 1 LIMIT 15 OFFSET 0) subquery_for_count
First of all, I do not understand what is the reason to send an additional query to get a count of data that we already have, I mean that after we got data from database in #visitors we can count it with ruby without need to send additional queries to DB.
Second - I thought that maybe there are some ways to use something like .total_count that will generate similar count(*) query but without that useless limit/offset?
you should except limit and offset
http://guides.rubyonrails.org/active_record_querying.html#except .
See how kaminari does it
https://github.com/kaminari/kaminari/blob/92052eedf047d65df71cc0021a9df9df1e2fc36e/lib/kaminari/models/active_record_relation_methods.rb#L11
So it might be something like
total = #visitors.except(:offset, :limit, :order).count
Suppose you want to find the last record entered into the database (highest ID) matching a string: Model.where(:name => 'Joe'). There are 100,000+ records. There are many matches (say thousands).
What is the most efficient way to do this? Does PostgreSQL need to find all the records, or can it just find the last one? Is this a particularly slow query?
Working in Rails 3.0.7, Ruby 1.9.2 and PostgreSQL 8.3.
The important part here is to have a matching index. You can try this small test setup:
Create schema xfor testing:
-- DROP SCHEMA x CASCADE; -- to wipe it all for a retest or when done.
CREATE SCHEMA x;
CREATE TABLE x.tbl(id serial, name text);
Insert 10000 random rows:
INSERT INTO x.tbl(name) SELECT 'x' || generate_series(1,10000);
Insert another 10000 rows with repeating names:
INSERT INTO x.tbl(name) SELECT 'y' || generate_series(1,10000)%20;
Delete random 10% to make it more real life:
DELETE FROM x.tbl WHERE random() < 0.1;
ANALYZE x.tbl;
Query can look like this:
SELECT *
FROM x.tbl
WHERE name = 'y17'
ORDER BY id DESC
LIMIT 1;
--> Total runtime: 5.535 ms
CREATE INDEX tbl_name_idx on x.tbl(name);
--> Total runtime: 1.228 ms
DROP INDEX x.tbl_name_idx;
CREATE INDEX tbl_name_id_idx on x.tbl(name, id);
--> Total runtime: 0.053 ms
DROP INDEX x.tbl_name_id_idx;
CREATE INDEX tbl_name_id_idx on x.tbl(name, id DESC);
--> Total runtime: 0.048 ms
DROP INDEX x.tbl_name_id_idx;
CREATE INDEX tbl_name_idx on x.tbl(name);
CLUSTER x.tbl using tbl_name_idx;
--> Total runtime: 1.144 ms
DROP INDEX x.tbl_name_id_idx;
CREATE INDEX tbl_name_id_idx on x.tbl(name, id DESC);
CLUSTER x.tbl using tbl_name_id_idx;
--> Total runtime: 0.047 ms
Conclusion
With a fitting index, the query performs more than 100x faster.
Top performer is a multicolumn index with the filter column first and the sort column last.
Matching sort order in the index helps a little in this case.
Clustering helps with the simple index, because still many columns have to be read from the table, and these can be found in adjacent blocks after clustering. It doesn't help with the multicolumn index in this case, because only one record has to be fetched from the table.
Read more about multicolumn indexes in the manual.
All of these effects grow with the size of the table. 10000 rows of two tiny columns is just a very small test case.
You can put the query together in Rails and the ORM will write the proper SQL:
Model.where(:name=>"Joe").order('created_at DESC').first
This should not result in retrieving all Model records, nor even a table scan.
This is probably the easiest:
SELECT [columns] FROM [table] WHERE [criteria] ORDER BY [id column] DESC LIMIT 1
Note: Indexing is important here. A huge DB will be slow to search no matter how you do it if you're not indexing the right way.