I have an array of objects in my controller, paginated by Kaminari:
#pics = Pic.page(params[:page]).per(12).order(sort_column)
I render each new set of 12 pics on the same page with an AJAX request, for a manual infinite scroll effect. The sort_column bit sorts the pics according to most_viewed attribute within the object.
My problem - sometimes, when there are a number of pics with the same most_viewed attribute, - i.e. there are 10 images with "20" as their most_viewed value - I will get repeated objects when the new set of 12 is revealed with the AJAX request.
Here's an example of what I mean... I've clicked the "load more" (i've left it in to demonstrate), and there are objects repeated when the new batch loads.
Any help would be much appreciated. Thanks in advance.
You have to order on count AND another column (id would be the best), in this way SQL will returns a list that will not change (because right now SQL does not know what to do if X records have the same count).
You can change .order(sort_column) to .order("#{sort_column} ASC, id ASC"), and it should work.
Related
For considerably long period of time I’ve been struggling the following problem. This is an example of data stored in the DB:
> show series
flights,cycleId=1535,cycleIdx=0,engineId=2,flightId=1696,flightIdx=0,type=fil
flights,cycleId=1535,cycleIdx=0,engineId=2,flightId=1696,flightIdx=0,type=std
flights,cycleId=1535,cycleIdx=0,engineId=2,flightId=1696,flightIdx=0,type=raw
...
and my intention is to select a specific one by using a query like this:
SELECT * FROM flights WHERE type='fil' AND engineId= '2' AND flightId = '1696' AND flightIdx = '0' AND cycleId = '1535' AND cycleIdx = '0'
Such query, however, yields always zero results. Zilch.
Selecting the first (and only) tag works fine:
SELECT * FROM flights WHERE cycleId = '1535'
but using this condition on any other tag, like for example
SELECT * FROM flights WHERE type='fil'
does never return a single row. Querying only the first tag and nothing else works.
Could you please give me a hint what am I doing wrong? From all I have found people are always selecting just by a single tag but never more. What is the part that I cannot see?
Many thanks for any ideas!
I believe I have discovered the reason: two keys from the tags made by mistake their way into the fields. I spotted the trouble when listing the tag and fields keys as
show tag keys
show field keys
Deleting all records does not remove the keys from these lists and the problem persists. One need to drop the entire database to restore the order of things.
I am currently trying to order some ActiveRecord records by score. In my controller i have:
def index
#pagy, #drivers = pagy(
Driver.select(
'drivers.*',
'(drivers.no_races + drivers.no_poles + drivers.no_podiums + drivers.no_wins) AS score'
).reorder('drivers.score DESC'),
page: params[:page],
items: 16
)
end
I am using Pagy for the pagination. As the code shows, I do a query to select all drivers and then add together 3 columns in the table as 'score'. I then want to order by score going from high to low and show 16 records per page.
When the page loads it seems to order by driver id. I can't see anywhere else that i have an order by, but i did add reorder to override anything else.
Anyway for whatever reason i'm stuck with the wrong ordering. Any direction is appreciated :-)
Mockup - http://29qg.hatchboxapp.com/drivers
Ok so it was a simple error in the end. "score" is ambiguous because i was using score as an alias and i also had a column in the table called score.
I have a query that loads thousands of objects and I want to tame it by using find_in_batches:
Car.includes(:member).where(:engine => "123").find_in_batches(batch_size: 500) ...
According to the docs, I can't have a custom sorting order: http://www.rubydoc.info/docs/rails/4.0.0/ActiveRecord/Batches:find_in_batches
However, I need a custom sort order of created_at DESC. Is there another method to run this query in chunks like it does in find_in_batches so that not so many objects live on the heap at once?
Hm I've been thinking about a solution for this (I'm the person who asked the question). It makes sense that find_in_batches doesn't allow you to have a custom order because lets say you sort by created_at DESC and specify a batch_size of 500. The first loop goes from 1-500, the second loop goes from 501-1000, etc. What if before the 2nd loop occurs, someone inserts a new record into the table? That would be put onto the top of the query results and your results would be shifted 1 to the left and your 2nd loop would have a repeat.
You could argue though that created_at ASC would be safe then, but it's not guaranteed if your app specifies a created_at value.
UPDATE:
I wrote a gem for this problem: https://github.com/EdmundMai/batched_query
Since using it, the average memory of my application has HALVED. I highly suggest anyone having similar issues to check it out! And contribute if you want!
The slower manual way to do this, is to do something like this:
count = Cars.includes(:member).where(:engine => "123").count
count = count/500
count += 1 if count%500 > 0
last_id = 0
while count > 0
ids = Car.includes(:member).where("engine = "123" and id > ?", last_id).order(created_at: :desc).limit(500).ids #which plucks just the ids`
cars = Cars.find(ids)
#cars.each or #cars.update_all
#do your updating
last_id = ids.last
count -= 1
end
Can you imagine how find_in_batches with sorting will works on 1M rows or more? It will sort all rows every batch.
So, I think will be better to decrease number of sort calls. For example for batch size equal to 500 you can load IDs only (include sorting) for N * 500 rows and after it just load batch of objects by these IDs. So, such way should decrease have queries with sorting to DB in N times.
I'm using Mongoid to work with MongoDB. Everything is fine, I like it very much and so on. In my blog application (posts controller, index action) I have this code:
#posts = Post.without(:comments)
#posts = #posts.my_search(params[:s]) if params[:s]
#posts = #posts.order_by([:created_at, :desc])
#posts = #posts.where(:pid.lt => params[:p].to_i+1) if params[:p]
#posts = #posts.limit(items_per_page+1)
The part with "where" is implementation of my own pagination method (allows to page results in one direction only, but without skip(), what I consider a plus). Now, there are few small problems that make me feel uncomfortable:
For my pagination to work I need to get the last post within that limit. But when I do #posts.last I'm getting last document of the whole query without limit. Ok, this is strange, but not a big problem. Other than that, query results act like almost-ordinary-array, so at this moment I'm getting the last element with #posts.pop (funny, but it doesn't remove any documents) or #posts.fetch(-1)
I have a feeling that this isn't "right way" and there mush be something more elegant. Also
#posts.count generates second query exactly the same as first one (without limit) but with "count" only and I don't like it.
If I make the last line look like
#posts = #posts.limit(items_per_page+1).to_ary
to convert query results into array, everything generates only one query (good), but now #posts.count stops reporting what I need (total amount of documents without limit applied) and behaves exactly like #posts.size - it returns items_per_page+1 or less (bad).
So, here are my questions:
1) What is a "correct" way to get the last document of query results within given limit?
2) How to get total amount of documents with given conditions applied without generating additional query?
UPD:
3) #posts.first generates additional query, how to prevent it and just get first document before I iterate all documents?
Getting the last document:
Post.last
Getting last document with some other queries:
Post.order_by([:created_at, :desc]).last
Getting total number documents:
Post.order_by([:created_at, :desc]).count
Recommendation: Just use the built in pagination
#posts = Post.limit(10).paginate(:page=>pararms[:page])
later:
<%= will_paginate #posts %>
Regarding the additional queries -- mongoid lazy loads everything:
#posts = Post.all #no query has been run yet
#posts.first #Ok, a query has finally been run because you are accessing the objects
I have a model object which did not have a counter cache on it before and I added it via a migration. The thing is, I tried and failed to set the starting value of the counter cache based on the number of child objects I already had in the migration. Any attempt to update the cache value did not get written to the database. I even tried to do it from the console but it was never going to happen. Any attempt to write directly to that value on the parent was ignored.
Changing the number of children updated the counter cache (as it should), and removing the ":counter_cache => true" from the child would let me update the value on the parent. But that's cheating. I needed to be able to add the counter cache and then set its starting value to the number of children in the migration so I could then start with correct values for pages which would show it.
What's the correct way to do that so that ActiveRecord doesn't override me?
You want to use the update_counters method, this blog post has more details:
josh.the-owens.com add a counter cache to an existing db-table
This RailsCasts on the topic is also a good resource:
http://railscasts.com/episodes/23-counter-cache-column
The canonical way is to use reset_counter_cache, i.e.:
Author.find_each do |author|
Author.reset_counter_cache(author.id, :books)
end
...and that's how you should do it if those tables are of modest size, i. e. <= 1,000,000 rows.
BUT: for anything large this will take on the order of days, because it requires two queries for each row, and fully instantiates a model etc.
Here's a way to do it about 5 orders of magnitude faster:
Author
.joins(:books)
.select("authors.id, authors.books_count, count(books.id) as count")
.group("authors.id")
.having("authors.books_count != count(books.id)")
.pluck(:id, :books_count, "count(books.id)")
.each_with_index do |(author_id, old_count, fixed_count), index|
puts "at index %7i: fixed author id %7i, new books_count %4i, previous count %4i" % [index, author_id, fixed_count, old_count] if index % 1000 == 0
Author.update_counters(author_id, books_count: fixed_count - old_count)
end
It's also possible to do it directly in SQL using just a single query, but the above worked well enough for me. Note the somewhat convoluted way it uses the difference of the previous count to the correct one: this is necessary because update_counters doesn't allow setting an absolute value, but only to increase/decrease it. The column is otherwise marked readonly.