Rails Postgres query Position of item relative to others? - ruby-on-rails

Question

I'm building a model against Amazon's SQS Standard Queue which can send updates out of order.
My goal is to properly order them.
I am longpolling to copy all data from the queue into my DB.
table example - lets say I fetch some messages and process them
id | published_at | run_at | payload
1 | 1:11pm | nil | ...
2 | 1:12pm | nil | ...
3 | 1:13pm | nil | ...
4 | 1:14pm | nil | ...
5 | 1:15pm | nil | ...
Then I fetch some more, and we can see that a few odd messages are now outdated.
id | published_at | run_at | payload
1 | 1:11pm | 1:15 | ...
2 | 1:12pm | 1:15 | ...
3 | 1:13pm | 1:15 | ...
4 | 1:14pm | 1:15 | ...
5 | 1:15pm | 1:15 | ...
6 | 1:13pm | nil | ...
7 | 1:14pm | nil | ...
8 | 1:16pm | nil | ...
if i were to order by published_at, you can see that the queue needs to be re-processed starting at ID=6 down to make sure messages are processed in order.
id | published_at | run_at | payload
1 | 1:11pm | 1:15 | ...
2 | 1:12pm | 1:15 | ...
3 | 1:13pm | 1:15 | ...
6 | 1:13pm | nil | ...
4 | 1:14pm | 1:15 | ...
7 | 1:14pm | nil | ...
5 | 1:15pm | 1:15 | ...
8 | 1:16pm | nil | ...
There is value in processing data accurately, and very little problem with processing twice so re-running is not a problem.
I am mostly curious how best to best find the oldest item that has not been ran, and start running from that moment forward.
Currently doing:
# fetch oldest publish_time that has not been ran
first_publish_time = AnyOfferChange.where(run_at: nil).minimum(:publish_time)
if first_publish_time
# start there, and process in ascending order
AnyOfferChange.order("publish_time DESC").where("publish_time >= ?",first_publish_time).reverse.each(&:process!)
end
It feels quite fragile, I'd like to fetch the position and use it as a limit.
limit = AnyOfferChange.where(run_at: nil).order("publish_time ASC").pluck("POSITION SOMETHIN(SOMETHING)").first
if limit > 0
# start there, and process in ascending order
AnyOfferChange.order("publish_time DESC").limit(limit).reverse.each(&:process!)
end

Answer 1

The follow SQL query will give you the oldest publish_time:
AnyOfferChange.where(run_at: nil).minimum(:publish_time)
Or, if you want one record:
AnyOfferChange.where(run_at: nil).order(publish_time: :asc).first
This will limit the SQL query the oldest row that has not ran.
Fetch all records that not ran from old till new:
result = AnyOfferChanges.where(run_at: nil).order(publish_time: :asc)
# or
result = AnyOfferChanges.where(run_at: nil).order(:publish_time) # Defaults to :asc
result.each(&:process!) # Process result. See note below for batch info.
Fetch all records that did not ran with exactly the oldest publish_time (that did not ran):
# See note below to prevent unwanted SQL execution for the statements
# below when executing in the terminal.
# Create shorthand.
any_offer_changes = AnyOfferChange.arel_table
# Build query parts.
not_ran = AnyOfferChange.where(run_at: nil)
oldest_publish_time = not_ran.select(any_offer_changes[:publish_time].minimum)
# All records that not ran with with the oldest publish time.
result = not_ran.where(publish_time: oldest_publish_time)
result.each(&:process!) # Process result. See note below for batch info.
This will result in fetching all records with the lowest publish time in one SQL query, using a sub-query.
The reason I use another way than AnyOfferChange.where(run_at: nil).minimum(:publish_time) to fetch the minimum for the last part. Is that this query will break the chain and create multiple SQL queries instead of one. Whereas AnyOfferChange.where(run_at: nil).select(any_offer_changes[:run_at].minimum) will keep the chain intact when used in a where statement.
notes:
unwanted SQL execution
When run one by one this will result in multiple queries since the #inspect (used to show you the result) will trigger the SQL to execute. In the terminal follow each statement with ;nil, to prevent execution when building a #where chain. This is not needed when executed in a script.
using "batches"
For large amounts of records you may have to limit the resulting values. Rails support batches, but they don't respect the given order. To keep the order you can create your own batch, although probably less efficient. This can be done like so:
result = AnyOfferChange.where(run_at: nil).order(:publish_time).limit(100)
result.each(&:process!) while result.reload.any?
Assuming you set the run_at attribute in #process!, otherwise the above will result in an endless loop.