I was able to get an access token working with oauth, but the end point limits the maximum records per request to 100.
I need to get more than that, and am wondering if there is a simple/common way to do this?
I'd like to keep making requests until I get all the records. For example:
#products = JSON.parse(#access_token.get("/api/rest/products?page=#{#n}&limit=100").body)
I may need 10,000+ of the products. Is this possible?
Assuming #products is an array of products you should be able to do this
#products = []
(1..100).each do |page|
#products << JSON.parse(#access_token.get("/api/rest/products?page=#{page}&limit=100").body)
end
If you don't know exactly how many pages or products there are, unless the api provide you that, you can do something like this to stop fetching when there are no more products.
#products = []
# Assuming you never want more then 10.000
(1..100).each do |page|
new_products = JSON.parse(#access_token.get("/api/rest/products?page=#{page}&limit=100").body)
#products << new_products
break if new_products.size < 100
end
Related
My application monitors ProductionJobs, derived from BusinessProcesses in successive versions. Thus the unique key of ProductionJob class is composed of business_process_id and version fields.
Initially, the ProductionJob index would display the list of objects (including all versions) using an Arel structured query (#production_jobs).
But it is more convinient to only show the last version of each ProductionJob. So I created a query (#recent_jobs) to retrieve the last version of the ProductionJob for a given BusinessProces.
Joining the 2 queries should return only the last version of each ProductionJob. This is what I can't achieve with my knowledge of Arel, and I would be grateful if you could show me how to do!
Here is the code in production_jobs_controller:
a) Arel objects setup
private
def jobs
ProductionJob.arel_table
end
def processes # jobs are built on the processes
BusinessProcess.arel_table
end
def flows # flows provide a classifiaction to processes
BusinessFlow.arel_table
end
def owners # owner of the jobs
User.arel_table.alias('owners')
end
def production_jobs # job index
jobs.
join(owners).on(jobs[:owner_id].eq(owners[:id])).
join(processes).on(jobs[:business_process_id].eq(processes[:id])).
join(flows).on(processes[:business_flow_id].eq(flows[:id])).
join_sources
end
def job_index_fields
[jobs[:id],
jobs[:code].as("job_code"),
jobs[:status_id],
jobs[:created_at],
jobs[:updated_by],
jobs[:updated_at],
jobs[:business_process_id],
jobs[:version],
processes[:code].as("process_code"),
flows[:code].as("statistical_activity_code"),
owners[:name].as("owner_name")]
end
def order_by
[jobs[:code], jobs[:updated_at].desc]
end
# Latest jobs
def recent_jobs
jobs.
join(owners).on(jobs[:owner_id].eq(owners[:id])).
join_sources
end
def recent_jobs_fields
[ jobs[:code],
jobs[:business_process_id].as('bp_id'),
jobs[:version].maximum.as('max_version')
]
end
b) The index method
# GET /production_jobs or /production_jobs.json
def index
#production_jobs = ProductionJob.joins(production_jobs).
pgnd(current_playground).
where("business_flows.code in (?)", current_user.preferred_activities).
order(order_by).
select(job_index_fields).
paginate(page: params[:page], :per_page => params[:per_page])
#recent_jobs = ProductionJob.joins(recent_jobs).select(recent_jobs_fields).group(:business_process_id, :code)
#selected_jobs = #production_jobs.joins(#recent_jobs).where(business_process_id: :bp_id, version: :max_version)
Unfortunately, #selected_jobs returns a nil object, even though #production_jobs and #recent_jobs show linkable results. how should I build the #selected_jobs statement to reach the expected result?
Thanks a lot!
After several trials, I finally included the sub-request in a 'where ... in()' clause. This may not be optimal, and I am open to other proposals.
The result can be understood as the following:
#recent_jobs provide the list ProductionJobs'last versions, based on their code and version
#production_jobs provide the list of all ProductionJobs
#selected_jobs adds the where clause to #production_jobs, based on the #recent_jobs:
The last request is updated to:
#selected_jobs = #production_jobs
.where("(production_jobs.code,
production_jobs.business_process_id,
production_jobs.version)
in (?)",
#recent_jobs
)
It works this way, but I'd be glad to receive suggestions to enhance this query. Thanks!
I'm running a query like the below:
Item.where("created_at >=?", Time.parse(params[:created_at])).where(status_id: params[:status_id])
...where the user can decide to NOT provide a parameter, in which case it should be excluded from the query entirely. For example, if the user decides to not pass a created_at and not submit it, I want to run the following:
Item.where(status_id: params[:status_id])
I was thinking even if you had a try statement like Time.try(:parse, params[:created_at]), if params[created_at] were empty, then the query would be .where(created_at >= ?", nil) which would NOT be the intent at all. Same thing with params[:status_id], if the user just didn't pass it, you'd have a query that's .where(status_id:nil) which is again not appropriate, because that's a valid query in itself!
I suppose you can write code like this:
if params[:created_at].present?
#items = Item.where("created_at >= ?", Time.parse(params[:created_at])
end
if params[:status_id].present?
#items = #items.where(status_id: params[:status_id])
end
However, this is less efficient with multiple db calls, and I'm trying to be more efficient. Just wondering if possible.
def index
#products = Product.where(nil) # creates an anonymous scope
#products = #products.status(params[:status]) if params[:status].present?
#products = #products.location(params[:location]) if params[:location].present?
#products = #products.starts_with(params[:starts_with]) if params[:starts_with].present?
end
You can do something like this. Rails is smart in order to identify when it need to build query ;)
You might be interested in checking this blog It was very useful for me and can also be for you.
If you read #where documentation, you can see option to pass nil to where clause.
blank condition :
If the condition is any blank-ish object, then #where is a no-op and returns the current relation.
This gives us option to pass conditions if valid or just return nil will produce previous relation itself.
#items = Item.where(status_condition).where(created_at_condition)
private
def status_condition
['status = ?', params[:status]] unless params[:status].blank?
end
def created_at_condition
['created_at >= ?', Time.parse(params[:created_at])] unless params[:created_at].blank?
end
This would be another option to achieve the desired result. Hope this helps !
This is something I struggle with, or whenever I do it it seems to be messy.
I'm going to ask the question in a very generic way as it's not a single problem I'm really trying to solve.
I have an API that I want to consume some data from, e.g. via:
def get_api_results(page)
results = HTTParty.get("api.api.com?page=#{page}")
end
When I call it I can retrieve a total.
results["total"] = 237
The API limits the number of records I can retrieve in one call, say 20. So I need to call it a few more times.
I want to do something like the following, ideally breaking it into pieces so I can use things like delayed_job..etc
def get_all_api_pages
results = get_api_results(1)
total = get_api_results(1)["total"]
until page*20 > total do |p|
results += get_api_results(p)
end
end
I always feel like I'm writing rubbish whenever I try and solve this (and I've tried to solve it in a number of ways).
The above, for example, leaves me at the mercy of an error with the API, which knocks out all my collected results if I hit an error at any point.
Wondering if there is just a generally good, clean way of dealing with this situation.
I don't think you can have that much cleaner...because you only receive the total once you called the API.
Have you tried to build your own enum for this. It encapsulates the ugly part. Here is a bit of sample code with a "mocked" API:
class AllRecords
PER_PAGE = 50
def each
return enum_for(:each) unless block_given?
current_page = 0
total = nil
while total.nil? || current_page * PER_PAGE < total
current_page += 1
page = load_page(current_page)
total = page[:total]
page[:items].each do |item|
yield(item)
end
end
end
private
def load_page(page)
if page == 5
{items: Array.new(37) { rand(100) }, total: 237}
else
{items: Array.new(50) { rand(100) }, total: 237}
end
end
end
AllRecords.new.each.each_with_index do |item, index|
p index
end
You can surely clean that out a bit but i think that this is nice because it does not collect all the items first.
I'm implementing a search system that uses name, tags, and location. There is a has_and_belongs_to_many relationship between Server and Tag. Here's what my search method currently looks like:
def self.search(params)
#servers = Server.all
if params[:name]
#servers = #servers.where "name ILIKE ?", "%#{params[:name]}%"
end
if params[:tags]
#tags = Tag.find params[:tags].split(",")
# How do I eliminate servers that do not have these tags?
end
# TODO: Eliminate those that do not have the location specified in params.
end
The tags parameter is just a comma-separated list of IDs. My question is stated in a comment in the if params[:tags] conditional block. How can I eliminate servers that do not have the tags specified?
Bonus question: any way to speed this up? All fields are optional, and I am using Postgres exclusively.
EDIT
I found a way to do this, but I have reason to believe it will be extremely slow to run. Is there any way that's faster than what I've done? Perhaps a way to make the database do the work?
tags = Tag.find tokens
servers = servers.reject do |server|
missing_a_tag = false
tags.each do |tag|
if server.tags.find_by_id(tag.id).nil?
missing_a_tag = true
end
end
missing_a_tag
end
Retrieve the servers with all the given tags with
if params[:tags]
tags_ids = params[:tags].split(',')
#tags = Tag.find(tags_ids)
#servers = #servers.joins(:tags).where(tags: {id: tags_ids}).group('servers.id').having("count(*) = #{tags_ids.count}")
end
The group(...).having(...) part selects the servers with all requested tags. If you're looking for servers which have at least one of the tags, remove it.
With this solution, the search is done in a single SQL request, so it will be better than your solution.
In Rails, I have a Product model. Sometimes I need to get multiple products at the same time (but the list is completely dynamic, so it can't be done on the Rails side).
So, let's say for this call I need to get products 1, 3, 9, 24 in one call. Is this possible? If so, do I need a custom route for this and what do I put in my controller?
i.e. does something like this work? /products/1,3,9,24
I don't think you should need to change the routes at all. You should just have to parse them in your controller/model.
def show
#products = Product.find params[:id].split(',')
end
If you then send a request to http://localhost/products/1,3,9,24, #products should return 4 records.
I would consider this a request to index with a limited scope, kind of like a search, so I would do:
class ProductsController < ApplicationController
def index
#products = params[:product_ids] ? Product.find(params[:product_ids]) : Product.all
end
end
and then link to this with a url array:
<%= link_to 'Products', products_path(:product_ids => [1, 2, 3]) %>
this creates the standard non-indexed url array that looks kind of like
product_ids[]=1&product_ids[]=2 ...
Hope that helps.
Product.where(:id => params[:ids].split(','))