I am trying to cache the results of API calls. My application makes multiple calls with the same result and I would like it to use the cache to save time. When I use Rails.cache.fetch the block of code (with the API request) is executed every time even though the keys are the same. I have enabled caching in the development environment using rails dev:cache.
I tried to test in the rails console but my local rails console also won't store any keys in the cache. On heroku, the console works for caching but the application still sends every API request.
I have tried to use both memory and file-based caching locally.
Here is the method in application_record.rb
I am removing the parameters that are not consistent or important for my purpose and then using the parameters and path as the cache key.
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
def send_request(params,path)
key = params.except(:token, :start_date, :end_date)
key[:path] = path
puts key.to_s
Rails.cache.fetch(key, expires_in: 5.minutes) do
if (!$token || $token == '') then self.authenticate end
params[:token] = $token
url = URI("https://<api-domain>/"+path+"/?"+params.to_query)
puts '**** sending request *****'
begin
Net::HTTP.start(url.host, url.port, :use_ssl => url.scheme == 'https') do |http|
request = Net::HTTP::Get.new(url)
request["cache-control"] = 'no-cache'
response = http.request(request)
if response.code == "200"
return response
elsif response.code[0] == "4"
puts "recursive = "+response.code
puts response.read_body
$token = nil
send_request(params,path)
else
puts "request_report server error"
puts response.code
puts JSON.parse(response.read_body)
response
end
end
rescue
response = "SocketError: check that the server allows outbound https and that the bucksense API is live"
end
end
end
The log shows that requests are made every time. The first and second requests for both of these campaigns are exactly the same.
Entering new campaign: Show********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
Entering new campaign: ********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
I expect this to make API calls only when the same request has not been made within 5 minutes. Instead, it makes the API call every single time.
Thanks for the help, let me know if I'm making a stupid mistake, or if this is a poor way to achieve my desired results.
HTTP::Response objects are not serializable, so they can't be stored in the cache. (See this Rails comment.)
Related
So I have an application running Ruby 2.7.6 and Rails 6.1. Also using rom and rom-http for API calls. We have suddenly started to see our URLs in the browser, as well as server side API request parameters get overwritten with one of the symbols we use to represent an ID in the code. For example a request that should be https://sampleapi.json?currency=USD&locale=EN will be turned into https://sampleapi.json?currency=%3Asearch_id&locale=EN.
:search_id is a symbol we do use as a request parameter in many other places where we send a unique id related to a user search. But somehow, it has been injecting itself into almost all API calls as the actual value (encoded as %3Asearch_id). Sometimes we see API calls filled with almost every request parameter set this way.
The suspect is that something in activeresource or one of the rom gems has changed at some point and we never accounted for it.
There are some requests that would have &search_id as a request parameter... but with this bug the symbol :search_id itself is getting into the right side of param values.
Does this sound familiar to anyone?
Request handler code that processes data to form the final request:
def call(dataset)
uri = URI(dataset.uri)
uri.path += [dataset.name, dataset.path.presence].compact.join('/') + '.json'
# recaptcha api endpoint urls
Rails.logger.info "api endpoint is #{dataset.name}"
if (dataset.name == '/hotels/rooms' && Settings.enable_recaptcha)
api_path = dataset.name.split('/')
uri = URI(Settings.recaptcha_api_gateway_url)
uri.path += ['/', api_path.last, dataset.path.presence].compact.join('/')
end
if Rails.env == 'test' && dataset.name == '/hotels/rateshopping'
dataset.params[:client_ip] = "74.125.228.110"
end
if dataset.request_method == :get
uri.query = URI.encode_www_form(dataset.params.symbolize_keys.to_a.sort)
else
uri.query = "ip_address=#{$user_ip}"
end
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = uri.scheme.to_s == 'https'
request_klass = Net::HTTP.const_get(ROM::Inflector.classify(dataset.request_method))
Rails.logger.info "dataset.params: #{dataset.params}"
request = request_klass.new(uri.request_uri)
dataset.headers.each_with_object(request) do |(header, value), request|
request[header.to_s] = value
end
if dataset.request_method == :post
request['Content-Type'] = 'application/json; charset=utf-8'
request.body = dataset.params.to_json
end
with_cache(dataset, uri) do
with_logging(dataset, uri) { http.request(request) }
end
end
Let's say that I am calling the following code from inside a loop with a 1-second sleep/delay between each iteration and the URL is an API. How do I make sure that Net::HTTP is using the same API session for all the calls? I know the documentation says Net::HTTP.new will try to reuse the same connection. But how do I verify that? Is there a session ID that I can pull out of Net::HTTP?
request = Net::HTTP::Put.new(url)
url = URI(url)
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request["Accept"] = 'application/json'
request["Content-Type"] = 'application/json'
request["Authorization"] = #auth_key
request["cache-control"] = 'no-cache'
request.body = request_body.to_json if request_body
response = http.request(request)
double check the following against the ruby version you are running on
For one, I don't think there is any session ID from what I can see which would be quite a useful feature. Next, looking at the source code, we see the variable setting in lib/net/http.rb in such methods as:
def do_finish
#started = false
#socket.close if #socket
#socket = nil
end
# Returns true if the HTTP session has been started.
def started?
#started
end
# Finishes the HTTP session and closes the TCP connection.
# Raises IOError if the session has not been started.
def finish
raise IOError, 'HTTP session not yet started' unless started?
do_finish
end
Where do_finish sets the instance variable #socket to nil and #socket is used as a BufferedIO instance to run HTTP requests through
So I would write an override method for the finish method and raise an alert when it calls on do_finish.
Looking through the comments start is the safest bet to use the same session, so you could use a start block and compare the id of the instance variable does not change
Net::HTTP.start(url) do |http|
before = http.instance_variable_get(:#socket)
loop do
instance_var = http.instance_variable_get(:#socket)
break unless before == instance_var
end
end
I'm using Ruby on Rails 5 and I need to execute the following command in my application:
curl -F 'client_id=126581840734567' -F 'client_secret=678ebe1b3b8081231aab27dff738313' -F 'grant_type=authorization_code' -F 'redirect_uri=https://uri.com/' -F 'code=AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q' https://api.instagram.com/oauth/access_token
so that it returns something like:
{"access_token": "IGQVJYS0k8V6ZACRC10WjYxQWtyMVRZAN8VXamh0RVBZAYi34RkFlOUxXZnTJsbjlEfnFJNmprQThmQ4hTckpFUmJEaXZAnQlNYa25aWURnX3hpO12NV1VMWDNMWmdIT3FicnJfZAVowM3VldlVWZAEViN1ZAidHlyU2VDMUNuMm2V", "user_id": 17231445640157812}
Is there a way to make Rails execute those types of commands? I was trying the following:
uri = URI.parse('https://api.instagram.com/oauth/access_token')
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri)
request.set_form_data({
"client_id" => "126581840734567",
"client_secret" => "678ebe1b3b8081231aab27dff738313",
"grant_type" => "authorization_code",
"redirect_uri" => "http://nace.network/",
"code" => params[:code]
})
res = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
but I get the following error:
end of file reached
in this line:
res = Net::HTTP.start(uri.hostname, uri.port) do |http|
http.request(request)
end
You're using HTTPS, so you need to add this to your code:
Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
res = http.request(request)
end
But if you don't need persistent connections, you could also use this:
res = Net::HTTP.post_form(uri,
"client_id" => "126581840734567",
"client_secret" => "678ebe1b3b8081231aab27dff738313",
"grant_type" => "authorization_code",
"redirect_uri" => "http://nace.network/",
"code" => params[:code]
)
Also, you could consider using a library like Faraday, which is a lot easier to deal with.
Edit
This is from TinMan's comment below, sound points.
Using cURL from inside Ruby or Rails is extremely valuable. There is an incredible amount of functionality inside cURL that isn't implemented in Rails or Ruby; Even Ruby's HTTP clients have a hard time replicating it, so cURL is very acceptable depending on the needs of the application. And, depending on the application, because cURL is in compiled C, it could easily outrun pure Ruby clients.
Curl is a means of issuing HTTP (or HTTPs) requests from the command line.
You don't want to use CURL in Rails. You want to issue HTTP requests from within Rails. Using curl is okay, it's one way to issue HTTP requests from with Rails.
We can refine that down further to, you want to issue HTTP requests from Ruby. Narrowing/distilling down to the most basic version of the problem is always good to do.
We knew all this already probably - still worth writing down for us all to benefit from!
Use HTTP in Ruby
We want to use a HTTP Client. There are many but, for this I'm going to use Faraday (a gem) 'cause I like it.
You've made a good start with Ruby's built in NET:HTTP but I prefer Faraday's DSL. It results in more readable and extendable code.
So, here is a class! I barely tested this so, use as a starting point. Make sure you write some unit tests for it.
# This is a Plain Old Ruby Object (PORO)
# It will work in Rails but, isn't Rails specific.
require 'faraday' # This require is needed as it's a PORO.
class InstagramOAuth
attr_reader :code
# The code parameter will likely change frequently, so we provide it
# at run time.
def initialize(code)
#code = code
end
def get_token
connection.get('/oauth/access_token') do |request|
request.params[:code] = code
end
end
private
def connection
#connection ||= Faraday.new(
url: instagram_api_url,
params: params,
ssl: { :ca_path => https_certificate_location }
)
end
def instagram_api_url
#url ||= 'https://api.instagram.com'
end
# You need to find out where these are for your self.
def https_certificate_location
'/usr/lib/ssl/certs'
end
def params
# These params likely won't change to often so we set a write time
# in the class like this.
{
client_id: '126581840734567',
client_secret: '678ebe1b3b8081231aab27dff738313',
grant_type: 'authorization_code',
redirect_uri: 'https://uri.com/'
}
end
end
# How do we use it? Like so
# Your big old authorisation code from your question
code = 'AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU'\
'7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz'\
'5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H'\
'-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q'
# This will return a Faraday::Response object but, what is in it?
response = InstagramOAuth.new(code).get_token
# Now we've got a Hash
response_hash = response.to_hash
puts 'Request made'
puts "Request full URL: #{response_hash[:url]}"
puts "HTTP status code: #{response_hash[:status]}"
puts "HTTP response body: #{response_hash[:body]}"
When I ran the snippet above I got the following. The class works, you just need to tweak the request params until you get what you want. Hopefully the class demonstrates how to send HTTP requests in Ruby/Rails.
Request made
Request full URL: https://api.instagram.com/oauth/access_token?client_id=126581840734567&client_secret=678ebe1b3b8081231aab27dff738313&code=AQBi4L2Ohy3Q_N3V48OygFm0zb3gEsL985x5TIyDTNDJaLs93BwXiT1tyGYWoCg1HlBDU7ZRjUfLL5HVlzw4G-7YkVEjp6Id2WuqOz0Ylt-k2ADwDC5upH3CGVtHgf2udQhLlfDnQz5NPsnmxjg4bW3PJpW5FaQs8fn1ztgYp-ssfAf6IRt2-sI45ZC8cqqr5K_12y0Nq_Joh0H-tTfVyNLKatIxHPCqRDb3tfqgmxim1Q&grant_type=authorization_code&redirect_uri=https%3A%2F%2Furi.com%2F
HTTP status code: 405
HTTP response body:
Additional Reading
. https://lostisland.github.io/faraday/usage/
. https://github.com/lostisland/faraday/wiki/Setting-up-SSL-certificates
I'm building out a platform for displaying data in charts that pulls from Zendesk's API. I'm running into trouble in that only 100 records at a time can be pulled with one call. How do I pull multiple pages of records from this resource?
Here is the code I use to make the call:
require 'net/http'
require 'uri'
require 'json'
#imports User data from the zendesk api and populates the database with it.
uri = URI.parse("https://samplesupport.zendesk.com/api/v2/users.json")
request = Net::HTTP::Get.new(uri)
request.content_type = "application/json"
request.basic_auth("sampleguy#sample.com", "samplepass")
req_options = {
use_ssl: uri.scheme == "https",
}
#response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
puts #response.body
puts #response.message
puts #response.code
This works fine for calling down one 'page' of resources...any help with grabbing multiple pages using my script would be greatly appreciated. Thank you!
Based on ZenDesk's documentation they return a next_page attribute in their payload. So you should just check for its existence and then query again if it exists. Repeat as needed.
require 'json'
# setup to query for the first page
results = JSON.parse(#response.body)
users = results['users'] #to get the users
if results['next_page']
# Do another query to results['next_page'] URL and add to users list
I am using Ruby on Rails 3 and I am trying to implement an API. I would like to solve some strange problems that I have on returning data after a web client HTTP GET Request. In few words, problems are in the response body returned values for which I get "too much" "" (see the examples belowe) and, sometime, in returning JSON data.
On the web service side in my Rack middleware I have:
class Testing
def initialize(app)
#app = app
end
def call(env)
accounts = Account.find([1,2,3])
resp_test = accounts.count
[200, {}, resp_test] # No [200, {'Content-Type' => 'application/json'}, resp_test]
end
end
On the client side if I see the response I have
# debug response.body
---
In this case the problem is the accounts.count that returns a value of "" in the response body. It is possible that accounts.count doesn't do what it should do.
I also encountered some problems when I didn't return JSON data. For example, debugging variables on the client side, sometime I got a body response value of "" if I didn't return JSON DATA like this:
# On the service side in the Rack middleware file
[200, {}, resp_test] # No[200, {'Content-Type' => 'application/json'}, resp_test.to_json]
The response are:
# Case don't returning JSON data
# debug response.body
---
# Case returning JSON data
# debug response.body
--- test_value
What is the problem? If it is accounts.count or Account.find([1,2,3]), how can I make that to work in order to return correct value?
first your middleware should be after ActiveRecord::ConnectionAdapters::ConnectionManagement when calling rake middleware
second try to return resp_test.to_s