So I have an application running Ruby 2.7.6 and Rails 6.1. Also using rom and rom-http for API calls. We have suddenly started to see our URLs in the browser, as well as server side API request parameters get overwritten with one of the symbols we use to represent an ID in the code. For example a request that should be https://sampleapi.json?currency=USD&locale=EN will be turned into https://sampleapi.json?currency=%3Asearch_id&locale=EN.
:search_id is a symbol we do use as a request parameter in many other places where we send a unique id related to a user search. But somehow, it has been injecting itself into almost all API calls as the actual value (encoded as %3Asearch_id). Sometimes we see API calls filled with almost every request parameter set this way.
The suspect is that something in activeresource or one of the rom gems has changed at some point and we never accounted for it.
There are some requests that would have &search_id as a request parameter... but with this bug the symbol :search_id itself is getting into the right side of param values.
Does this sound familiar to anyone?
Request handler code that processes data to form the final request:
def call(dataset)
uri = URI(dataset.uri)
uri.path += [dataset.name, dataset.path.presence].compact.join('/') + '.json'
# recaptcha api endpoint urls
Rails.logger.info "api endpoint is #{dataset.name}"
if (dataset.name == '/hotels/rooms' && Settings.enable_recaptcha)
api_path = dataset.name.split('/')
uri = URI(Settings.recaptcha_api_gateway_url)
uri.path += ['/', api_path.last, dataset.path.presence].compact.join('/')
end
if Rails.env == 'test' && dataset.name == '/hotels/rateshopping'
dataset.params[:client_ip] = "74.125.228.110"
end
if dataset.request_method == :get
uri.query = URI.encode_www_form(dataset.params.symbolize_keys.to_a.sort)
else
uri.query = "ip_address=#{$user_ip}"
end
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = uri.scheme.to_s == 'https'
request_klass = Net::HTTP.const_get(ROM::Inflector.classify(dataset.request_method))
Rails.logger.info "dataset.params: #{dataset.params}"
request = request_klass.new(uri.request_uri)
dataset.headers.each_with_object(request) do |(header, value), request|
request[header.to_s] = value
end
if dataset.request_method == :post
request['Content-Type'] = 'application/json; charset=utf-8'
request.body = dataset.params.to_json
end
with_cache(dataset, uri) do
with_logging(dataset, uri) { http.request(request) }
end
end
Related
Let's say that I am calling the following code from inside a loop with a 1-second sleep/delay between each iteration and the URL is an API. How do I make sure that Net::HTTP is using the same API session for all the calls? I know the documentation says Net::HTTP.new will try to reuse the same connection. But how do I verify that? Is there a session ID that I can pull out of Net::HTTP?
request = Net::HTTP::Put.new(url)
url = URI(url)
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request["Accept"] = 'application/json'
request["Content-Type"] = 'application/json'
request["Authorization"] = #auth_key
request["cache-control"] = 'no-cache'
request.body = request_body.to_json if request_body
response = http.request(request)
double check the following against the ruby version you are running on
For one, I don't think there is any session ID from what I can see which would be quite a useful feature. Next, looking at the source code, we see the variable setting in lib/net/http.rb in such methods as:
def do_finish
#started = false
#socket.close if #socket
#socket = nil
end
# Returns true if the HTTP session has been started.
def started?
#started
end
# Finishes the HTTP session and closes the TCP connection.
# Raises IOError if the session has not been started.
def finish
raise IOError, 'HTTP session not yet started' unless started?
do_finish
end
Where do_finish sets the instance variable #socket to nil and #socket is used as a BufferedIO instance to run HTTP requests through
So I would write an override method for the finish method and raise an alert when it calls on do_finish.
Looking through the comments start is the safest bet to use the same session, so you could use a start block and compare the id of the instance variable does not change
Net::HTTP.start(url) do |http|
before = http.instance_variable_get(:#socket)
loop do
instance_var = http.instance_variable_get(:#socket)
break unless before == instance_var
end
end
I am trying to cache the results of API calls. My application makes multiple calls with the same result and I would like it to use the cache to save time. When I use Rails.cache.fetch the block of code (with the API request) is executed every time even though the keys are the same. I have enabled caching in the development environment using rails dev:cache.
I tried to test in the rails console but my local rails console also won't store any keys in the cache. On heroku, the console works for caching but the application still sends every API request.
I have tried to use both memory and file-based caching locally.
Here is the method in application_record.rb
I am removing the parameters that are not consistent or important for my purpose and then using the parameters and path as the cache key.
class ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
def send_request(params,path)
key = params.except(:token, :start_date, :end_date)
key[:path] = path
puts key.to_s
Rails.cache.fetch(key, expires_in: 5.minutes) do
if (!$token || $token == '') then self.authenticate end
params[:token] = $token
url = URI("https://<api-domain>/"+path+"/?"+params.to_query)
puts '**** sending request *****'
begin
Net::HTTP.start(url.host, url.port, :use_ssl => url.scheme == 'https') do |http|
request = Net::HTTP::Get.new(url)
request["cache-control"] = 'no-cache'
response = http.request(request)
if response.code == "200"
return response
elsif response.code[0] == "4"
puts "recursive = "+response.code
puts response.read_body
$token = nil
send_request(params,path)
else
puts "request_report server error"
puts response.code
puts JSON.parse(response.read_body)
response
end
end
rescue
response = "SocketError: check that the server allows outbound https and that the bucksense API is live"
end
end
end
The log shows that requests are made every time. The first and second requests for both of these campaigns are exactly the same.
Entering new campaign: Show********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
Entering new campaign: ********************************
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1D", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
{:metrics=>"wins,win_rate,ctr,clicks_global", :timezone=>"UTC", :type=>"1", :method=>"getdata", :groupby=>"P1M", :dimensions=>"advertiser_name,campaign_name", :path=>"3.0/report/thirdpart"}
**** sending request *****
Campaign finished: ********************************
I expect this to make API calls only when the same request has not been made within 5 minutes. Instead, it makes the API call every single time.
Thanks for the help, let me know if I'm making a stupid mistake, or if this is a poor way to achieve my desired results.
HTTP::Response objects are not serializable, so they can't be stored in the cache. (See this Rails comment.)
I have a rails app where an action never finishes and then times out.
Find the diagram below for better illustration.
My rails apps action is called
The action POSTs some data to another app
The other app needs something to complete the computation and calls a different action than the first of the Rails app
The other app gets a response and finishes the computation
The other app responds to the rails apps POST request
The view is rendered accordingly
Now the issue: The other app never gets a response from the main app. After the Rails apps request times out however, the response is sent (however too late of course) so I think it is somehow cued.
I don't understand how to fix that. I use rails 5 and Puma which should be able to handle parallel calls. Its also not a local issue, same happens in prod.
I use the recommended puma.rb config from Heroku
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
What do I do to fix this cueing?
Controller:
# New method
def live_preview_page
preview_locale = params[:preview_locale]
date = params[:date] # The date to preview
page_id = params[:id]
return if locale.nil? || locale =~ /not/ || date.nil?
all_templates = Template.all.order('name ASC') # Maybe move to render_live_editor_page
if date == "all"
active_modules = #page.page_modules.order(rank: :asc)
else
active_modules = #page.page_modules.order(rank: :asc).to_a.valid_for(date: date.to_date)
puts "Active modules: #{active_modules.count}"
end
active_modules_json = active_modules.each do |content_module|
content_module.body = YAML.load(content_module.body).to_json
end
response = helpers.render_preview(active_modules, all_templates, preview_locale)
renderer = ContentRenderer.new
actionController = ActionController::Base.new
rendered_helper = actionController.render_to_string(
partial: '/pages/preview-helper-snippet', locals: {
all_templates: all_templates, # For select when creating new modules
modulesData: active_modules_json, # For rendering the JSON containing the data for the editor
current_page: #page.id,
localeLinks: renderer.generateStgPreviewURLs(SettingService.get_named_locales, #page.id),
locale: preview_locale,
all_locales: SettingService.locales_for_live_editor,
all_sites_and_locales: SettingService.get_sites_and_locales
})
proxy_service = ProxyService.new
proxy_service.get_page do |error, page_wrapper|
# Note: Issue is that Vapor app generates warnings inline template : encountered \r in middle of line, treated as a mere space
rendered_body_with_helper = response.body.force_encoding("UTF-8") + rendered_helper
decorated_page = page_wrapper.gsub("__WIDGET__", rendered_body_with_helper)
render inline: decorated_page
return
end
end
Helper
def render_preview(active_modules, all_templates, preview_locale)
req = Request.new
preview_body = {
modules: active_modules,
templates: all_templates,
sites: SettingService.get_sites,
configuration: {
locale: preview_locale,
site: "DE"
}
}
req.send_request(
url: "#{ENV["RENDER_SERVICE_URL"]}/preview",
body: preview_body,
options: {
type: :post,
json: true,
username: ENV["RENDER_SERVICE_BASIC_AUTH_USERNAME"],
password: ENV["RENDER_SERVICE_BASIC_AUTH_PASSWORD"]
}
) do |response_code, response|
return response
end
end
Request is just a thin wrapper
require "uri"
require "net/http"
class Request
# Yields resonse_code (int), response
# Parameters besides url: are optional
def send_request(url:, body: {}, header: {}, options: {})
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
if options.key? :type
case options[:type]
when :get
request = Net::HTTP::Get.new(uri.request_uri, header)
when :post
request = Net::HTTP::Post.new(uri.request_uri, header)
end
else
# Default is GET
request = Net::HTTP::Get.new(uri.request_uri, header)
end
if options.key?(:username) && options.key?(:password)
request.basic_auth options[:username], options[:password]
end
unless body.class == String
body = body.to_json.to_s
end
request.body = body unless body.empty?
puts request.body
# SSL is default
if options.key? :ssl
http.use_ssl = options[:ssl]
else
http.use_ssl = Rails.configuration.force_ssl
#http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
if options.key? :json
request.add_field("Content-Type", "application/json")
end
response = http.request(request)
yield response.code.to_i, response
end
end
The following answer is (probably) not the answer you want - but it's the answer you need:
The best way to fix this is to avoid the loop in the request/response logic (where the rails app calls itself through the other app).
Concurrency might help delay the onset of the issue, but the issue will always occur as long as the loop exists.
For example, assume you have an 100 requests from clients to the Rails app.
Rails will call the other app and the other app's request will be queued as request number 101.
This can be solved with 100 threads (for example, 10 workers with 10 threads each)...
But what will your app do with 200 client requests?
This cycle is endless, the more clients you have the more concurrency you require before you experience DoS.
The only solution is to avoid the loop to begin with.
Either break it up to 3 apps or (better yet), avoid dependencies between micro services.
I am new to ruby programming. I was trying to write below ruby code to create a comment in Github gist.
uri = URI.parse("https://api.github.com/gists/xxxxxxxxxxx/comments")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = (uri.scheme == 'https')
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(uri.request_uri)
request.body = {
"body" => "Chef run failed "
}.to_json
response = http.request(request)
response2 = JSON.parse(response.body)
puts response2
But I executed below script then I always get {"message"=>"Not Found", "documentation_url"=>"https://developer.github.com/v3"}
Don't know what I am doing wrong. Appreciate help.
Make sure that you're authenticated first. From the Github API docs on Authentication:
There are three ways to authenticate through GitHub API v3. Requests
that require authentication will return 404 Not Found, instead of 403
Forbidden, in some places. This is to prevent the accidental leakage
of private repositories to unauthorized users.
You can do this by creating an OAuth2 token and setting it as an HTTP header:
request['Authorization'] = 'token YOUR_OAUTH_TOKEN'
You can also pass the OAuth2 token as a POST parameter:
uri.query = URI.encode_www_form(access_token: 'YOUR_OAUTH_TOKEN')
# Encoding really isn't necessary though for this though, so this should suffice
uri.query = 'access_token=YOUR_OAUTH_TOKEN'
It has been a while since I have used Rails. I currently have a curl request as follows
curl -X GET -H 'Authorization: Element TOKEN, User TOKEN' 'https://api.cloud-elements.com/elements/api-v2/hubs/marketing/ping'
All I am looking to do is to be able to run this request from inside of a rails controller, but my lack of understanding when it comes to HTTP requests is preventing me from figuring it out to how best handle this. Thanks in advance.
Use this method for HTTP requests:
def api_request(type , url, body=nil, header =nil )
require "net/http"
uri = URI.parse(url)
case type
when :post
request = Net::HTTP::Post.new(uri)
request.body = body
when :get
request = Net::HTTP::Get.new(uri)
when :put
request = Net::HTTP::Put.new(uri)
request.body = body
when :delete
request = Net::HTTP::Delete.new(uri)
end
request.initialize_http_header(header)
#request.content_type = 'application/json'
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') {|http| http.request request}
end
Your example will be:
api_request(:get, "https://api.cloud-elements.com/elements/api-v2/hubs/marketing/ping",nil, {"Authorization" => "Element TOKEN, User TOKEN" })
It would be something like the following. Note that the connection will be blocking, so it can tie up your server depending on how quickly the remote host returns the HTTP response and how many of these requests you are making.
require 'net/http'
# Let Ruby form a canonical URI from our URL
ping_uri = URI('https://api.cloud-elements.com/elements/api-v2/hubs/marketing/ping')
# Pass the basic configuration to Net::HTTP
# Note, this is not asynchronous. Ruby will wait until the HTTP connection
# has closed before moving forward
Net::HTTP.start(ping_uri.host, ping_uri.port, :use_ssl => true) do |http|
# Build the request using the URI as a Net::HTTP::Get object
request = Net::HTTP::Get.new(ping_uri)
# Add the Authorization header
request['Authorization'] = "Element #{ELEMENT_TOKEN}, User #{user.token}"
# Actually send the request
response = http.request(request)
# Ruby will automatically close the connection once we exit the block
end
Once the block exits, you can use the response object as necessary. The response object is always a subclass (or subclass of a subclass) of Net::HTTPResponse and you can use response.is_a? Net::HTTPSuccess to check for a 2xx response. The actual body of the response will be in response.body as a String.