Rails Concurrency issue lockup - ruby-on-rails

Question

I have a rails app where an action never finishes and then times out.
Find the diagram below for better illustration.
My rails apps action is called
The action POSTs some data to another app
The other app needs something to complete the computation and calls a different action than the first of the Rails app
The other app gets a response and finishes the computation
The other app responds to the rails apps POST request
The view is rendered accordingly
Now the issue: The other app never gets a response from the main app. After the Rails apps request times out however, the response is sent (however too late of course) so I think it is somehow cued.
I don't understand how to fix that. I use rails 5 and Puma which should be able to handle parallel calls. Its also not a local issue, same happens in prod.
I use the recommended puma.rb config from Heroku
workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
ActiveRecord::Base.establish_connection
end
What do I do to fix this cueing?
Controller:
# New method
def live_preview_page
preview_locale = params[:preview_locale]
date = params[:date] # The date to preview
page_id = params[:id]
return if locale.nil? || locale =~ /not/ || date.nil?
all_templates = Template.all.order('name ASC') # Maybe move to render_live_editor_page
if date == "all"
active_modules = #page.page_modules.order(rank: :asc)
else
active_modules = #page.page_modules.order(rank: :asc).to_a.valid_for(date: date.to_date)
puts "Active modules: #{active_modules.count}"
end
active_modules_json = active_modules.each do |content_module|
content_module.body = YAML.load(content_module.body).to_json
end
response = helpers.render_preview(active_modules, all_templates, preview_locale)
renderer = ContentRenderer.new
actionController = ActionController::Base.new
rendered_helper = actionController.render_to_string(
partial: '/pages/preview-helper-snippet', locals: {
all_templates: all_templates, # For select when creating new modules
modulesData: active_modules_json, # For rendering the JSON containing the data for the editor
current_page: #page.id,
localeLinks: renderer.generateStgPreviewURLs(SettingService.get_named_locales, #page.id),
locale: preview_locale,
all_locales: SettingService.locales_for_live_editor,
all_sites_and_locales: SettingService.get_sites_and_locales
})
proxy_service = ProxyService.new
proxy_service.get_page do |error, page_wrapper|
# Note: Issue is that Vapor app generates warnings inline template : encountered \r in middle of line, treated as a mere space
rendered_body_with_helper = response.body.force_encoding("UTF-8") + rendered_helper
decorated_page = page_wrapper.gsub("__WIDGET__", rendered_body_with_helper)
render inline: decorated_page
return
end
end
Helper
def render_preview(active_modules, all_templates, preview_locale)
req = Request.new
preview_body = {
modules: active_modules,
templates: all_templates,
sites: SettingService.get_sites,
configuration: {
locale: preview_locale,
site: "DE"
}
}
req.send_request(
url: "#{ENV["RENDER_SERVICE_URL"]}/preview",
body: preview_body,
options: {
type: :post,
json: true,
username: ENV["RENDER_SERVICE_BASIC_AUTH_USERNAME"],
password: ENV["RENDER_SERVICE_BASIC_AUTH_PASSWORD"]
}
) do |response_code, response|
return response
end
end
Request is just a thin wrapper
require "uri"
require "net/http"
class Request
# Yields resonse_code (int), response
# Parameters besides url: are optional
def send_request(url:, body: {}, header: {}, options: {})
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
if options.key? :type
case options[:type]
when :get
request = Net::HTTP::Get.new(uri.request_uri, header)
when :post
request = Net::HTTP::Post.new(uri.request_uri, header)
end
else
# Default is GET
request = Net::HTTP::Get.new(uri.request_uri, header)
end
if options.key?(:username) && options.key?(:password)
request.basic_auth options[:username], options[:password]
end
unless body.class == String
body = body.to_json.to_s
end
request.body = body unless body.empty?
puts request.body
# SSL is default
if options.key? :ssl
http.use_ssl = options[:ssl]
else
http.use_ssl = Rails.configuration.force_ssl
#http.verify_mode = OpenSSL::SSL::VERIFY_NONE
end
if options.key? :json
request.add_field("Content-Type", "application/json")
end
response = http.request(request)
yield response.code.to_i, response
end
end

Answer 1

The following answer is (probably) not the answer you want - but it's the answer you need:
The best way to fix this is to avoid the loop in the request/response logic (where the rails app calls itself through the other app).
Concurrency might help delay the onset of the issue, but the issue will always occur as long as the loop exists.
For example, assume you have an 100 requests from clients to the Rails app.
Rails will call the other app and the other app's request will be queued as request number 101.
This can be solved with 100 threads (for example, 10 workers with 10 threads each)...
But what will your app do with 200 client requests?
This cycle is endless, the more clients you have the more concurrency you require before you experience DoS.
The only solution is to avoid the loop to begin with.
Either break it up to 3 apps or (better yet), avoid dependencies between micro services.