How to properly use Sidekiq to process background tasks in Rails - ruby-on-rails

Question

So, i've generated a rails app using https://github.com/Shopify/shopify_app - and for the most part have the app working as intended - it's goal is to get product quantities from an external stock management API, and then update the variant quantities in Shopify with the latest quantities from that stock management system.
My problem is that the initial POST request to the external API responds with a large number of products - this takes upwards of 15 seconds sometimes. In addition to this, another portion of my app then takes this response, and for every product in the response that also exists in Shopify, it will make a PUT request to Shopify to update the variant quantities. As with the initial request, this also takes upwards of 10-15 seconds.
My problem is that i'm hosting the app on Heroku, and as a result i've hit their 30 second request timeout limit. As a result I need to use a background worker to offset at least one of the requests above (perhaps both) to a worker queue. I've gone with the widely recommended Sidekiq gem - https://github.com/mperham/sidekiq - which is easy enough to set up.
My problem is that I don't know how to get the results from the finished Sidekiq worker job, and then use that again within the Controller - I also don't know if this is best practice (i'm a little new to Rails/App development).
I've included my controller (prior to breaking it down into workers) that currently runs the app below - I guess I just need some advice - am I doing this correctly - should some of this logic be inside a Model, and if so how would that model then communicate with the Controller, and then how would Sidekiq then fit into all of it.
Appreciate any advice or assistance, thanks.
class StockManagementController < ShopifyApp::AuthenticatedController
require 'uri'
require 'net/http'
require 'json'
require 'nokogiri'
require 'open-uri'
require 'rexml/document'
def new
#token = StockManagementController.new
end
def get_token
url = URI('https://external.api.endpoint/api/v1/AuthToken')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
#HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
#HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']
request = Net::HTTP::Post.new(url)
request['content-type'] = 'application/x-www-form-urlencoded'
request['cache-control'] = 'no-cache'
request.body = 'username=' + #HEROKU_ENV_USERNAME + '&password=' + #HEROKU_ENV_PASSWORD + '&grant_type=password'
response = http.request(request)
responseJSON = JSON.parse(response.read_body)
session[:accessToken] = responseJSON['access_token']
if session[:accessToken]
flash[:notice] = 'StockManagement token generation was successful.'
redirect_to '/StockManagement/product_quantity'
else
flash[:alert] = 'StockManagement token generation was unsuccessful.'
end
end
def product_quantity
REXML::Document.entity_expansion_text_limit = 1_000_000
#theToken = session[:accessToken]
if #theToken
url = URI('https://external.api.endpoint/api/v1/ProductQuantity')
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(url)
request['authorization'] = 'bearer ' + #theToken + ''
request['content-type'] = 'application/xml'
request['cache-control'] = 'no-cache'
response = http.request(request)
responseBody = response.read_body
finalResponse = Hash.from_xml(responseBody).to_json
resultQuantity = JSON.parse finalResponse
#connectionType = resultQuantity['AutomatorResponse']['Type']
#successResponse = resultQuantity['AutomatorResponse']['Success']
#errorResponse = resultQuantity['AutomatorResponse']['ErrorMsg']
productQuantityResponse = resultQuantity['AutomatorResponse']['ResponseString']
xmlResponse = Hash.from_xml(productQuantityResponse).to_json
jsonResponse = JSON.parse xmlResponse
#fullResponse = jsonResponse['StockManagement']['Company']['InventoryQuantitiesByLocation']['InventoryQuantity']
# This hash is used to store the final list of items that we need in order to display the item's we've synced, and to show the number of items we've sycned successfully.
#finalList = Hash.new
# This array is used to contain the available products - this is used later on as a way of only rendering
#availableProducts = Array.new
# Here we get all of the variant data from Shopify.
#variants = ShopifyAPI::Variant.find(:all, params: {})
# For each peace of variant data, we push all of the available SKUs in the store to the #availableProducts Array for use later
#variants.each do |variant|
#availableProducts << variant.sku
end
#Our final list of products which will contain details from both the Stock Management company and Shopify - we will use this list to run api calls against each item
#finalProductList = Array.new
puts "Final product list has #{#fullResponse.length} items."
puts #fullResponse.inspect
# We look through every item in the response from Company
#fullResponse.each_with_index do |p, index|
# We get the Quantity and Product Code
#productQTY = p["QtyOnHand"].to_f.round
#productCode = p["Code"].upcase
# If the product code is found in the list of available products in the Shopify store...
if #availableProducts.include? #productCode
#variants.each do |variant|
if #productCode === variant.sku
if #productQTY != 0
#finalProductList << {
"sku" => variant.sku,
"inventory_quantity" => variant.inventory_quantity,
"old_inventory_quantity" => variant.old_inventory_quantity,
"id" => variant.id,
"company_sku" => #productCode,
"company_qty" => #productQTY
}
end
end
end
end
end
# If we get a successful response from StockManagement, proceed...
if #finalProductList
flash[:notice] = 'StockManagement product quantity check was successful.'
puts "Final product list has #{#finalProductList.length} items."
puts #finalProductList
#finalProductList.each do |item|
#productSKU = item["sku"]
#productInventoryQuantity = item["inventory_quantity"]
#productOldInventoryQuantity = item["old_inventory_quantity"]
#productID = item["id"]
#companySKU = item["company_sku"]
#companyQTY = item["company_qty"]
url = URI("https://example.myshopify.com/admin/variants/#{#productID}.json")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Put.new(url)
request["content-type"] = 'application/json'
request["authorization"] = 'Basic KJSHDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDFKJHSDF'
request["cache-control"] = 'no-cache'
request.body = "{\n\t\"variant\": {\n\t\t\"id\": #{#productID},\n\t\t\"inventory_quantity\": #{#companyQTY},\n\t\t\"old_inventory_quantity\": #{#productOldInventoryQuantity}\n\t}\n}"
# This is the line that actually runs the put request to update the quantity.
response = http.request(request)
# Finally, we populate the finalList has with response information.
#finalList[#companySKU] = ["","You had #{#productOldInventoryQuantity} in stock, now you have #{#companyQTY} in stock."]
end
else
# If the overall sync failed, we flash an alert.
flash[:alert] = 'Quantity synchronisation was unsuccessful.'
end
# Lastly we get the final number of items that were synchronised.
#synchronisedItems = #finalList.length
# We flash this notification, letting the user known how many products were successfully synchronised.
flash[:notice] = "#{#synchronisedItems} product quantities were synchronised successfully."
# We then pretty print this to the console for debugging purposes.
pp #finalList
else
flash[:alert] = #errorResponse
end
end
end

Answer 1

First of all, your product_quantity method is way too long. You should break it into smaller parts. 2nd, http.verify_mode = OpenSSL::SSL::VERIFY_NONE should not be done in production. The example you've provide along with your question are too complex and are therefore difficult to answer. It sounds like you need a basic understanding of design patterns and this is not a specific ruby question.
If your app needs to make realtime API calls inside of a controller this is a poor design. You don't want to keep requests of any kind waiting for more than a couple of seconds at most. You should consider WHY you need to make these requests in the first place. If it's data you need quick access to, you should write background jobs to scrape the data on a schedule and store it in your own database.
If a user of your app makes a request which needs to wait for the API's response, you could write a worker to handle fetching the API data and eventually send a response to the user's browser probably using actioncable.
For your constant definitions you probably should do this in an initializer wihich you would keep in my_app_root/config/initializers/constants.rb which get loaded into your app at runtime. You could just call them where need using te ENV[] syntax but if you prefer simpler constants drop the # since that naming convention in ruby is for instance objects.
#app_root/config/initializers/constants.rb
HEROKU_ENV_USERNAME = ENV['HEROKU_ENV_USERNAME']
HEROKU_ENV_PASSWORD = ENV['HEROKU_ENV_PASSWORD']