Rails - multiple theads to avoid the slack 3 second API response rule - ruby-on-rails

I am working with the slack API. My script does a bunch of external processing and in some cases it can take around 3-6 seconds. What is happening is the Slack API expects a 200 response within 3 seconds and because my function is not finished within 3 seconds, it retries again and then it ends up posting the same automated responses 2-3 times.
I confirmed this by commenting out all the functions and I had no issue, it posted the responses to slack fine. I then added sleep 10 and it done the same responses 3 times so the ohly thing different was it took longer.
From what I read, I need to have threaded responses. I then need to first respond to the slack API in thread 1 and then go about processing my functions.
Here is what I tried:
def events
Thread.new do
json = {
"text": "Here is your 200 response immediately slack",
}
render(json: json)
end
puts "--------------------------------Json response started----------------------"
sleep 30
puts "--------------------------------Json response completed----------------------"
puts "this is a successful response"
end
When I tested it the same issue happened so I tried using an online API tester and it hits the page, waits 30 seconds and then returns the 200 response but I need it to respond immediately with the 200, THEN process the rest otherwise I will get duplicates.
Am I using threads properly or is there another way to get around this Slack API 3 second response limit? I am new to both rails and slack API so a bit lost here.
Appreciate the eyes :)

I would recommend using ActionJob to run the code in the background if you don't need to use the result of the code in the response. First, create an ActiveJob job by running:
bin/rails generate job do_stuff
And then open up the file created in app/jobs/do_stuff_job.rb and edit the #perform function to include your code (so the puts statements and sleep 30 in your example). Finally, from the controller action you can call DoStuff.perform_later and your job will run in the background! Your final controller action will look something like this:
def events
DoStuff.perform_later # this schedules DoStuff to be done later, in
# the background, so it will return immediately
# and continue to the next line.
json = {
"text": "Here is your 200 response immediately slack",
}
render(json: json)
end
As an aside, I'd highly recommend never using Thread.new in rails. It can create some really confusing behavior especially in test scripts for a number of reasons, but usually because of how it interacts with open connections and specifically ActiveRecord.

Related

How to get Twilio Studio Flow execution via call sid

We're trying to include the IVR steps in our UI, but to get the steps I have to make several API calls. That's fine, except the only way I can seem to get the relevant info is to load all flow executions.
If I could pass the flow.sid via the HTTP Request widget then I could go fetch the info I need later instead of having to iterate through all the previous executions. I tried passing {{flow.data}} as the request body, thinking it was JSON, but it ends up being empty.
Here's a spike that someone wrote for us, modified to just work with a single execution.
require "httparty"
STUDIO_FLOW_SID = "FW***"
AUTH = {username: ENV["TWILIO_ACCOUNT_SID"], password: ENV["TWILIO_AUTH_TOKEN"]}
DATE_CREATED_FROM = "2019-09-01T000000Z"
DATE_CREATED_TO = "2019-10-01T000000Z"
# Retrieves all executions in the given date range
executions_url = "https://studio.twilio.com/v1/Flows/#{STUDIO_FLOW_SID}/Executions?DateCreatedFrom=#{DATE_CREATED_FROM}&DateCreatedTo=#{DATE_CREATED_TO}"
response = HTTParty.get(executions_url, basic_auth: AUTH)
# If I can get the individual execution from the IVR {{flow.data}}
# that would be ideal
execution = response.parsed_response["executions"].first
execution_context_url = execution["links"]["execution_context"]
execution_context = HTTParty.get(execution_context_url, basic_auth: AUTH)
# Or, if I could work backwards and get the execution context ID from
# the call somehow, that would work too.
call_sid = execution_context.parsed_response["context"]["trigger"]["call"]["CallSid"]
steps = HTTParty
.get(execution["links"]["steps"], basic_auth: AUTH)
.parsed_response["steps"]
.sort_by { |step| step["date_created"] }
.map { |step| step["transitioned_to"] }
.select { |step| step.include?("option") || step.include?("menu") }
puts [call_sid, steps].inspect
tl;dr -
I either need the Flow execution info passed in an HTTP Request widget or I need to work backwards from a CallSid to get the execution steps.
Twilio developer evangelist here.
The execution Sid can be accessed in the flow's data, under flow.sid.
This was missing in the documentation, but I've just added it here: https://www.twilio.com/docs/studio/user-guide#context-variables
Note: {{flow.sid}} doesn't currently appear in Studio's autocomplete, but it's there, I promise!
I found a way to get the execution from the call:
Get the list of incoming phone numbers: https://www.twilio.com/docs/phone-numbers/api/incomingphonenumber-resource#read-multiple-incomingphonenumber-resources
Search for the number using the destination of the call
Then you can get the flowid by parsing voice_url
With this flow_id and the date the call arrived, you can get the list of executions: https://www.twilio.com/docs/studio/rest-api/v2/execution#read-a-list-of-executions
If you have more than one, you can use the calling number to search in the list.
Anyway, if you are calling the function from Studio, philnash answer is better :)

How would I use rspec to test a method who's job is to post to a webhook?

I'm using rspec to test my application and I'm having a hard time figuring out how to test this. The Slack::Notifier's job is to send a post request to a webhook. Once I call this method in Rspec, I don't know how to see the response. Also, is it possible to match the format of this text to an expected text somewhere? My method is below. Thanks.
def notify
offset = 14400 #UTC to EST
notifier = Slack::Notifier.new Rails.application.secrets.slack_organization_name, Rails.application.secrets.slack_token, channel: "##{Rails.application.secrets.slack_channel}", username: Rails.application.secrets.slack_user_name
notifier.ping(":white_check_mark: *USAGE SUMMARY for #{(Time.now - offset).to_formatted_s(:long) }*")
count = 0
current_time = Time.now.to_i
live_response.each do |r|
if r["properties"]["time"] > ((current_time - offset) - 60) #&& r["properties"]["$initial_referring_domain"] == "capture.com"
notifier.ping("
*Name:* #{r["properties"]["$name"]}
*Event:* #{r["event"]}
*Keywords:* #{r["properties"]["keywords"]}
*Organization:* #{r["properties"]["organizationName"]}
*Email:* #{r["properties"]["$email"]}
*Time:* #{Time.at(r["properties"]["time"] + offset).utc.to_datetime.in_time_zone("Eastern Time (US & Canada)").to_formatted_s(:long_ordinal)}
*More Data:* #{ANALYTICS_URL}#{r["properties"]["distinct_id"]}
__________________________________________________
")
count +=1
end
end
notifier.ping("*There were #{count} events in this report.*")
end
Testing network communications (like API calls) is a tricky thing. Personally I would rely on programming by contract and testing in isolation - i.e. assume the external service is working fine and it responds positively for valid request.
Then you test your client code by checking that you are actually sending a valid request. For this stub the method where control exits your code into a library/system code. For example if you are making a HTTP GET request using a gem like HTTParty, then stub HTTParty.get i.e. HTTParty.stub(:get) and in that stub verify that correct parameters were sent.
On the other side of the spectrum you should also simulated both positive and negative responses from the web service and make sure your client code handles it in expected manner.
If you are making a real then you are introducing a lot of dependencies on your test : a test setup of external service, risk of network issues (timeout, n/w breakdown, etc) problems with external service and may be more.
If you yourself are writing that webservice too then test that one also in isolation, i.e by simulating valid and invalid inputs and making sure they are handled properly. This part is pretty much your controller specs or request specs.
Once again, this is my opinion. Suggestions to do this in a better way and constructive criticism on the shortcomings of this approach are definitely welcome.

Queuing API calls to fit rate limit

Using Full Contact API, but they have a rate limit of 300calls/minute. I currently have it to set that it does an API call when uploading the CSV file of emails. I want to queue it such that once it hits the rate limit or does 300 calls, it waits for 1 minute and proceeds. Then I will put delayed_job on it. How can I do that? A quick fix is to use
sleep 60
but how do I find it such that it made 300 calls already, make it sleep or queue it for next set?
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
hashy = row.to_hash
email = hashy["email"]
begin
Contact.create!(email: email, contact_hash: FullContact.person(email: email).to_json)
rescue FullContact::NotFound
Contact.create!(email: email, contact_hash: "Not Found")
end
end
end
There are several issues to think about here - is there going to be a single process using your API key at any one time, or is it possible that multiple processes would be running at once? If you have multiple delayed_job workers, I think the latter is likely. I haven't used delayed_jobs enough to give you a good solution to that, but my feeling is you would be restricted to a single worker.
I am currently working on a similar problem with an API with a restriction of 1 request every 0.5 seconds, with a maximum of 1000 per day. I haven't worked out how I want to track the per-day usage yet, but I've handled the per-second restriction using threads. If you can frame the restriction as "1 request every 0.2 seconds", that might free you up from having to track it on a minute-by-minute basis (though you still have the issue of how to keep track multiple workers).
The basic idea is that I have an request method that splits a single request into a queue of request parameters (based on the maximum number of objects allowed per request by the api), and then another method iterates over that queue and calls a block which sends the actual request to the remote server. Something like this:
def make_multiple_requests(queue, &block)
result = []
queue.each do |request|
timer = Thread.new { sleep REQUEST_INTERVAL }
execution = Thread.new { result << yield(request) }
[timer, execution].each(&:join)
end
result
end
To use it:
make_multiple_requests(queue) do |request|
your_request_method_goes_here(request)
end
The main benefit here is that if a request takes longer than the allowed interval, you don't have to wait around for the sleep to finish, and you can start your next request right away. It just guarantees that the next request won't start until at least the interval has passed. I've noticed that even though the interval is set correctly, I occasionally get an 'over-quota' response from the API. In those cases, the request is retried after the appropriate interval has passed.

Optimal way to structure polling external service (RoR)

I have a Rails application that has a Document with the flag available. The document is uploaded to an external server where it is not immediately available (takes time to propogate). What I'd like to do is poll the availability and update the model when available.
I'm looking for the most performant solution for this process (service does not offer callbacks):
Document is uploaded to app
app uploads to external server
app polls url (http://external.server.com/document.pdf) until available
app updates model Document.available = true
I'm stuck on 3. I'm already using sidekiq in my project. Is that an option, or should I use a completely different approach (cron job).
Documents will be uploaded all the time and so it seems relevant to first poll the database/redis to check for Documents which are not available.
See this answer: Making HTTP HEAD request with timeout in Ruby
Basically you set up a HEAD request for the known url and then asynchronously loop until you get a 200 back (with a 5 second delay between iterations, or whatever).
Do this from your controller after the document is uploaded:
Document.delay.poll_for_finished(#document.id)
And then in your document model:
def self.poll_for_finished(document_id)
document = Document.find(document_id)
# make sure the document exists and should be polled for
return unless document.continue_polling?
if document.remote_document_exists?
document.available = true
else
document.poll_attempts += 1 # assumes you care how many times you've checked, could be ignored.
Document.delay_for(5.seconds).poll_for_finished(document.id)
end
document.save
end
def continue_polling?
# this can be more or less sophisticated
return !document.available || document.poll_attempts < 5
end
def remote_document_exists?
Net::HTTP.start('http://external.server.com') do |http|
http.open_timeout = 2
http.read_timeout = 2
return "200" == http.head(document.path).code
end
end
This is still a blocking operation. Opening the Net::HTTP connection will block if the server you're trying to contact is slow or unresponsive. If you're worried about it use Typhoeus. See this answer for details: What is the preferred way of performing non blocking I/O in Ruby?

How to test for asynchronous HTTP requests in ruby using EventMachine

I'm getting messages of a RabbitMQ queue and each message is a URL that I want to make a request to. Now I'm using the AMQP gem to subscribe to the queue and that uses EventMachine, so I'm using the the em-http-request library to make the http requests. According to the documentation here: https://github.com/igrigorik/em-http-request/wiki/Parallel-Requests
The following will issue asynchronous http-requests:
EventMachine.run {
http1 = EventMachine::HttpRequest.new('http://google.com/').get
http2 = EventMachine::HttpRequest.new('http://yahoo.com/').get
http1.callback { }
http2.callback { }
end
So when I subscribe to the RabbitMQ queue I have the following code:
x = 0
EventMachine.run do
connection = AMQP.connect(:host => '127.0.0.1')
channel = AMQP::Channel.new(connection)
channel.prefetch(50)
queue = channel.queue("http.requests")
exchange = channel.direct("")
queue.subscribe do |metadata, payload|
url = payload.inspect
eval "
#http#{x} = EventMachine::HttpRequest.new(url).get
#http#{x}.callback do
puts \"got a response\"
puts #http#{x}.response
end
x = x+1
"
end
end
This dynamically creates new variables and creates new http requests, similar to the way described in the em-http-request documentation. But is there a way to test whether the requests are actually being made asynchronously? Is it possible to write to the console every time a get request is fired off so I can see they are fired off one after the other without waiting for a response?
You can try running tcpdump and analysing the output. If you see the TCP three-way handshakes for the two connections being interleaved then the connections are happening in parallel.
This can't really be part of an automated test though, if that's what you're trying to aim for. I would be happy just to verify that the library does what it says it does once and not make it part of a test suite.
A very simple example, demonstrating exactly what you want:
require 'em-http-request'
EM.run do
# http://catnap.herokuapp.com/3 delays the HTTP response by 3 seconds.
http1 = EventMachine::HttpRequest.new('http://catnap.herokuapp.com/3').get
http1.callback { puts 'callback 1' }
http1
puts 'fired 1'
http2 = EventMachine::HttpRequest.new('https://www.google.com/').get
http2.callback { puts 'callback 2' }
puts 'fired 2'
end
Output (for me):
fired 1
fired 2
callback 2
callback 1
Depending on your internet connection, Heroku and Google, the response to the second HTTP request will likely come in first and you can be sure, the requests are indeed done in parallel.

Resources