I'm integrating a communications api and whenever a text/voice reaches my server(rails controller), I have to send back an OK (200) to the api. I want to send this response before executing my code block because if my code breaks (and is unable to send the OK), the communcations api keeps sending the messages for up to 3 days. Now that just complicates the problem already on my server because it would keep breaking as the same message keeps on coming.
I did some research and found two solutions.
Solution 1: The first solution is below (my current implementation) and it doesnt seem to be working (unless I didnt read the log files properly or I'm hallucinating).
def receive_text_message
head :ok, :content_type => 'text/html'
# A bunch of code down here
end
I thought this should do (per rails doc), but I'm not sure it does.
Solution 2: the second implementation which I'm contemplating is to quickly create a new process/thread to execute the code block and kill off the process that received the message...that way the api gets its OK very quickly and it doesnt have to wait on the successful execution of my code block. I could the spawnling (or spawn) gem to do this. I would go with creating a process since I use passenger (community) server. But new processes would eat up more RAM, plus I think it is harder to debug child processes/thread (i might be wrong on this)
Thanks for the help!
Side question: does rails attempt to restart a process after it just failed?
You could opt for returning a 200 in your controller and start a sidekiq job. That way the 200 will be returned immediately and your controller will be ready to process the next job. So no waste of time and resources in your controller. The let the worker to do the real hard job.
In your controller
def receive_text_message
head :ok, :content_type => 'text/html'
HardWorker.perform_async(params)
end
In your sidekiq worker:
class HardWorker
include Sidekiq::Worker
def perform(params)
# 'Doing hard work'
end
end
I like sidekiq mostly because it is handling the resources more nicely compared to rescue.
Related
I have a ruby on rails web application deployed on Heroku.
This web app fetches some job feeds of given URLs as XMLs. Then regulates these XMLs and creates a single XML file. It worked pretty well for a while. However, since the #of URLs and job ads increases, it does not work at all. This process sometimes takes up to 45 secs since there are over 35K job vacancies (Heroku sends timeout after 30 secs). I am having an H12 timeout error. This error led me to read this worker dynos and background processing.
I figured out that I should apply the approach below:
Scalable-approach Heroku
Now I am using Redis and Sidekiq on my project. And I am able to create a background worker to do all the dirty work. But here is my question.
Instead of doing this call in the controller class:
def apply
send_data Aggregator.new(providers: providers).call,
type: 'text/xml; charset=UTF-8;',
disposition: 'attachment; filename=indeed_apply_yes.xml'
end
I am doin this perform_async call.
def apply
ReportWorker.perform_async(Time.now)
redirect_to health_path #and returns status 200 ok
end
I implemented this class: ReportWorker calls the Aggregator Service. data_xml is the field that I need to show somewhere or be downloaded automatically when it's ready.
class ReportWorker
include Sidekiq::Worker
sidekiq_options retry: false
data_xml = nil
def perform(start_date)
url_one = 'https://www.examplea.com/abc/download-xml'
url_two = 'https://www.exampleb.com/efg/download-xml'
cursor = 'stop'
providers = [url_one, url_two, cursor]
puts "SIDEKIQ WORKER GENERATING THE XML-DATA AT #{start_date}"
data_xml = Aggregator.new(providers: providers).call
puts "SIDEKIQ WORKER GENERATED THE XML-DATA AT #{Time.now}"
end
end
I know that It's not recommended to make send_data/file methods accessible out of Controller classes. Well, any suggestions on how to do it?
Thanks in advance!!
Do you can set up some database on your application? And then store record about completed jobs there, also you can save the entire file in database, but i recommend some cloud storage (like amazon s3).
And after that you can show current status of queued jobs on some page for user, with button 'download' after job has done
I have an API which uses a Service, in which I have used Ruby thread to reduce the response time of the API. I have tried to share the context using the following example. It was working fine with Rails 4, ruby 2.2.1
Now, we have upgraded rails to 5.2.3 and ruby 2.6.5. After which service has stopped working. I can call the service from Console, it works fine. But with API call, service becomes unresponsive once it reaches CurrencyConverter.new. Any Idea what can be the issue?
class ParallelTest
def initialize
puts "Initialized"
end
def perform
# Our sample set of currencies
currencies = ['ARS','AUD','CAD','CNY','DEM','EUR','GBP','HKD','ILS','INR','USD','XAG','XAU']
# Create an array to keep track of threads
threads = []
currencies.each do |currency|
# Keep track of the child processes as you spawn them
threads << Thread.new do
puts currency
CurrencyConverter.new(currency).print
end
end
# Join on the child processes to allow them to finish
threads.each do |thread|
thread.join
end
{ success: true }
end
end
class CurrencyConverter
def initialize(params)
#curr = params
end
def print
puts #curr
end
end
If I remove the CurrencyConverter.new(currency), then everything works fine. CurrencyConverter is a service object that I have.
Found the Issue
Thanks to #anothermh for this link
https://guides.rubyonrails.org/threading_and_code_execution.html#wrapping-application-code
https://guides.rubyonrails.org/threading_and_code_execution.html#load-interlock
As per the blog, When one thread is performing an autoload by evaluating the class definition from the appropriate file, it is important no other thread encounters a reference to the partially-defined constant.
Only one thread may load or unload at a time, and to do either, it must wait until no other threads are running application code. If a thread is waiting to perform a load, it doesn't prevent other threads from loading (in fact, they'll cooperate, and each perform their queued load in turn, before all resuming running together).
This can be resolved by permitting concurrent loads.
https://guides.rubyonrails.org/threading_and_code_execution.html#permit-concurrent-loads
Rails.application.executor.wrap do
urls.each do |currency|
threads << Thread.new do
CurrencyConverter.new(currency)
puts currency
end
ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
threads.map(&:join)
end
end
end
Thank you everybody for your time, I appreciate.
Don't re-invent the wheel and use Sidekiq instead. 😉
From the project's page:
Simple, efficient background processing for Ruby.
Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails to make background processing dead simple.
With 400+ contributors, and 10k+ starts on Github, they have build a solid parallel job execution process that is production ready, and easy to setup.
Have a look at their Getting Started to see it by yourself.
How can I detect if a particular request is still active?
For example I have this request uuid:
# my_controller.rb
def my_action
request.uuid # -> ABC1233
end
From another request, how can I know if the request with uuid ABC1233 is still working?
For the curious:
Following beanstalk directives I am running cron jobs using URL requests.
I don't want to start the next iteration if the previous one is still running. I can not just relay in a ini/end flag updated by the request because the request some times dies before it finishes.
Using normal cron tasks I was managing this properly using the PID of the process.
But I don't think I can use PID any more because processes in a web server can be reused among different requests.
I don't think Rails (or more correctly, Rack) has support for this since (to the best of my knowledge) each Rails request doesn't know about any other requests. You may try to get access to all running threads (and even processes) but such implementation (if even possible) seems ugly to me
.
How about implementing it yourself?
class ApplicationController < ActionController::Base
before_filter :register_request
after_filter :unregister_request
def register_request
$redis.set request.uuid
end
def unregister_request
$redis.unset request.uuid
end
end
You'll still need to figure out what to do with exceptions since after_filters are skipped (perhaps move this whole code to a middleware: on the before phase of the middleware it writes the uuid to redis and on the after phase it removes the key ). There's a bunch of other ways to achieve this I'm sure and obviously substitute redis with your favorite persistence of choice.
Finally I recovered my previous approach based on PIDs.
I implemented something like this:
# The Main Process
module MyProcess
def self.run_forked
Process.fork do
SynchProcess.run
end
Process.wait
end
def self.run
RedisClient.set Process.pid # store the PID
... my long process code is here
end
def self.still_alive?(pid)
!!Process.kill(0, pid) rescue false
end
end
# In one thread I can do
MyProcess.run_forked
# In another thread I can do
pid = RedisClient.get
MyProcess.still_alive?(pid) # -> true if the process still running
I can call this code from a Rails request and even if the request process is reused the child one is not and I can monitor the PID of the child process to see if the Ruby process is still running.
I am new to Resque. I want to make and send an excel file with the help of a Resque worker. I I have written that code in worker class. Now I have called the worker inside the controller function as:
Resque.redis = Redis.new(:host => "#{Rails.configuration.redis_machine}", :port => Rails.configuration.redis_port)
Resque.enqueue(MakeAndSend,records,email)
Here records is rows of data that are to be printed to the excel file and email is the mail id of the user to which this excel file is to be sent.
But the problem is that my code is get stuck at this line, the following line is getting executed only when the job is over. But as far as I understood from studying resque that the following line should execute immediately.
So if anyone helps me to figure this out, I will be really grateful.
EDIT Here above records is actually collection of objects. So I don't have any option to send ids here. Also, the redis server is not getting hit.
Looking at it, I don't think it would be that line which is the problem - it will likely be something behind that line, such as the worker file or something
I can see one main issue you'll likely have, and there will be others:
-> You're passing variables (ActiveRecord objects) to the Resque queue -- it's convention to send the id's of the variables so the worker file can call the databases when it runs
Here is some live code for our Resque queues:
#app/models/message.rb (this gets called when we run "send_broadcast" function)
def broadcast!
self.subscribers.each do |subscriber|
Resque.enqueue(MailoutQueue, id, subscriber.id)
end
end
#app/workers/mailout_queue.rb
class MailoutQueue
#queue = :mailer
def self.perform(message_id, recipient_id)
MessageMailer.send_message(message_id, recipient_id).deliver
end
end
MessageMailer.rb is just a standard message sender file
If you can post your worker file code, it will help us appreciate where the real issue maybe, so we can fix appropriately!
We're using ElasticSearch with the Tire gem on a Rails app. For our integration tests, we delete and recreate the index before each example, something along the lines of
Foo.index.delete
Foo.create_elasticsearch_index
Where Foo includes Tire::Model::Persistence. But we started having missing index errors from time to time when running our test suite on CI. I went ahead and enabled debugging and then I found the following in the logs:
# 2013-10-04 09:25:05:839 [DELETE] ("test_index")
#
curl -X DELETE http://some-server:9200/test_index
# 2013-10-04 09:25:05:840 [200]
#
# {
# "ok": true,
# "acknowledged": true
# }
# 2013-10-04 09:25:05:852 [HEAD] ("test_index")
#
curl -I "http://some-server:9200/test_index"
# 2013-10-04 09:25:05:852 [200]
As you can see, I get a 200 ok response for the DELETE request, but then when Tire does a HEAD request to see if the index exists before creating it, it still returns a 200 instead of a 404. This happens randomly, most of the times it works ok, but at some point in the test suite it will fail.
I tried waiting for a yellow status between the delete and the create operations, but it didn't work. So my question here would be, is the delete index operation asynchronous in some way? (Couldn't find anything on the ES doc about that). Is there any way to wait for the index to be deleted? (Since the yellow status doesn't work).
Edit: Thanks #PinnyM for the clarification. So all the operations through the HTTP API are indeed asynchronous. So the question that remains is, how can I wait for the index to be deleted so I can create it again?
Before each example, you delete and recreate the index. As part of that process you could then ensure that the delete process has completed (asynchronously) by polling for the exists function to return false... In crappy pseudo-code...
max_wait = 5
while wait < max_wait and Tire.index("test_index").exists:
wait some more
if wait > max_wait:
throw WaitedTooLongException()
The Tire docs for exist are here - essentially they are doing the same index polling you found in the debug code:
http://rubydoc.info/github/karmi/tire/master/Tire/Index#exists?-instance_method and here's the call: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-exists.html
Personally, I would adjust your tests that use the ES instance... If you put the not-exists loop at the start of the test and the index tear-down at the end of the test, then you could get two benefits. You could:
Assert that the index didn't exist when testing started - nice and clean.
Ask for index tear down at the end of a test and run your next test which might not need ES - possible slight speed-up depending on your App and CI implementation of test ordering.
To wait for asynchronous operations to finish I am using in my RSpecs
collection.__elasticsearch__.refresh_index!
Very useful when I just indexed a new document and I want to run a search on it immediately after in my spec.
the .refresh_index! is a blocking method. Actually I believe all ! methods might be... have you tried the other ! methods ?
collection.__elasticsearch__.delete_index!
collection.__elasticsearch__.create_index!
(Using ES 5.x with the appropriate elasticsearch-ruby/rails gems without Tire)