I am implementing an json api using rails. I wish to make requests to another web service using delayed job to prevent it from blocking my rails app. So far so good. So i have a function defined in my model which does a http POST to this other web service.
However, the other web service is is an asynchronous api with callbacks. Hence I want to also receive callbacks from this api within my delayed job.
Is this possible? Can I have a http listener in my delayed job whose port number I can control or know within my code?
Related
I need to create an API for my rails project, and the API main purpose is to start actions that will run in background for some time (using ActiveJob).
I need the API client to be able to monitor the background job using long-polling. The client makes the request and if the job is still running, the request waits until some timeout to give the result, or returns earlier in casethe job has terminated in the meantime.
If I ware to design a full-stack application, I'd use ActionCable to notify the client when the job terminates. But here, I don't want to make public any ActionCable websocket API and I want to stick with plain HTTP.
Is it possible to use ActionCable to wait for messages on a channel in a Rails controller (server-side)? Can such controller cause a connection count limit, especially with postgresql connections (which I won't be using while I'm waiting on Redis for ActionCable messages)?
I'd love to divide end-points to two types.
One is for normal request, and the other is only to handle webhook requests.
Is this possible to set up another process for webhook handling, and prevent the webhook from blocking the normal request?
I'll go for Unicorn for the application server, and Nginx for web server.
I have a Rails API which can handle requests from the clients. Clients use that API to perform analysis of their data. Client POSTs the data to API, API checks if that data have been analysed before. If so API just respond with analysis result. If the data haven't been analyzed before API:
Tells client that analysis started.
Establishes the connection with analyzing microservice.
Performs asynchronous (or deferred or i don't know) request to the analyzing microservice and waiting for response. The analysis takes much time so neither the API nor the microservice should be blocked while doing it.
When the response from analyzing microservice is returned API hands it to the client.
The main issue for me is to set up things such way that client could receive somehow the message "Your data had been sent to analysis" right after he performed the request. And then when analysis will be done client could receive its result.
The question is what approach I have to use in that case? Async responses, deferred responses, something else? And what known solutions could help me with that? Any gems?
I'm new to that stuff so I'm really sorry if I ask dumb questions.
If using HTTP you can only have one response to every request. To send multiple responses, i.e. "work in progress", then later the "results", you would need to use a different protocol, e.g. web sockets.
Since HTTP is so very common I'd stick with that in combination with background jobs. There are a couple of options which spring to mind.
Polling: The API kicks off a background jobs (to call the microservice) and responds to the client with a URL which the client can ping periodically for the result. The URL would respond with some kind of "work in progress" status until the result is actually ready). The URL would need to include some kind of id so the API can lookup the background job.
The API would potentially have two URLS; /api/jobs/new and /api/jobs/<ID>. They would, in Rails, map to a controller new and show action.
Webhooks: Have the client include a URL of its own in the request. Once the result is available have the background job hit the given URL with the result.
Either way, if using HTTP, you will not be able to handle the whole thing within a request/response, you will have to use some kind of background processing (so request to the microservice happens in a different process). You could look at Sidekiq, for example.
Here is an example for polling:
URL: example.com/api/jobs/new
web app receives client request
generates a unique id for the request, SecureRandom.uuid.
starts a background job (Sidekiq) passing in the uuid and any other parameters needed
respond with URL such as example.com/api/jobs/
--
background job
sends request to microservice API and waits for response
saves result to database with uuid
--
URL: example.com/api/jobs/UUID
look in database for UUID, if not found respond that job is "in progress". If found return result found in database.
Depending on what kind of API you use. I assume your clients interact via HTTP.
If you want to build an asynchronous API over HTTP the first thing that you should do: accept the request, create a job, handle it in the background and immediately return.
For the client to get the response you have to 2 options:
Implement a status endpoint where clients can periodically poll the status of the job
Implement a callback via webhooks. So the client has to provide a URL which you then call after you're done.
A good start for background processing is the sidekiq gem or more general ActiveJob that ships with Rails.
I'm building an API using Rails where requests come in and they need to be executed by a cluster of workers running on a different server (these workers call remote APIs and parse the data, etc...). I'm going to be using Sidekiq or Resque to handle the queueing/processing of that.
My issue is the client needs to wait while this is happening and the controller needs to return the response to the client once it's complete. How would I handle this in the controller? We're using a redis backend, so I was thinking something along the lines of subscribing to a pub/sub channel and waiting for the worker to publish a status message. The controller would wait for a set time period and then return a 'check back later' response to the client if it doesn't receive a message in time. What would be the best way to implement that, or is there a better solution?
Do not make your clients wait! There are a lot of issues if you make the controller block for a long running job:
Other programs may assume the request timed out (proxies, browsers, scripts, etc.)
It makes your API endpoints become a source for denial of service
It requires you to put more engineering work into web servers (since a rails process can't handle another web request while it's handling the blocking call)
Part of the reason of using Sidekiq or Resque is the avoid controllers that do heavily lifting during the http request.
Instead, background jobs should report their status to the database. Then web server should query and return to the client the latest status from the database.
If clients need more immediate feedback, you can:
make clients constantly poll
post request to the client (if the API consumer is another webserver)
use another protocol mechanism (eg - websockets).
I am making a rails app and I am wondering whether it is possible to setup an asynchronous/callback architecture in the controller layer. I am trying to do the following:
When a HTTP request is made to /my_app/foo, I want to asynchronously dish out two jobs - a naive ranking job and a complicated ranking job both of which rank 1000 posts - to several worker machines. I want to setup a callback method in the controller for each job which is called when the respective job is finished. If the complicated job does not return within X milliseconds, I want to return the output from the naive job. Otherwise, I want to return the output from the complicated job.
It is important to note that I want these jobs to performed in parallel. What is the best way to implement such a system in Rails? I am using Apache Phusion Passenger as my rails server if that helps.
Thanks.
Sounds like you should be using background jobs. In that case, when a request comes in, you would start / queue two jobs which would be picked up and processed by a worker, which acts independently of your Rails app.
Here a few links that could be of help:
https://www.ruby-toolbox.com/categories/Background_Jobs
http://railscasts.com/episodes/171-delayed-job
http://railscasts.com/episodes/243-beanstalkd-and-stalker
http://railscasts.com/episodes/271-resque
http://rubyrogues.com/queues-and-background-processing/
It's possible to issue several HTTP request asynchronously in Rails. However, it's impossible to make Rails event-driven.
In can send several HTTP request asynchronously with libraries such as Typhoeus. However, you might have concurrency issue if your timeout is too long.
Otherwise, you can try some event-driven web framework such as Cramp and Goliath. They are both based on EventMachine, so you can try em-http-request.
Try using rabbitmq where you can post a message on queue and expect the response in reply queue. The queue consumer can be even implemented in Scala for fastness. amqp gem would suffice what I am saying. Rails controller with amqp binding would be even more nice if possible(I am exploring that option having endpoints with amqp binding instead of http). That would solve enough no of problems