I'd love to divide end-points to two types.
One is for normal request, and the other is only to handle webhook requests.
Is this possible to set up another process for webhook handling, and prevent the webhook from blocking the normal request?
I'll go for Unicorn for the application server, and Nginx for web server.
Related
Suppose there is a web server that hosts arbitrary user-controlled content under some paths - public IPFS gateways are the example that got me thinking about this. Is it possible for that server to prevent pages that it serves from installing service workers on clients (and thus spoofing content for non-user-controlled paths)?
There's some helpful info in the service worker specification:
An HTTP request to fetch a service worker's script resource will
include the following header:
Service-Worker Indicates this request is a service worker's script
resource request.
Note: This header helps administrators log the requests and detect
threats.
If you'd like to make sure that your web server doesn't allow any service worker registrations, one approach would be to check for the Service-Worker header on incoming requests and have your web server return an appropriate HTTP error response (anything 4xx or 5xx would workâmaybe 403 or 412?) whenever you detect that.
I have a Rails API which can handle requests from the clients. Clients use that API to perform analysis of their data. Client POSTs the data to API, API checks if that data have been analysed before. If so API just respond with analysis result. If the data haven't been analyzed before API:
Tells client that analysis started.
Establishes the connection with analyzing microservice.
Performs asynchronous (or deferred or i don't know) request to the analyzing microservice and waiting for response. The analysis takes much time so neither the API nor the microservice should be blocked while doing it.
When the response from analyzing microservice is returned API hands it to the client.
The main issue for me is to set up things such way that client could receive somehow the message "Your data had been sent to analysis" right after he performed the request. And then when analysis will be done client could receive its result.
The question is what approach I have to use in that case? Async responses, deferred responses, something else? And what known solutions could help me with that? Any gems?
I'm new to that stuff so I'm really sorry if I ask dumb questions.
If using HTTP you can only have one response to every request. To send multiple responses, i.e. "work in progress", then later the "results", you would need to use a different protocol, e.g. web sockets.
Since HTTP is so very common I'd stick with that in combination with background jobs. There are a couple of options which spring to mind.
Polling: The API kicks off a background jobs (to call the microservice) and responds to the client with a URL which the client can ping periodically for the result. The URL would respond with some kind of "work in progress" status until the result is actually ready). The URL would need to include some kind of id so the API can lookup the background job.
The API would potentially have two URLS; /api/jobs/new and /api/jobs/<ID>. They would, in Rails, map to a controller new and show action.
Webhooks: Have the client include a URL of its own in the request. Once the result is available have the background job hit the given URL with the result.
Either way, if using HTTP, you will not be able to handle the whole thing within a request/response, you will have to use some kind of background processing (so request to the microservice happens in a different process). You could look at Sidekiq, for example.
Here is an example for polling:
URL: example.com/api/jobs/new
web app receives client request
generates a unique id for the request, SecureRandom.uuid.
starts a background job (Sidekiq) passing in the uuid and any other parameters needed
respond with URL such as example.com/api/jobs/
--
background job
sends request to microservice API and waits for response
saves result to database with uuid
--
URL: example.com/api/jobs/UUID
look in database for UUID, if not found respond that job is "in progress". If found return result found in database.
Depending on what kind of API you use. I assume your clients interact via HTTP.
If you want to build an asynchronous API over HTTP the first thing that you should do: accept the request, create a job, handle it in the background and immediately return.
For the client to get the response you have to 2 options:
Implement a status endpoint where clients can periodically poll the status of the job
Implement a callback via webhooks. So the client has to provide a URL which you then call after you're done.
A good start for background processing is the sidekiq gem or more general ActiveJob that ships with Rails.
I have two Rails apps that talk to one another. A few times a day, requests from app A show up in duplicate (or triplicate/quadruplicate) at app B. All outbound and inbound requests are logged. The logs show that app A is sending one outbound request and that app B receives that request twice or more during the same second.
App B sits behind Apache and an Amazon Elastic Load Balancer.
I am not sure where to look or even what questions to ask to hone in on what might be causing this issue. If you need more data, I would be happy to provide it.
The retries are likely coming out of the Amazon Elastic Load Balancer or some network component (like a router, for example). I've seen similar behavior when using other load balancers (like Citrix NetScaler) as well.
Basically, the request gets an idle timeout at some level in the request chain. If that timeout doesn't send a proper HTTP 5xx status back to the client (for example it could just silently close the connection) then any components between the source of the timeout and the client can potentially decide to retry the request depending on how they are configured.
Tracking down which components cause the retries can be very challenging. My recommendation is to make sure your Rails applications always respond quickly to each other. If the requests can't complete quickly, consider perhaps a background/polling solution or a non-HTTP communication method (WebSockets for example).
I'm building an API using Rails where requests come in and they need to be executed by a cluster of workers running on a different server (these workers call remote APIs and parse the data, etc...). I'm going to be using Sidekiq or Resque to handle the queueing/processing of that.
My issue is the client needs to wait while this is happening and the controller needs to return the response to the client once it's complete. How would I handle this in the controller? We're using a redis backend, so I was thinking something along the lines of subscribing to a pub/sub channel and waiting for the worker to publish a status message. The controller would wait for a set time period and then return a 'check back later' response to the client if it doesn't receive a message in time. What would be the best way to implement that, or is there a better solution?
Do not make your clients wait! There are a lot of issues if you make the controller block for a long running job:
Other programs may assume the request timed out (proxies, browsers, scripts, etc.)
It makes your API endpoints become a source for denial of service
It requires you to put more engineering work into web servers (since a rails process can't handle another web request while it's handling the blocking call)
Part of the reason of using Sidekiq or Resque is the avoid controllers that do heavily lifting during the http request.
Instead, background jobs should report their status to the database. Then web server should query and return to the client the latest status from the database.
If clients need more immediate feedback, you can:
make clients constantly poll
post request to the client (if the API consumer is another webserver)
use another protocol mechanism (eg - websockets).
I am implementing an json api using rails. I wish to make requests to another web service using delayed job to prevent it from blocking my rails app. So far so good. So i have a function defined in my model which does a http POST to this other web service.
However, the other web service is is an asynchronous api with callbacks. Hence I want to also receive callbacks from this api within my delayed job.
Is this possible? Can I have a http listener in my delayed job whose port number I can control or know within my code?