I have two Rails apps that talk to one another. A few times a day, requests from app A show up in duplicate (or triplicate/quadruplicate) at app B. All outbound and inbound requests are logged. The logs show that app A is sending one outbound request and that app B receives that request twice or more during the same second.
App B sits behind Apache and an Amazon Elastic Load Balancer.
I am not sure where to look or even what questions to ask to hone in on what might be causing this issue. If you need more data, I would be happy to provide it.
The retries are likely coming out of the Amazon Elastic Load Balancer or some network component (like a router, for example). I've seen similar behavior when using other load balancers (like Citrix NetScaler) as well.
Basically, the request gets an idle timeout at some level in the request chain. If that timeout doesn't send a proper HTTP 5xx status back to the client (for example it could just silently close the connection) then any components between the source of the timeout and the client can potentially decide to retry the request depending on how they are configured.
Tracking down which components cause the retries can be very challenging. My recommendation is to make sure your Rails applications always respond quickly to each other. If the requests can't complete quickly, consider perhaps a background/polling solution or a non-HTTP communication method (WebSockets for example).
Related
In our application we observe multiple ( two ) these same requests send from mobile application to server in milliseconds apart.
As we discuss the problem with dev team, they said they don't send two requests from an application perspective, but on the server-side, we see exactly these same two requests.
Does anybody know if iOS has this type of functionality to keep resending this same request in case of a lost connection or any other case? ( This is milliseconds that server doesn't respond yet )
The application should send only one request, wait for response success/failure, and then resend as needed. So far as we know, there is no logic in the application itself that will trigger sending two requests from the app to the server in milliseconds apart.
Thank you for any suggestions.
It's hard to tell without looking at the code or knowing your network infrastructure.
What I'd suggest to do first is to run the app through a debugging proxy server like Charles, Proxyman or mitmproxy. If it shows multiple requests, most likely the app is to blame, I'd bet on a concurrency bug.
If the debugging proxy shows just one request but your server observes two, you'll have to check your network infrastructure, it might be that some load balancer or reverse proxy is configured incorrectly.
We have a Vaadin 14 (Flow) application which is fronted by an Apache reverse proxy that integrates with Gluu for authentication using OpenID (mod_auth_openidc).
This is generally working fine, except when users leave their browser open with the application idle for a long time, until the max session time of the OpenID session is reached. The problem is, at that point the Vaadin client keeps trying to send heartbeat requests. This, in combination with this mod_auth_openidc issue, results in state cookies piling up and reaching a limit so that the user has to close her browser before being able to re-login.
I've tried various things (unsuccessfully) in order to get the server to instruct the browser to visit a logout URL when a heartbeat request is received after session timeout (in combination with vaadin.closeIdleSessions=true), but even if it worked it wouldn't be a solution for other browser tabs that may also be open at that time and sending heartbeat requests.
What we really want is to limit the number of times the Vaadin client retries to send the heartbeat requests (say max 3 times) and then just stop sending requests (maybe display a message to re-login).
Is this possible in any way? The current workaround is to disable the heartbeats completely, but this doesn't seem ideal (Vaadin won't detect idle UIs).
The UI instance has a ReconnectDialogConfiguration which includes reconnectAttempts property to control how many times to re-try requests (including heartbeat requests). Default seems to be 10000.
In Vaadin 14 (LTS) this can be set using PageConfigurator.
In Vaadin 18 (latest release) this is done using AppShellConfigurator
Disabling the heartbeats is the way to go. If you need something more nuanced than that, you'll need to make a change (maybe add a configuration option) in how Heartbeats work in Vaadin. Creating a ticket in GitHub could be a good place to start.
I'm using Electron, which is based on Chromium, to create an offline desktop application.
The application uses a remote site, and we are using a service worker to offline parts of the site. Everything is working great, except for a certain situation that I call the "airplane wifi situation".
Using Charles, I have restricted download bandwidth to 100bytes/s. The connection is sent through webview.loadURL which eventually calls LoadURLWithParams in Chromium. The problem is that it does not fail and then activate the service worker, like no connection at all would. Once the request is sent, it waits forever for the response.
My question is, how do I timeout the request after a certain amount of time and load everything from the service worker as if the user was truly offline?
An alternative to writing this yourself is to use the sw-toolbox library, which provides routing and runtime caching strategies for service workers, along with some built in options for helping with these sorts of advanced use cases. In particular, you'd want to use the networkTimeoutSeconds parameter to configure the amount of time to wait for a response from the network before you fall back to a previously cached response.
You can use it like the following:
toolbox.router.get(
new RegExp('my-api\\.com'),
toolbox.networkFirst, {
networkTimeoutSeconds: 10
}
);
That would configure a route that matched GET requests with URLs containing my-api.com, and applied a network-first strategy that will automatically fall back to the previously cached response after 10 seconds.
I have deployed a Rails app at Engineyard in production and staging environment. I am curious to know if every HTTP request for my app initializes new instance of my Rails App or not?
Rails is stateless, which means each request to a Rails application has its own environment and variables that are unique to that request. So, a qualified "yes", each request starts a new instance[1] of your app; you can't determine what happened in previous requests, or other requests happening at the same time. But, bear in mind the app will be served from a fixed set of workers.
With Rails on EY, you will be running something like thin or unicorn as the web server. This will have a defined number of workers, let's say 5. Each worker can handle only one request at a time, because that's how rails works. So if your requests take 200ms each, that means you can handle approximately 5 requests per second, for each worker. If one request takes a long time (a few seconds), that worker is not available to take any other requests. Workers are typically not created and removed on Engineyard; they are set up and run continuously until you re-deploy, though for something like Heroku, your app may not have any workers (dynos) and if there are no requests coming in it will have to spin up.
[1] I'm defining instance, as in, a new instance of the application class. Each model and class will be re-instantiated and the #request and #session built from scratch.
According to what I have understood. No, It will definitely not initialize new instance for every request. Then again two questions might arise.
How can multiple user simultaneously login and access my system without interference?
Even though one user takes up too much processing time, how is another user able to access other features.
Answer to the first question is that HTTP is stateless, everything is stored in session, which is in cookie, which is in client machine and not in server. So when you send a HTTP request for a logged in user, browser actually sends the HTTP request with the required credentials/user information from clients cookies to the server without the user knowing it. Multiple requests are just queued and served accordingly. Since our server are very very fast, I feel its just processing instantly.
For the second query, your might might be concurrency. The server you are using (nginx, passenger) has the capacity to serve multiple request at same time. Even if our server might be busy for a particular user(Lets say for video processing), it might serve another request through another thread so that multiple user can simultaneously access our system.
I'm building an API using Rails where requests come in and they need to be executed by a cluster of workers running on a different server (these workers call remote APIs and parse the data, etc...). I'm going to be using Sidekiq or Resque to handle the queueing/processing of that.
My issue is the client needs to wait while this is happening and the controller needs to return the response to the client once it's complete. How would I handle this in the controller? We're using a redis backend, so I was thinking something along the lines of subscribing to a pub/sub channel and waiting for the worker to publish a status message. The controller would wait for a set time period and then return a 'check back later' response to the client if it doesn't receive a message in time. What would be the best way to implement that, or is there a better solution?
Do not make your clients wait! There are a lot of issues if you make the controller block for a long running job:
Other programs may assume the request timed out (proxies, browsers, scripts, etc.)
It makes your API endpoints become a source for denial of service
It requires you to put more engineering work into web servers (since a rails process can't handle another web request while it's handling the blocking call)
Part of the reason of using Sidekiq or Resque is the avoid controllers that do heavily lifting during the http request.
Instead, background jobs should report their status to the database. Then web server should query and return to the client the latest status from the database.
If clients need more immediate feedback, you can:
make clients constantly poll
post request to the client (if the API consumer is another webserver)
use another protocol mechanism (eg - websockets).