I want to know if it is possible that if my NGINX send a request to Unicorn and NGINX doesn't receive an answer, can NGINX send the request to Unicorn again?
Not normally, no. I can think of a workaround, but I'm sure that I would advise it. Using the Nginx upstream module you can define multiple upstream backend ends. Nginx has no way of knowing if the the different upstream servers listed are actually the same backend, nor does it care. The docs for the upstream module say: " If an error occurs during communication with a server, the request will be passed to the next server."
So if you arrange to have your app available at two different addresses (sockets, domain names, IPs or ports), then you can set Nginx to use the upstream module to try the same request twice.
If the the app was not responding properly in time the first time the request was made, there's a good chance it won't respond successfully on the follow-up request. Using the upstream module in the intended way is recommended instead: Use two unique app servers, who might be sharing state through a backend resource like a database.
Related
From the Electron renderer, I am accessing a local GraphQL endpoint served by a Django instance on my computer, which I'd like to do over HTTP, not HTTPS. But Electron's Chromium seems to intercept my fetch request and preemptively return a 307 redirect.
So if my fetch request is POST to http://local.myapp.com:3000/v1/graphql, then Chromium returns a 307 and forces a redirect to https://local.myapp.com:3000/v1/graphql, which fails because my server is listening on port 3000 and for my use case I can't do a local cert for local.myapp.com.
Theoretically the first insecure request should be hitting an nginx docker container listening on port 3000 without any SSL requirement. And nginx is proxying the request to a Hasura container. But I'm not even seeing the requests in the nginx access logs, so I'm pretty sure the request is being intercepted by Chromium.
I believe this StackOverflow comment summarizes well why this is happening: https://stackoverflow.com/a/34213531
Although I don't recall ever returning a Strict-Transport-Security header from my GraphQL endpoint or Django server.
I have tried the following code without success to turn off this Chromium behavior within my Electron app:
import { app, } from 'electron'
app.commandLine.appendSwitch('ignore-certificate-errors',)
app.commandLine.appendSwitch('allow-insecure-localhost', )
app.commandLine.appendSwitch('ignore-urlfetcher-cert-requests', )
app.commandLine.appendSwitch('allow-running-insecure-content', )
I have also tried setting the fetch options to include {redirect: 'manual'} and {redirect: 'error'}. I can prevent the redirect but that doesn't do me any good because I need to make a successful request to the endpoint to get my data.
I tried replacing the native fetch with electron-fetch (link) and cross-fetch (link) but there seems to be no change in behavior when I swap either of those out.
Edit: Also, making the request to my GraphQL outside of Electron with the exact same header and body info works fine (via Insomnia).
So I have a couple of questions:
Is there a way to programmatically view/clear the list of HSTS domains that is being used by Chromium within Electron?
Is there a better way to accomplish what I'm trying to do?
I think the issue might be from the server, most servers don't allow HTTP in any possible way, they'll drop the data transfer and redirect you to HTTPS and there's a clear reason why they would do that.
Imagine you have an app that connects through HTTPS to send your API in return for some data, if someone just changed the https:// to http:// that'd mean the data will be sent un-encrypted and no matter what you do with your API key, it'll be exposed, that's why the servers don't ever allow any HTTP request, they don't accept even a single bit of data.
I could think of two solutions.
Chromium is not the reason for the redirect, our Django instance might be configured as production or with HTTPS listeners.
Nginx might be the one who's doing the redirecting (having a little bit of SSL def on the configuration)
Last but not least, just generate a cert with OpenSSL (on host http://local.myapp.com:3000/) note: include the port and use that on your Django instance. You can trust the certificate so that it could work everywhere on your computer.
Suppose there is a web server that hosts arbitrary user-controlled content under some paths - public IPFS gateways are the example that got me thinking about this. Is it possible for that server to prevent pages that it serves from installing service workers on clients (and thus spoofing content for non-user-controlled paths)?
There's some helpful info in the service worker specification:
An HTTP request to fetch a service worker's script resource will
include the following header:
Service-Worker Indicates this request is a service worker's script
resource request.
Note: This header helps administrators log the requests and detect
threats.
If you'd like to make sure that your web server doesn't allow any service worker registrations, one approach would be to check for the Service-Worker header on incoming requests and have your web server return an appropriate HTTP error response (anything 4xx or 5xx would work—maybe 403 or 412?) whenever you detect that.
Is it possible to increase CloudFlare's time-out? If yes, how?
My code takes a while to execute and I wasn't planning on Ajaxifying it the coming days.
No, CloudFlare only offers that kind of customisation on Enterprise plans.
CloudFlare will time out if it fails to establish a HTTP handshake after 15 seconds.
CloudFlare will also wait 100 seconds for a HTTP response from your server before you will see a 524 timeout error.
Other than this there can be timeouts on your origin web server.
It sounds like you need Inter-Process Communication. HTTP should not be used a mechanism for performing blocking tasks without sending responses, these kind of activities should instead be abstracted away to a non-HTTP service on the server. By using RabbitMQ (or any other MQ) you can then pass messages from the HTTP element of your server over to the processing service on your webserver.
I was in communication with Cloudflare about the same issue, and also with the technical support of RabbitMQ.
RabbitMQ suggested using Web Stomp which relies on Web Sockets. However Cloudflare suggested...
Websockets would create a persistent connection through Cloudflare and
there's no timeout as such, but the best way of resolving this would
be just to process the request in the background and respond asynchronously, and serve a 'Loading...' page or similar, rather than having the user to wait for 100 seconds. That would also give a better user experience to the user as well
UPDATE:
For completeness, I will also record here that
I also asked CloudFlare about running the report via a subdomain and "grey-clouding" it and they replied as follows:
I will suggest to verify on why it takes more than 100 seconds for the
reports. Disabling Cloudflare on the sub-domain, allow attackers to
know about your origin IP and attackers will be attacking directly
bypassing Cloudflare.
FURTHER UPDATE
I finally solved this problem by running the report using a thread and using AJAX to "poll" whether the report had been created. See Bypassing CloudFlare's time-out of 100 seconds
Cloudflare doesn't trigger 504 errors on timeout
504 is a timeout triggered by your server - nothing to do with Cloudflare.
524 is a timeout triggered by Cloudflare.
See: https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#502504error
524 error? There is a workaround:
As #mjsa mentioned, Cloudflare only offers timeout settings to Enterprise clients, which is not an option for most people.
However, you can disable Cloudflare proxing for that specific (sub)domain by turning the orange cloud into grey:
Before:
After:
Note: it will disable extra functionalities for that specific (sub)domain, including IP masking and SSL certificates.
As Cloudflare state in their documentation:
If you regularly run HTTP requests that take over 100 seconds to
complete (for example large data exports), consider moving those
long-running processes to a subdomain that is not proxied by
Cloudflare. That subdomain would have the orange cloud icon toggled to
grey in the Cloudflare DNS Settings . Note that you cannot use a Page
Rule to circumvent Error 524.
I know that it cannot be treated like a solution but there is a 2 ways of avoiding this.
1) Since this timeout is often related to long time generating of something, this type of works can be done through crontab or if You have access to SSH you can run a PHP command directly to execute. In this case connection is not served through Cloudflare so it goes as long as your configuration allows it to run. Check it on Google how to run scripts from command line or how to determine them in crontab by using /usr/bin/php /direct/path/to/file.php
2) You can create subdomain that is not added to cloudlflare and move Your script there and run them directly through URL, Ajax call or whatever.
There is a good answer on Cloudflare community forums about this:
If you need to have scripts that run for longer than around 100 seconds without returning any data to the browser, you can’t run these through Cloudflare. There are a couple of options: Run the scripts via a grey-clouded subdomain or change the script so that it kicks off a long-running background process and quickly returns a status which the browser can poll until the background process has completed, at which point the full response can be returned. This is the way most people do this type of action as keeping HTTP connections open for a long time is unreliable and can be very taxing also.
This topic on Stackoverflow is high in SERPs so I decided to write down this answer for those who will find it usefull.
https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#502504error
Cloudflare 524 error results from a web page taking more than 100 seconds to completely respond.
This can be overridden to (up to) 600 seconds ... if you change to "Enterprise" Cloudflare account. The cost of Enterprise is roughtly $40k per year (annual contract required).
If you are getting your results with curl, you could use the resolve option to directly access your IP, not using the Cloudflare proxy IP:
For example:
curl --max-time 120 -s -k --resolve lifeboat.com:443:127.0.0.1 -L https://lifeboat.com/blog/feed
The simplest way to do this is to increase your proxy waiting timeout.
If you are using Nginx for instance you can simply add this line in your /etc/nginx/sites-availables/your_domain:
location / {
...
proxy_read_timeout 600s; # this increases it by 10mins; feel free to change as you see fit with your needs.
...
}
If the issue persists, make sure you use let's encrypt to secure your server alongside Nginx and then disable the orange cloud on that specific subdomain on Cloudflare.
Here are some resources you can check to help do that
installing-nginx-on-ubuntu-server
secure-nginx-with-let's-encrypt
We have a Rails app that we run on Unicorn (2 workers) and nginx. We want to integrate a 3rd party API where processing of a single request takes between 1 and 20 seconds. If we simply create a new controller that proxies to that service the entire app suffers, because it takes only 2 people to make a request to that service via our API and for 20 seconds the rest of the users can't access the rest of our app.
We're thinking about 2 solutions.
Create a separate node.js server that will do all of the requests to the 3rd party API. We would only use Rails for authentication/authorization in this case, and we would redirect the requests to node via nginx using X-Accel-Redirect header (as described here http://blog.bitbucket.org/2012/08/24/segregating-services/)
Replace Unicorn with Thin or Rainbow! and keep proxying in our Rails app, which could then, presumably, allow us to handle many more concurrent connections.
Which solution might we be better off? Or is there something else we could do.
I personally feel that nodes even-loop is better suited for the job here, because in option 2 we would still be blocking many threads and waiting for HTTP requests to finish and in option 1, we could be doing more requests while waiting for the slow ones to finish.
Thanks!
We've been using the X-Accel-Redirect solution in production for a while now and it's working great.
In nginx config under server, we have entries for external services (written in node.js in our case), e.g.
server {
...
location ^~ /some-service {
internal;
rewrite ^/some-service/(.*)$ /$1 break;
proxy_pass http://location-of-some-service:5000;
}
}
In rails we authenticate and authorize the requests and when we want to pass it to some other service, in the controller we do something like
headers['X-Accel-Redirect'] = '/some-service'
render :nothing => true
Now, rails is done with processing the request and hands it back to nginx. Nginx sees the x-accel-redirect header and replays the request to the new url - /some-service which we configured to proxy to our node.js service. Unicorn and rails can now process new requests even if node.js+nginx is still processing that original request.
This way we're using Rails as our main entry point and gatekeeper of our application - that's where authentication and authorization happens. But we were able to move a lot of functionality into these smaller, standalone node.js services when that's more appropriate.
You can use EventMachine in your existing Rails app which would mean much less re-writing. Instead of making a net/http request to the API, you would make a EM::HttpRequest request to the API and add a callback. This is similar to node.js option but does not require a special server IMO.
I'm sure many will consider this a dumb question but I can't find a straight answer anywhere on the web:
Can Unicorn be configured to handle HTTPS requests directly, without reverse-proxying the request through another web server such as Apache or Nginx? (And if so, how?)
Unicorn is optimized to handle fast connections and is not meant to be used for direct client communication. If you consider using https I would assume that your are transferring data through an potentially uncontrollable line. It's better to use another web server for terminating the ssl connection.