Rails app running on puma and nginx keeps dying every few hours with Bad Gateway - ruby-on-rails

I have a rails app that I just deployed to Digital Ocean and it's running on Puma and Nginx.
Eventually all it returns is a bad gateway and this is what is in the error.log
2014/09/09 22:23:06 [error] 5729#0: *3059 connect() to unix:///var/www/mysite/mysite_app.sock failed (111: Connection refused) while connecting to upstream, client: 67.5.19.192, server: mysite.com, request: "GET / HTTP/1.1", upstream: "http://unix:///var/www/mysite/mysite_app.sock:/", host: "mysite.com"
To fix it, I just restart puma and it seems to work.
How can I debug this to figure out why it keeps dying?
Here's my nginx config:
upstream mysite {
server unix:///var/www/mysite/mysite_app.sock;
}
server {
listen 80;
server_name mysite.com;
root /var/www/mysite/current/public;
client_max_body_size 20M;
location / {
proxy_pass http://mysite; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~* ^/assets/ {
# Per RFC2616 - 1 year maximum expiry
expires 1y;
add_header Cache-Control public;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
}
EDIT
Could this be caused from running out of memory?
Here's my current state of memory, but as I keep running this command every so often, the amount of free memory goes down and once i restart puma it jumps back up to like 150.
$ free -m
total used free shared buffers cached
Mem: 490 440 50 0 17 84
-/+ buffers/cache: 338 151
Swap: 0 0 0

It seems this is actually an issue with ruby 2.1 (specifically i'm using 2.1.2) and its garbage collection.
A google search like this seems to have lots of various threads on it http://bit.ly/1s2vBC0
Here's a ruby bug ticket on the issue: https://bugs.ruby-lang.org/issues/9607

Lack of memory can be issue, but you better look into puma and rails logs, not nginx only. In the application folder:
tail -f log/puma*
tail -f log/production.log

I've had similar issues but I note that 2.1.3 has now been released and specifically discusses memory issues:
https://www.ruby-lang.org/en/news/2014/09/19/ruby-2-1-3-is-released/
I'm going to try it now!

Related

Deluge client not passing through nginx grpc

I have a Deluge client (in a docker container - that's likely irrelevant).
I want to be able to connect to the daemon from the outside world while having it behind a reverse proxy.
I don't necessarily need TLS, but I suspect http2 may require it.
What works:
connecting locally on the network to the Deluge RPC with a Deluge desktop, Android and WebUI client works well.
sending requests to the nginx server is OK (I can see logs as I hit nginx)
All the networking around (firewalls, port forwardings, DNS are fine)
What doesn't work:
Deluge client can't connect to the http server
nginx config:
server {
server_name deluge.example.com;
listen 58850;
location / {
proxy_pass grpc://localhost:58846;
}
ssl_certificate /etc/ssl/nginx/example.com.pem;
ssl_certificate_key /etc/ssl/nginx/example.com.key;
proxy_request_buffering off;
gzip off;
charset utf-8;
error_log /var/log/nginx/nginx_deluge.log debug;
}
Major edit:
As it turns out, I believed the JSON RPC and gRPC are more similar than just the "RPC" in the name. Hence my "original" issue "nginx deluge rpc doesn't work", is no longer relevant.
Unfortunately, the "same" issue still prevails. I still can't connect to the proxy even when using a regular HTTP proxy while I can make HTTP requests locally.
I will surely post an update or even an answer should I figure it out in the next days...
When I try to connect with the Deluge client, I get this error message in the log file:--
2022/06/14 16:59:55 [info] 1332115#1332115: *7 client sent invalid method while reading client request line, client: <REDACTED IPv4>, server: deluge.example.com, request: " Fu�Uq���U����a(wU=��_`. a��¹�(���O����f�"
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http finalize request: 400, "?" a:1, c:1
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 event timer del: 17: 243303738
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http special response: 400, "?"
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 http set discard body
2022/06/14 16:59:55 [debug] 1332115#1332115: *7 HTTP/1.1 400 Bad Request
Server: nginx/1.22.0
Date: Tue, 14 Jun 2022 16:59:55 GMT
Content-Type: text/html
Content-Length: 157
Connection: close
When I change the line listen 58850; to listen 58850 http2;, as I probably should, I get the following error: (log verbosity set to "debug")
2022/06/14 15:04:00 [info] 1007882#1007882: *3654 client sent invalid method while reading
client request line, client: <REDACTED IPv4>,
server: deluge.example.com, request: "x�;x��;%157?O/3/-�#�D��"
The gibberish there is seemingly identical when trying to connect from a different network from a different device. It was Dx�;x��;%157?O/3/-�#�E�, (there is a D as first character now) but all other attempts are again without the leading D.
or this error: (log verbosity set to "info")
2022/06/14 17:09:13 [info] 1348282#1348282: *14 invalid connection preface while processing HTTP/2 connection, client: <REDACTED IPv4>, server: 0.0.0.0:58850
I tried decoding the gibberish between various encodings, in hoping it would be just bad encoding of a better error message or a lead to a solution.
I looked through the first two pages of Google hoping the error messages were pointing me to a solution someone else has had to my problem.
environment:
Docker version 20.10.17, build 100c70180f
nginx version: nginx/1.22.0
deluged 2.0.5
libtorrent: 2.0.6.0

Secure websocket connection fails on iOS due to BoringSSL certificate verification failure

I'm unable to connect to a secure websocket connection due to the following error on iOS (12 & 13).
It seems there is a failure validating the certificate for the end-point but I can't find a way to determine the reason. I can connect to the websocket end-point from a browser client. I also verified the certificate using https://www.ssllabs.com/ssltest/ and don't see any issues. The SSL certificate is issued from Let's Encrypt.
Below is the console output I could capture from the device and console app on my mac laptop:
XCODE DEBUG CONSOLE
2020-04-29 16:18:03.501170-0700 [BoringSSL] boringssl_context_handle_fatal_alert(1873) [C12.1:1][0x1151122e0] write alert, level: fatal, description: certificate unknown
2020-04-29 16:18:03.501366-0700 [BoringSSL] boringssl_context_error_print(1863) boringssl ctx 0x282eb41b0: 4450062232:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-283.102.1/ssl/handshake.cc:369:
2020-04-29 16:18:03.510648-0700 [BoringSSL] boringssl_session_handshake_incomplete(164) [C12.1:1][0x1151122e0] SSL library error
2020-04-29 16:18:03.510740-0700 [BoringSSL] boringssl_session_handshake_error_print(111) [C12.1:1][0x1151122e0] 4450062232:error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/boringssl/boringssl-283.102.1/ssl/handshake.cc:369:
2020-04-29 16:18:03.510837-0700 [BoringSSL] nw_protocol_boringssl_handshake_negotiate_proceed(726) [C12.1:1][0x1151122e0] handshake failed at state 12288
2020-04-29 16:18:03.532541-0700 [strings] ERROR: Network.NWError not found in table Error of bundle CFBundle 0x109308b40 </System/Library/Frameworks/CoreFoundation.framework> (framework, loaded)
🛑 error: Optional("The operation couldn’t be completed. (NETWORK.NWERROR error 2.)")
MAC CONSOLE APP
default 16:18:03.712414-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: waiting for data to read [2]
default 16:18:03.712477-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: waiting for data to read [2]
default 16:18:03.712583-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: waiting for data to read [2]
default 16:18:03.712646-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: waiting for data to read [2]
default 16:18:03.712707-0700 boringssl_context_message_handler(2258) [C12.1:1][0x1151122e0] Reading SSL3_RT_HANDSHAKE 122 bytes
default 16:18:03.713628-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_hello_retry_request
default 16:18:03.713687-0700 boringssl_context_add_handshake_message_pending(578) [C12.1:1][0x1151122e0] Adding message(2)
default 16:18:03.713747-0700 boringssl_context_message_handler(2258) [C12.1:1][0x1151122e0] Writing SSL3_RT_CHANGE_CIPHER_SPEC 1 bytes
default 16:18:03.713807-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_server_hello
default 16:18:03.713865-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_encrypted_extensions
default 16:18:03.714153-0700 boringssl_context_message_handler(2258) [C12.1:1][0x1151122e0] Reading SSL3_RT_HANDSHAKE 10 bytes
default 16:18:03.714219-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_certificate_request
default 16:18:03.714277-0700 boringssl_context_message_handler(2258) [C12.1:1][0x1151122e0] Reading SSL3_RT_HANDSHAKE 3105 bytes
default 16:18:03.714335-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_server_certificate
default 16:18:03.714395-0700 boringssl_context_info_handler(1983) [C12.1:1][0x1151122e0] Client handshake state: TLS 1.3 client read_server_certificate_verify
default 16:18:03.714452-0700 boringssl_context_message_handler(2258) [C12.1:1][0x1151122e0] Reading SSL3_RT_HANDSHAKE 264 bytes
default 16:18:03.714628-0700 boringssl_context_copy_peer_sct_list(1003) [C12.1:1][0x1151122e0] SSL_get0_signed_cert_timestamp_list returned no SCT extension data
default 16:18:03.714914-0700 boringssl_helper_create_sec_trust_with_certificates(607) [C12.1:1][0x1151122e0] SecTrustCreateWithCertificates result: 0
default 16:18:03.714973-0700 boringssl_helper_create_sec_trust_with_certificates(612) [C12.1:1][0x1151122e0] SecTrustSetOCSPResponse result: 0
default 16:18:03.715033-0700 boringssl_helper_create_sec_trust_with_certificates(621) [C12.1:1][0x1151122e0] No TLS-provided SCTs
default 16:18:03.715167-0700 boringssl_context_certificate_verify_callback(2071) [C12.1:1][0x1151122e0] Asyncing for verify block
default 16:18:03.715225-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: certificate evaluation result pending [16]
default 16:18:03.715418-0700 boringssl_context_certificate_verify_callback(2040) [C12.1:1][0x1151122e0] Verification already in progress.
default 16:18:03.715481-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: certificate evaluation result pending [16]
default 16:18:03.716084-0700 boringssl_context_certificate_verify_callback(2040) [C12.1:1][0x1151122e0] Verification already in progress.
default 16:18:03.716145-0700 boringssl_session_handshake_incomplete(170) [C12.1:1][0x1151122e0] Handshake incomplete: certificate evaluation result pending [16]
default 16:19:16.180121-0700 boringssl_context_message_handler(2258) [C6.1:2][0x11321ccb0] Writing SSL3_RT_ALERT 2 bytes
default 16:19:16.180202-0700 boringssl_context_handle_warning_alert(1893) [C6.1:2][0x11321ccb0] write alert, level: warning, description: close notify
default 16:19:16.180285-0700 boringssl_session_disconnect(504) [C6.1:2][0x11321ccb0] SSL_shutdown 0
default 16:19:16.181104-0700 nw_protocol_boringssl_remove_input_handler(1012) [C6.1:2][0x11321ccb0] nw_protocol_boringssl_remove_input_handler forced true
default 16:19:16.181169-0700 nw_protocol_boringssl_remove_input_handler(1030) [C6.1:2][0x11321ccb0] Transferring nw_protocol_boringssl_t handle back into ARC for autorelease
default 16:19:33.510159-0700 boringssl_context_message_handler(2258) [C8.1:2][0x11328fd50] Writing SSL3_RT_ALERT 2 bytes
default 16:19:33.510247-0700 boringssl_context_handle_warning_alert(1893) [C8.1:2][0x11328fd50] write alert, level: warning, description: close notify
default 16:19:33.510309-0700 boringssl_session_disconnect(504) [C8.1:2][0x11328fd50] SSL_shutdown 0
default 16:19:33.510922-0700 nw_protocol_boringssl_remove_input_handler(1012) [C8.1:2][0x11328fd50] nw_protocol_boringssl_remove_input_handler forced true
default 16:19:33.511105-0700 nw_protocol_boringssl_remove_input_handler(1030) [C8.1:2][0x11328fd50] Transferring nw_protocol_boringssl_t handle back into ARC for autorelease
The NGINX server block for the end-point is:
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name me.example.com;
ssl_certificate /etc/letsencrypt/live/me.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/me.example.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/me.example.com/chain.pem;
location /ws {
proxy_pass http://upstreamserver;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
}
}
Any thoughts on why this is an issue on iOS? I also posted this in Apple's Dev Forums
I just wrote a class to help with sockets and the new apple TLS restrictions (see my readme). You are most likely misconfiguring your certificates. Along with my helper class that handles the socket, I tried to also layout how to create certificates that will work as of the new iOS TLS restrictions.
https://github.com/eamonwhiter73/IOSObjCWebSockets
Issue was a mis-configuration in the SSL certificate issued by LetsEncrypt and the NGINX setup. Due to Apple's Certificate Transparency Policy, there is now stricter certificate verification on iOS. Got a good answer on the Apple dev forums that helped trace the problem to the following:
Certificate issued by LetsEncrypt was missing embedded Signed Certificate Timestamps (SCT). I had to re-create my certificate with "OCSP Must Staple" support:
certbot --nginx --hsts --staple-ocsp --must-staple -d domain.example.com
To verify embedded SCT's are created for the certificate you can run openssl x509 -in cert.pem -text and the output should contain something similar to:
Certificate:
Data:
X509v3 extensions:
Authority Information Access:
OCSP - URI:http://ocsp.int-x3.letsencrypt.org
CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
TLS Feature:
status_request
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 5E:A7:73:F9:DF:56:C0:E7:B5:36:48:7D:D0:49:E0:32:
7A:91:9A:0C:84:A1:12:12:84:18:75:96:81:71:45:58
Timestamp : May 1 21:02:14.817 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:2F:2C:22:85:50:DD:FD:DA:62:E9:60:BA:
95:6C:49:03:1E:9E:F9:6C:9F:AA:A0:17:65:7F:D7:D3:
A4:E7:CC:02:02:21:00:D4:2F:55:CF:F6:57:AC:BF:3E:
E5:8B:F5:A2:00:47:2D:C4:5E:A4:10:EE:D7:D6:B4:FF:
9E:21:1D:CC:6A:89:53
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 07:B7:5C:1B:E5:7D:68:FF:F1:B0:C6:1D:23:15:C7:BA:
E6:57:7C:57:94:B7:6A:EE:BC:61:3A:1A:69:D3:A2:1C
Timestamp : May 1 21:02:14.843 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:44:02:20:17:63:1D:8E:76:CA:E0:A2:5C:42:92:7C:
BC:06:60:C7:9B:46:BB:59:63:8F:E1:8A:BE:52:CB:15:
FD:C4:DE:09:02:20:28:EF:48:E1:4B:BD:9D:05:29:52:
FC:D9:5A:8B:82:08:9D:1A:A0:58:F0:33:FB:05:5E:E7:
56:A0:AE:64:84:C7
NGINX server block needed proper OCSP configuration, remove HTTP/2 support, and use a cert chain that has the intermediate certificates and embedded SCT's
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream ws-signal {
server localhost:8080;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name domain.example.com;
ssl_certificate /etc/letsencrypt/live/domain.example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/domain.example.com/privkey.pem; # managed by Certbot
# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/domain.example.com/fullchain.pem;
resolver 8.8.8.8 8.8.4.4;
location /ws {
proxy_pass http://ws-signal;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
}
}
I also had to upgrade Starscream to v4.0.3. This latest version uses URLSessionWebSocketTask for iOS 13+ and for iOS 12 seems to be overriding the verification to return true if certificate pinning is disabled. Still need to try with cert pinning enabled to see what happens.

Is it possible to use a self signed cert with a EC2 instance that requires a client cert from API Gateway

Here's my situation:
I'm using Elastic Beanstalk to spin up a single EC2 instance without an ELB. I want to have the instance only accessible through the API Gateway. So, I went the route of using client-side certificates for authentication, like what's described here.
My EC2 instance has Nginx serving a Rails application. I generated a self-signed certificate on my machine and configured Nginx to use that to serve stuff over https.
Everything seems fine, but when I try to invoke my proxy endpoint from the API Gateway console, I get a 500 error like below:
...
Thu Sep 14 02:27:05 UTC 2017 : Endpoint request URI: https://xxxxxxxxx.xxxxxxxxx.us-east-1.elasticbeanstalk.com/health
Thu Sep 14 02:27:05 UTC 2017 : Endpoint request headers: {x-amzn-apigateway-api-id=xxxxxxxxx, User-Agent=AmazonAPIGateway_xxxxxxxx, Accept-Encoding=identity}
Thu Sep 14 02:27:05 UTC 2017 : Endpoint request body after transformations:
Thu Sep 14 02:27:05 UTC 2017 : Sending request to https://xxxxxxxxx.xxxxxxxx.us-east-1.elasticbeanstalk.com/health
Thu Sep 14 02:27:05 UTC 2017 : Execution failed due to configuration error: General SSLEngine problem
Thu Sep 14 02:27:05 UTC 2017 : Method completed with status: 500
I'm thinking that it has something to do with the fact that I'm using a self-signed certificate on the backend. But do I really have to purchase a legitimate certificate in order to complete my setup? Are there any other solutions that would allow me to only accept requests to my EC2 instance only through the API Gateway?
I looked at the Lambda method that is described here, but I didn't want to add any more complexity or latency to the requests.
Here's my Nginx configuration for completeness:
server {
listen 443;
server_name localhost;
ssl on;
ssl_certificate /etc/pki/tls/certs/server.crt;
ssl_certificate_key /etc/pki/tls/certs/server.key;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_prefer_server_ciphers on;
ssl_client_certificate /etc/pki/tls/certs/api_gateway.cer;
ssl_verify_client on;
if ($ssl_protocol = "") {
return 444;
}
}
see my answer here AWS API Gateway - Use Client-Side SSL Certificates. Sot sure what incompatibility is with NGINX - i managed to create PoC and validate Authenticate with Client-SSL behavior
It appears at the time of this writing that API Gateway has a known incompatibility with NGINX around Client Certificates.

nginx redirects POST requests to GET request

I have Rails 4.1 application with runs on puma web server. I use nginx as a proxy server. Several days ago everything worked very well. I updated my application, and suddenly some POST requests started to redirected to same url but as GET request. I've tried rollback to previous working versions, no success.
I found very interesting behaviour. I tested my API with curl.
If I did POST request to the url
http://myapp.com/tasks/easy_task/calculate/ it redirects to same url
but as GET request.
Then I did POSTrequest to http://myapp.com/, returned 404
Then I did POSTrequest to http://myapp.com/tasks, returned 404
Then I did POSTrequest to http://myapp.com/tasks/easy_task, returned 404
Then I did POSTrequest to http://myapp.com/tasks/easy_task/calculate, returned 200. YAY!
Same thing happened when I used chrome's app Postman. First it redirected, but after previous steps it works well.
I use this app in my other application. I use RestClient to make http requests. When I try to make POST request it raises an exception RestClient::MovedPermanently (301 Moved Permanently).
I reinstalled nginx to 1.7.3.
Restarted server (virtual machine)
re deployed my app, deployed previous versions
no success :(
I found similar questions on stackoverflow, but non of them gave me clue to fix this issue. I hope you can help me to solve this problem. Thanks in advance!
Similar questions:
- POST request turns into GET request
- POST request mysteriously turn into GET request
nginx config:
$ cat /etc/nginx/sites-enabled/myapp.com.conf
# The file generated by Chef for mycompany
upstream myapp_mycompany_com {
server unix:/tmp/myapp.com-puma.sock;
}
server {
server_name myapp.com;
listen 80;
access_log /var/log/nginx/myapp.com-access.log;
error_log /var/log/nginx/myapp.com-error.log;
root /home/projects/mycompany/myapp.com/current/public;
gzip on;
gzip_types text/plain text/xml application/xml application/xml+rss
text/css text/javascript application/javascript application/json;
error_page 551 =503 #maintenance;
location #maintenance {
rewrite ^(.*)$ /system/maintenance.html break;
}
set $maintenance 0;
if (-f $document_root/system/maintenance.html) {
set $maintenance 1;
}
if ($request_uri = /favicon.ico) {
# Browsers will try to get favicon if it's not returned with 200ok status
set $maintenance 0;
}
if ($maintenance) {
# There can be several reasons for 503 error. We custom return 551 error
# to ensure maintenance.html is only shown when it's really maintenance
return 551;
}
rewrite ^/(.*)/$ /$1 permanent; # Truncate trailing slashes
try_files $uri #rails;
expires -1;
location = /favicon.ico {
try_files $uri =204;
access_log off;
log_not_found off;
}
location #rails {
proxy_pass http://myapp_mycompany_com;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_intercept_errors on;
expires -1;
}
error_page 500 502 503 504 /500.html;
error_page 403 /403.html;
error_page 404 /404.html;
client_max_body_size 50M;
keepalive_timeout 10;
}
Puma
$ bundle exec puma -d -e production -b unix:///tmp/myapp.com-puma.sock --pidfile /home/projects/mycompany/myapp.com/shared/tmp/pids/puma.pid
$
Example of access.log
123.123.123.123 - - [11/Jul/2014:05:44:17 +0000] "POST /tasks/easy_task/calculate/ HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:05:44:17 +0000] "GET /tasks/easy_task/calculate HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
...
123.123.123.123 - - [11/Jul/2014:06:04:17 +0000] "POST / HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:26 +0000] "POST /tasks HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:36 +0000] "POST /tasks/easy_task HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:42 +0000] "POST /tasks/easy_task/calculate HTTP/1.1" 200 104 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
TL;DR
If you want to completely redirect to a new resource and method and body of the requests should not be changed use 308 instead of 301 or 302.
301 is permanent redirect but 302 is temporary so search engines don't change urls associated with that website when 302 is used.
301 and 302 indicated method and body should not be altered, but not all user agents align with that. Read this explanation from Mozilla:
The HyperText Transfer Protocol (HTTP) 302 Found redirect status response code indicates that the resource requested has been temporarily moved to the URL given by the Location header. A browser redirects to this page but search engines don't update their links to the resource (in 'SEO-speak', it is said that the 'link-juice' is not sent to the new URL). Even if the specification requires the method (and the body) not to be altered when the redirection is performed, not all user-agents conform here - you can still find this type of bugged software out there. It is therefore recommended to set the 302 code only as a response for GET or HEAD methods and to use 307 Temporary Redirect instead, as the method change is explicitly prohibited in that case. In the cases where you want the method used to be changed to GET, use 303 See Other instead. This is useful when you want to give a response to a PUT method that is not the uploaded resource but a confirmation message such as: 'you successfully uploaded XYZ'.
308 and 307 both permanently redirect to a new resource but they guarantee body and method of request won't be altered. the difference is that 308 is permanent and 307 is temporary, so 308 will signal search engines to change urls. see this:
The only difference between 307 and 302 is that 307 guarantees that the method and the body will not be changed when the redirected request is made. With 302, some old clients were incorrectly changing the method to GET: the behavior with non-GET methods and 302 is then unpredictable on the Web, whereas the behavior with 307 is predictable. For GET requests, their behavior is identical.
I've found solution. When I did POST request, I used url which ends with slash, like http://myapp.com/tasks/easy_task/calculate/
When I used url without slash in the end, like http://myapp.com/tasks/easy_task/calculate everything works perfectly!
I think it is because of this rule
rewrite ^/(.*)/$ /$1 permanent; # Truncate trailing slashes
I am closing this issue. Tomorrow.
In my case, the redirect meant if I POST to http://... the proxy_pass is converted to a GET.
Change the URL to an https://... and it is passed on as a POST as intended.
location /dc1 {
proxy_pass http://localhost:8000;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host www.example.com;
}

Nginx, Rails and Oauth. upstream prematurely closed connection

I have a strange problem that appears in production environment only when I use nginx and unicorn. When I use unicorn without nginx, it doesn't happen.
The problem. I have a simple oauth authentication which allows users to signup through GitHub. After pressing "allow" on GitHub's authorization page, the user is being redirected to callback route. And then, he/she gets 302 Bad Gateway error.
Nginx log shows me this error (keys are replaced by "...")
2012/12/26 18:03:08 [error] 1467#0: *1 upstream prematurely closed
connection while reading response header from upstream, client:
10.0.2.2, server: _, request: "GET /auth/github/callback?code=&state=...
HTTP/1.1", upstream:
"http://unix:/tmp/unicorn.tm.sock:/auth/github/callback?code=...&state=...",
host: "localhost:3000"
Ther is my nginx config.
upstream unicorn {
server unix:/tmp/unicorn.tm.sock fail_timeout=0;
}
server {
listen 80 default deferred;
client_max_body_size 4G;
server_name _;
keepalive_timeout 75s;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
root /vagrant/public;
try_files $uri/index.html $uri.html $uri #app;
location #app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://unicorn;
proxy_buffer_size 16k;
proxy_busy_buffers_size 16k;
}
error_page 500 502 503 504 /500.html;
}
So my question is why does this happen and is there any possible fix for it?
I have been googling it for a while, but without luck.
update
Thanks for the comment, I have just tried to set fail_timeout=30s and it does help, however request takes about 40s to complete. But thanks anyway, I'll try to expirement with this parameters.
I updated my config a bit, according to suggestions, but still, get the same error.
Also, this is unicorn error log. Seems like it kills request which takes longer than 30s but I guess redirection from oauth site cannot possibly be that long...
(github) Request phase initiated.
(github) Callback phase initiated.
E, [2012-12-26T19:33:13.058183 #6002] ERROR -- : worker=0 PID:6005 timeout (31s > 30s), killing
E, [2012-12-26T19:33:13.067011 #6002] ERROR -- : reaped #<Process::Status: pid 6005 SIGKILL (signal 9)> worker=0
I, [2012-12-26T19:33:13.067198 #6002] INFO -- : worker=0 spawning...
I, [2012-12-26T19:33:13.068631 #6012] INFO -- : worker=0 spawned pid=6012
I, [2012-12-26T19:33:13.068726 #6012] INFO -- : Refreshing Gem list
I, [2012-12-26T19:33:17.140948 #6012] INFO -- : worker=0 ready
unicorn config
rails_env = ENV['RAILS_ENV'] || 'production'
worker_processes 1
listen "/tmp/unicorn.tm.sock", :backlog => 64
listen 8080, :tcp_nopush => true
timeout 30
pid "/tmp/unicorn.pid"
stderr_path "/tmp/unicorn.log"
stdout_path "/tmp/unicorn.log"
check_client_connection false
ERROR -- : worker=0 PID:6005 timeout (31s > 30s), killing
It goes without saying that you just need setup timeout to more then 30 in you unicorn config
Try at least
timeout 60
http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout

Resources