nginx redirects POST requests to GET request - ruby-on-rails

I have Rails 4.1 application with runs on puma web server. I use nginx as a proxy server. Several days ago everything worked very well. I updated my application, and suddenly some POST requests started to redirected to same url but as GET request. I've tried rollback to previous working versions, no success.
I found very interesting behaviour. I tested my API with curl.
If I did POST request to the url
http://myapp.com/tasks/easy_task/calculate/ it redirects to same url
but as GET request.
Then I did POSTrequest to http://myapp.com/, returned 404
Then I did POSTrequest to http://myapp.com/tasks, returned 404
Then I did POSTrequest to http://myapp.com/tasks/easy_task, returned 404
Then I did POSTrequest to http://myapp.com/tasks/easy_task/calculate, returned 200. YAY!
Same thing happened when I used chrome's app Postman. First it redirected, but after previous steps it works well.
I use this app in my other application. I use RestClient to make http requests. When I try to make POST request it raises an exception RestClient::MovedPermanently (301 Moved Permanently).
I reinstalled nginx to 1.7.3.
Restarted server (virtual machine)
re deployed my app, deployed previous versions
no success :(
I found similar questions on stackoverflow, but non of them gave me clue to fix this issue. I hope you can help me to solve this problem. Thanks in advance!
Similar questions:
- POST request turns into GET request
- POST request mysteriously turn into GET request
nginx config:
$ cat /etc/nginx/sites-enabled/myapp.com.conf
# The file generated by Chef for mycompany
upstream myapp_mycompany_com {
server unix:/tmp/myapp.com-puma.sock;
}
server {
server_name myapp.com;
listen 80;
access_log /var/log/nginx/myapp.com-access.log;
error_log /var/log/nginx/myapp.com-error.log;
root /home/projects/mycompany/myapp.com/current/public;
gzip on;
gzip_types text/plain text/xml application/xml application/xml+rss
text/css text/javascript application/javascript application/json;
error_page 551 =503 #maintenance;
location #maintenance {
rewrite ^(.*)$ /system/maintenance.html break;
}
set $maintenance 0;
if (-f $document_root/system/maintenance.html) {
set $maintenance 1;
}
if ($request_uri = /favicon.ico) {
# Browsers will try to get favicon if it's not returned with 200ok status
set $maintenance 0;
}
if ($maintenance) {
# There can be several reasons for 503 error. We custom return 551 error
# to ensure maintenance.html is only shown when it's really maintenance
return 551;
}
rewrite ^/(.*)/$ /$1 permanent; # Truncate trailing slashes
try_files $uri #rails;
expires -1;
location = /favicon.ico {
try_files $uri =204;
access_log off;
log_not_found off;
}
location #rails {
proxy_pass http://myapp_mycompany_com;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_intercept_errors on;
expires -1;
}
error_page 500 502 503 504 /500.html;
error_page 403 /403.html;
error_page 404 /404.html;
client_max_body_size 50M;
keepalive_timeout 10;
}
Puma
$ bundle exec puma -d -e production -b unix:///tmp/myapp.com-puma.sock --pidfile /home/projects/mycompany/myapp.com/shared/tmp/pids/puma.pid
$
Example of access.log
123.123.123.123 - - [11/Jul/2014:05:44:17 +0000] "POST /tasks/easy_task/calculate/ HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:05:44:17 +0000] "GET /tasks/easy_task/calculate HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
...
123.123.123.123 - - [11/Jul/2014:06:04:17 +0000] "POST / HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:26 +0000] "POST /tasks HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:36 +0000] "POST /tasks/easy_task HTTP/1.1" 404 713 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"
123.123.123.123 - - [11/Jul/2014:06:04:42 +0000] "POST /tasks/easy_task/calculate HTTP/1.1" 200 104 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2073.0 Safari/537.36"

TL;DR
If you want to completely redirect to a new resource and method and body of the requests should not be changed use 308 instead of 301 or 302.
301 is permanent redirect but 302 is temporary so search engines don't change urls associated with that website when 302 is used.
301 and 302 indicated method and body should not be altered, but not all user agents align with that. Read this explanation from Mozilla:
The HyperText Transfer Protocol (HTTP) 302 Found redirect status response code indicates that the resource requested has been temporarily moved to the URL given by the Location header. A browser redirects to this page but search engines don't update their links to the resource (in 'SEO-speak', it is said that the 'link-juice' is not sent to the new URL). Even if the specification requires the method (and the body) not to be altered when the redirection is performed, not all user-agents conform here - you can still find this type of bugged software out there. It is therefore recommended to set the 302 code only as a response for GET or HEAD methods and to use 307 Temporary Redirect instead, as the method change is explicitly prohibited in that case. In the cases where you want the method used to be changed to GET, use 303 See Other instead. This is useful when you want to give a response to a PUT method that is not the uploaded resource but a confirmation message such as: 'you successfully uploaded XYZ'.
308 and 307 both permanently redirect to a new resource but they guarantee body and method of request won't be altered. the difference is that 308 is permanent and 307 is temporary, so 308 will signal search engines to change urls. see this:
The only difference between 307 and 302 is that 307 guarantees that the method and the body will not be changed when the redirected request is made. With 302, some old clients were incorrectly changing the method to GET: the behavior with non-GET methods and 302 is then unpredictable on the Web, whereas the behavior with 307 is predictable. For GET requests, their behavior is identical.

I've found solution. When I did POST request, I used url which ends with slash, like http://myapp.com/tasks/easy_task/calculate/
When I used url without slash in the end, like http://myapp.com/tasks/easy_task/calculate everything works perfectly!
I think it is because of this rule
rewrite ^/(.*)/$ /$1 permanent; # Truncate trailing slashes
I am closing this issue. Tomorrow.

In my case, the redirect meant if I POST to http://... the proxy_pass is converted to a GET.
Change the URL to an https://... and it is passed on as a POST as intended.
location /dc1 {
proxy_pass http://localhost:8000;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host www.example.com;
}

Related

Rails/Nginx: Routes work on dev not on prod

My application works perfectly in my development environment, but routes don't work in my production server running nginx and Passenger, where I get:
404 Not Found
However, the index page does work as well as CSS and JavaScript.
I thought it was related to locale or i18n issues, but by changing routes.rb to remove :locale I got the same output.
Routes:
$ rake routes
Prefix Verb URI Pattern Controller#Action
GET /:locale(.:format) welcome#index
services GET /:locale/services(.:format) welcome#services
projects GET /:locale/projects(.:format) welcome#projects
contact GET /:locale/contact(.:format) welcome#contact
GET /:locale(.:format) welcome#index
root GET / welcome#index
rails_service_blob GET /rails/active_storage/blobs/:signed_id/*filename(.:format) active_storage/blobs#show
rails_blob_representation GET /rails/active_storage/representations/:signed_blob_id/:variation_key/*filename(.:format) active_storage/representations#show
rails_disk_service GET /rails/active_storage/disk/:encoded_key/*filename(.:format) active_storage/disk#show
update_rails_disk_service PUT /rails/active_storage/disk/:encoded_token(.:format) active_storage/disk#update
rails_direct_uploads POST /rails/active_storage/direct_uploads(.:format) active_storage/direct_uploads#create
Nginx/site-enable:
server {
listen 80;
listen [::]:80;
root /var/www/test.com/public;
index index.html index.htm index.nginx-debian.html;
server_name test.com www.test.com;
# Turn on Passenger
passenger_enabled on;
passenger_ruby /home/ubuntu/.rbenv/versions/2.5.1/bin/ruby;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
I'm using:
- Ruby 2.5.1
- gem 'rails', '~> 5.2.0'
production.log shows the rendering of the Welcome page but not /en/services for example when trying to go there:
I, [2020-02-23T17:08:28.146331 #19532] INFO -- : [4cd74925-0637-4b92-92aa-12f693691ecb] Processing by WelcomeController#index as HTML
I, [2020-02-23T17:08:28.146998 #19532] INFO -- : [4cd74925-0637-4b92-92aa-12f693691ecb] Rendering welcome/index.html.erb within layouts/application
I, [2020-02-23T17:08:28.147227 #19532] INFO -- : [4cd74925-0637-4b92-92aa-12f693691ecb] Rendered welcome/index.html.erb within layouts/application (0.1ms)
I, [2020-02-23T17:08:28.148286 #19532] INFO -- : [4cd74925-0637-4b92-92aa-12f693691ecb] Completed 200 OK in 2ms (Views: 1.4ms)
nginx/error.log:
[ N 2020-02-23 17:13:07.1716 25594/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2020-02-23 17:13:07.1994 25602/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2020-02-23 17:13:07.1995 25602/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2020-02-23 17:13:07.2058 25602/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 25602
nginx/access.log:
[23/Feb/2020:17:14:23 +0000] "GET /favicon.ico HTTP/1.1" 200 0 "http://www.XXX.fr/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36"
[23/Feb/2020:17:14:28 +0000] "GET /en/services HTTP/1.1" 404 209 "http://www.XXX.fr/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36"
How do I fix this?
I would start by looking into log file of your application, 404 response could be caused by many elements of your stack.
When you open your application root page you should see a request logged to log/production.log file (assuming this is in production environment). If you don't see it, look up the stack at nginx error log (normally Passenger is using this one for stderr).
I resolved by commenting the "location" part of the nginx conf:
# location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
# try_files $uri $uri/ =404;
# }

nginx not accepting POST form data

When i POST to server with this
POST /api/user/login HTTP/1.1
Host: xxxx.com
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
it works, but not this
POST /api/user/login HTTP/1.1
Host: xxxx.com
Cache-Control: no-cache
is it possible to get the first one work?
both of them were working, the thing that blocked me was
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
I changed those lines so that my application can handle 500 502 503 504 errors.
Now removed those lines and its working as expected. Might help somebody

Rails app running on puma and nginx keeps dying every few hours with Bad Gateway

I have a rails app that I just deployed to Digital Ocean and it's running on Puma and Nginx.
Eventually all it returns is a bad gateway and this is what is in the error.log
2014/09/09 22:23:06 [error] 5729#0: *3059 connect() to unix:///var/www/mysite/mysite_app.sock failed (111: Connection refused) while connecting to upstream, client: 67.5.19.192, server: mysite.com, request: "GET / HTTP/1.1", upstream: "http://unix:///var/www/mysite/mysite_app.sock:/", host: "mysite.com"
To fix it, I just restart puma and it seems to work.
How can I debug this to figure out why it keeps dying?
Here's my nginx config:
upstream mysite {
server unix:///var/www/mysite/mysite_app.sock;
}
server {
listen 80;
server_name mysite.com;
root /var/www/mysite/current/public;
client_max_body_size 20M;
location / {
proxy_pass http://mysite; # match the name of upstream directive which is defined above
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ~* ^/assets/ {
# Per RFC2616 - 1 year maximum expiry
expires 1y;
add_header Cache-Control public;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
}
EDIT
Could this be caused from running out of memory?
Here's my current state of memory, but as I keep running this command every so often, the amount of free memory goes down and once i restart puma it jumps back up to like 150.
$ free -m
total used free shared buffers cached
Mem: 490 440 50 0 17 84
-/+ buffers/cache: 338 151
Swap: 0 0 0
It seems this is actually an issue with ruby 2.1 (specifically i'm using 2.1.2) and its garbage collection.
A google search like this seems to have lots of various threads on it http://bit.ly/1s2vBC0
Here's a ruby bug ticket on the issue: https://bugs.ruby-lang.org/issues/9607
Lack of memory can be issue, but you better look into puma and rails logs, not nginx only. In the application folder:
tail -f log/puma*
tail -f log/production.log
I've had similar issues but I note that 2.1.3 has now been released and specifically discusses memory issues:
https://www.ruby-lang.org/en/news/2014/09/19/ruby-2-1-3-is-released/
I'm going to try it now!

Rails 4, Puma, Nginx - ActionController::Live Streaming dies after first chunk sent

Here's a bare-bones Rails 4 project I've set up to troubleshoot my problem:
https://github.com/rejacobson/rails4-streamtest
I have a route set up at /home/stream that should stream a line of text 5 times in 1 second intervals.
def stream
5.times do |n|
puts "Streaming: #{n}"
response.stream.write "Streaming: #{n+1}"
sleep 1
end
rescue IOError => e
puts 'Connection closed'
ensure
response.stream.close
end
When I run puma using tcp://, without nginx, the streaming works perfectly.
curl -N http://localhost:3000/home/stream
And I get the 5 lines streamed back, no problem.
When I introduce nginx, curl will output the first line but immediately exit after that. I do continue to see output from the puts calls on the server and logs, so I know the request is still processing in the 5.times loop.
It also doesn't throw an IOError exception like it would if the user cut the connection.
Here's what I've tried so far:
Different combinations of nginx directives in the main conf file and
the server conf.
Changing rails settings.
Various puma config settings.
Setting all kinds of different headers in the controller method in the hopes that it was a caching issue.
Searching the internet has provided very little help as well.
I'm at a loss and require some direction.
Here's the output of both curl calls, with and without nginx.
Without nginx
~/projects/streamtest ▰ master ▰
ryan mirage ▰▰▰▰ curl -i -N http://192.168.1.100:3000/home/stream
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Set-Cookie: request_method=GET; path=/
X-Request-Id: 9ce86358-4476-404a-97e5-769c16ec7b0c
X-Runtime: 0.978099
Transfer-Encoding: chunked
Streaming: 1Streaming: 2Streaming: 3Streaming: 4Streaming: 5
puma.stdout
Streaming: 0
Streaming: 1
Streaming: 2
Streaming: 3
Streaming: 4
[8048] 192.168.1.100 - - [14/Mar/2014 21:04:50] "GET /home/stream HTTP/1.1" 200 - 6.0661
With nginx
~/projects/streamtest ▰ master ▰
ryan mirage ▰▰▰▰ curl -i -N http://192.168.1.100:3000/home/stream
HTTP/1.1 200 OK
Server: nginx/1.4.5
Date: Sat, 15 Mar 2014 04:02:40 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
ETag: "a505e0aa3b11b25301a9a704252a519a"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: request_method=GET; path=/
X-Request-Id: 8983d199-026b-4082-a5f1-f1d6c886a3d6
X-Runtime: 0.016516
Streaming: 1
puma.stdout
Streaming: 0
[7558] 192.168.1.100 - - [14/Mar/2014 21:02:40] "GET /home/stream HTTP/1.0" 200 - 0.0214
Streaming: 1
Streaming: 2
Streaming: 3
Streaming: 4
What's interesting, and I've just noticed it, is that the location of the get request log line:
"GET /home/stream HTTP/1.0" 200
is different in each curl call, and is placed in relation to how much text is actually streamed.
Any ideas as to what's going on here? Why can't rails stream the entire thing when using nginx?
Solved
It turns out all that I needed was this line in my location directive to get streaming working through nginx:
proxy_http_version 1.1;
I discovered this fix when investigating the keepalive directive in nginx's upstream module:
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
This piece of text in particular clued me in:
For HTTP, the proxy_http_version directive should be set to “1.1” and the “Connection” header field should be cleared:
upstream http_backend {
server 127.0.0.1:8080;
keepalive 16;
}
server {
...
location /http/ {
proxy_pass http://http_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
...
}
}
It seems that nginx defaults to, proxy_http_version 1.0;
And according to Wikipedia, http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
HTTP/1.1 introduced chunked transfer encoding to allow content on persistent connections to be streamed rather than buffered.
So there was my problem.
TIL

rails application display blank html

Ruby 1.92
Rails 3.1.1
Nginx 1.1.6
Passenger 3.0.7
My rails application works fine on my laptop, but it does not work on amazon server. I open it in chrome, browser display nothing, HTTP header like below:
HTTP/1.1 200
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Status: 302
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.7
If I run rails server, it does work, I can get correct html. Nginx cannot generate correct content-length?
nginx config
server {
listen 8000;
server_name xxxx;
root /xxx/xxx/public;
passenger_enabled on;
rails_env development;
location = /favicon.ico {
expires max;
add_header Cache-Control public;
}
location ~* \.(png|gif|jpg|jpeg|css|js|swf|ico)(\?[0-9]+)?$ {
access_log off;
expires max;
add_header Cache-Control public;
}
}
http raw message is below
total 3496 bytes return, content-length is wrong,so browser do read rest html
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.7
X-UA-Compatible: IE=Edge
ETag: "12aa68a45bf774886311f827d2149cbe"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.499998
Server: nginx/1.1.6 + Phusion Passenger 3.0.7 (mod_rails/mod_rack)
<!DOCTYPE html>
<html>
<head>
<title>xxxx</title>
......
A HTTP status of 302 is usually a redirect. Are there any other headers or body content? I'd expect a location header telling the browser where to redirect to.

Resources