Rails 4, Puma, Nginx - ActionController::Live Streaming dies after first chunk sent - ruby-on-rails

Here's a bare-bones Rails 4 project I've set up to troubleshoot my problem:
https://github.com/rejacobson/rails4-streamtest
I have a route set up at /home/stream that should stream a line of text 5 times in 1 second intervals.
def stream
5.times do |n|
puts "Streaming: #{n}"
response.stream.write "Streaming: #{n+1}"
sleep 1
end
rescue IOError => e
puts 'Connection closed'
ensure
response.stream.close
end
When I run puma using tcp://, without nginx, the streaming works perfectly.
curl -N http://localhost:3000/home/stream
And I get the 5 lines streamed back, no problem.
When I introduce nginx, curl will output the first line but immediately exit after that. I do continue to see output from the puts calls on the server and logs, so I know the request is still processing in the 5.times loop.
It also doesn't throw an IOError exception like it would if the user cut the connection.
Here's what I've tried so far:
Different combinations of nginx directives in the main conf file and
the server conf.
Changing rails settings.
Various puma config settings.
Setting all kinds of different headers in the controller method in the hopes that it was a caching issue.
Searching the internet has provided very little help as well.
I'm at a loss and require some direction.
Here's the output of both curl calls, with and without nginx.
Without nginx
~/projects/streamtest ▰ master ▰
ryan mirage ▰▰▰▰ curl -i -N http://192.168.1.100:3000/home/stream
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Set-Cookie: request_method=GET; path=/
X-Request-Id: 9ce86358-4476-404a-97e5-769c16ec7b0c
X-Runtime: 0.978099
Transfer-Encoding: chunked
Streaming: 1Streaming: 2Streaming: 3Streaming: 4Streaming: 5
puma.stdout
Streaming: 0
Streaming: 1
Streaming: 2
Streaming: 3
Streaming: 4
[8048] 192.168.1.100 - - [14/Mar/2014 21:04:50] "GET /home/stream HTTP/1.1" 200 - 6.0661
With nginx
~/projects/streamtest ▰ master ▰
ryan mirage ▰▰▰▰ curl -i -N http://192.168.1.100:3000/home/stream
HTTP/1.1 200 OK
Server: nginx/1.4.5
Date: Sat, 15 Mar 2014 04:02:40 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
ETag: "a505e0aa3b11b25301a9a704252a519a"
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: request_method=GET; path=/
X-Request-Id: 8983d199-026b-4082-a5f1-f1d6c886a3d6
X-Runtime: 0.016516
Streaming: 1
puma.stdout
Streaming: 0
[7558] 192.168.1.100 - - [14/Mar/2014 21:02:40] "GET /home/stream HTTP/1.0" 200 - 0.0214
Streaming: 1
Streaming: 2
Streaming: 3
Streaming: 4
What's interesting, and I've just noticed it, is that the location of the get request log line:
"GET /home/stream HTTP/1.0" 200
is different in each curl call, and is placed in relation to how much text is actually streamed.
Any ideas as to what's going on here? Why can't rails stream the entire thing when using nginx?

Solved
It turns out all that I needed was this line in my location directive to get streaming working through nginx:
proxy_http_version 1.1;
I discovered this fix when investigating the keepalive directive in nginx's upstream module:
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
This piece of text in particular clued me in:
For HTTP, the proxy_http_version directive should be set to “1.1” and the “Connection” header field should be cleared:
upstream http_backend {
server 127.0.0.1:8080;
keepalive 16;
}
server {
...
location /http/ {
proxy_pass http://http_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
...
}
}
It seems that nginx defaults to, proxy_http_version 1.0;
And according to Wikipedia, http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol
HTTP/1.1 introduced chunked transfer encoding to allow content on persistent connections to be streamed rather than buffered.
So there was my problem.
TIL

Related

How to verify HSTS is working as expected in Rails application?

Most of the documentation talks about adding config.force_ssl = true which provides HSTS to Rails application. Well, how can I make sure its working as expected? How can I test HSTS header is added for every request? Any help is much appreciated.
Use curl -I https://your_website.domain or browser developer tools to view response headers. You're interested in the Strict-Transport-Security header:
curl -I https://www.ssllabs.com/
HTTP/2 200
date: Fri, 23 Apr 2021 15:25:17 GMT
server: Apache
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000 <-- this one
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: 0
x-frame-options: DENY
content-type: text/html;charset=ISO-8859-1
content-language: en-US
content-length: 6587
set-cookie: ...
content-security-policy: ...

Glyphicon issue / CORS problems still persist, even after using font_assets and custom defining headers

I recently deployed an app -- http://jobs.atlas-china.com
If you go on to http://jobs.atlas-china.com/jobs/2 , you'll notice that the glyphicon for the resume upload does not show up in firefox.
This is because of the Cross Site Header issue. I've tried to fix this by defining a default header, but it's still being wacky.
My application.rb looks like so --
module Atlas
class Application < Rails::Application
config.assets.precompile += %w(*.png *.jpg *.jpeg *.gif)
config.action_dispatch.default_headers = {
'X-Frame-Options' => 'ALLOWALL',
'X-XSS-Protection' => '1; mode=block',
'X-Content-Type-Options' => 'nosniff',
'Access-Control-Allow-Origin' => ENV['APP_URL']
}
end
end
The font assets are being loaded from cloudfront, so I though this could be a caching issue. However, if I try to make a curl request to the equivalent heroku url, I get --
~$ curl -i http://jobs.atlas-china.com/assets/bootstrap/glyphicons-halflings-regular-fcc658a3dec1be1cb0a9bb81f4c7c6de.woff
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=315360000
Cache-Control: public
Content-Type: font/x-woff
Date: Fri, 30 May 2014 05:13:52 GMT
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Last-Modified: Wed, 16 Apr 2014 06:51:18 GMT
Server: nginx/1.4.7
Content-Length: 23320
Connection: keep-alive
I still don't see anything about an Access-Control-Allow-Origin
What should I be doing here?
update
When I make a curl request to the root url, I do see it though!
curl -i http://jobs.atlas-china.com
HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://jobs.atlas-china.com
Cache-Control: max-age=0, private, must-revalidate
Content-Type: text/html; charset=utf-8
Date: Fri, 30 May 2014 05:21:10 GMT
Etag: "a73e238bf8cb6ccb7fdf53ae108e11c6"
Server: nginx/1.4.7 + Phusion Passenger 4.0.41
Set-Cookie: _atlas_session=UkVHc0ZSVko2QmdZMWp3djhuelpvUEtMRXVlU1FJclRSN1dpcTR6QWlDS0gyUU15UmdBY0dZWTMya0FtUnFST2RkVnBiWURUdkRTZVJLNk9JcUxlUnZKWHRWaWoxZnZPdThVVTVMMU5qRlpkQnJxUUVBWHQ3WjUreVZ4VENWeTE1WHF3Sit3ZVFQSzMxYmFRVER0aUpsNUN3OW5IOHJQenIzcU9ZcSt5cndwaWlQRXRheVA3dVZMbTVaek5CRFphLS1SNHo4YjlWQ0JXc0U1MlN5R1NjTVlnPT0%3D--0b975a5008287efb7f8114cbcbedd57b34b7d0f9; path=/; HttpOnly
Status: 200 OK
Strict-Transport-Security: max-age=0
X-Content-Type-Options: nosniff
X-Frame-Options: ALLOWALL
X-Powered-By: Phusion Passenger 4.0.41
X-Request-Id: 964dee7f-d49b-4d68-9a35-f8cd0f6371dc
X-Runtime: 0.147926
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive
Seems to work for me?
CORS
FYI
Seems like you know already, but CORS is basically a way to protect domains / servers from external AJAX requests. The standard setup is the server will deny requests from external domains automatically, regardless of the headers you send
The way to accept / deny any CORS request is to set the policy options on the Rails server, to allow requests from the domains you want. Rails has a gem to help you do this, called Rack-CORS:
#config/application.rb
config.middleware.use Rack::Cors do
allow do
origins 'your_domain.com' #-> has to be exact domain
resource '/your/url', :headers => :any, :methods => [:get, :post, :options]
end
end
I don't know if this will help you or not, but it will certainly give you some ideas as to how to DRY up your CORS headers, as well as giving you the ability to accept requests as you require

How to raise 505 exception in ruby on rails in development?

Is there a way to raise a 505 exception with ruby on rails in development?
I would like to testdrive my custom 505 exceptions but found no documentation on how to raise a 505 anywhere.
i am a little confused here. are you talking about a 505 status codes? or are you talking about raising exceptions?
i don't think that there is a 505 exception, so i assume you are talking about HTTP status code version not supported (http://en.wikipedia.org/wiki/List_of_HTTP_status_codes).
it could be as simple as this:
def index
render nothing: true, status: 505
end
if you call an action like that you will get something like this as a response:
curl -I http://hamburg.onruby.dev:5000/
HTTP/1.1 505 HTTP Version not supported
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
Content-Type: text/html; charset=utf-8
X-Meta-Request-Version: 0.2.8
Cache-Control: no-cache
Set-Cookie: locale=de; domain=onruby.dev; path=/; expires=Sat, 11 Oct 2014 07:45:52 -0000
X-Request-Id: b8232480-5585-4e99-bc4c-44019e41db6a
X-Runtime: 0.010615
Connection: close
Server: thin 1.5.1 codename Straight Razor

rails application display blank html

Ruby 1.92
Rails 3.1.1
Nginx 1.1.6
Passenger 3.0.7
My rails application works fine on my laptop, but it does not work on amazon server. I open it in chrome, browser display nothing, HTTP header like below:
HTTP/1.1 200
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Status: 302
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.7
If I run rails server, it does work, I can get correct html. Nginx cannot generate correct content-length?
nginx config
server {
listen 8000;
server_name xxxx;
root /xxx/xxx/public;
passenger_enabled on;
rails_env development;
location = /favicon.ico {
expires max;
add_header Cache-Control public;
}
location ~* \.(png|gif|jpg|jpeg|css|js|swf|ico)(\?[0-9]+)?$ {
access_log off;
expires max;
add_header Cache-Control public;
}
}
http raw message is below
total 3496 bytes return, content-length is wrong,so browser do read rest html
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.7
X-UA-Compatible: IE=Edge
ETag: "12aa68a45bf774886311f827d2149cbe"
Cache-Control: max-age=0, private, must-revalidate
X-Runtime: 0.499998
Server: nginx/1.1.6 + Phusion Passenger 3.0.7 (mod_rails/mod_rack)
<!DOCTYPE html>
<html>
<head>
<title>xxxx</title>
......
A HTTP status of 302 is usually a redirect. Are there any other headers or body content? I'd expect a location header telling the browser where to redirect to.

enabling rails page caching causes http header charset to disappears

I need charset to be utf-8, which seem to be the case by default. Recently I enabled page caching for a few static pages:
caches_page :about
The caching works fine, and I see the corresponding about.html and contact.html pages generated in my /public folder, except when the page renders, it's no longer in utf-8.
After googling for a bit I tried looking at the http headers with wget, before and after caching:
first time:
$wget --server-response http://localhost:3000/about
HTTP request sent, awaiting response...
1 HTTP/1.1 200 OK
2 X-Ua-Compatible: IE=Edge
3 Etag: "f7b0b4dea015140f3b5ad90c3a392bef"
4 Connection: Keep-Alive
5 Content-Type: text/html; charset=utf-8
6 Date: Sun, 12 Jun 2011 03:44:22 GMT
7 Server: WEBrick/1.3.1 (Ruby/1.8.7/2009-06-12)
8 X-Runtime: 0.235347
9 Content-Length: 5520
10 Cache-Control: max-age=0, private, must-revalidate
cached:
$wget --server-response http://localhost:3000/about
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:3000... connected.
HTTP request sent, awaiting response...
1 HTTP/1.1 200 OK
2 Last-Modified: Sun, 12 Jun 2011 03:34:42 GMT
3 Connection: Keep-Alive
4 Content-Type: text/html
5 Date: Sun, 12 Jun 2011 03:39:53 GMT
6 Server: WEBrick/1.3.1 (Ruby/1.8.7/2009-06-12)
7 Content-Length: 5783
as a result the page displays in ISO-8859-1 and I get a bunch of garbled text. Does anyone know how I can prevent this undesirable result? Thank you.
The solution will depend on the server used.
When you use page cache, the servers reads the server directly, so the rails stack does not provide encoding information to the server. Then the server default apply.
If you're using apache with passenger, add to the configuration:
AddDefaultCharset UTF-8
If you need specific charsets, use a solution like the one in http://www.philsergi.com/2007/06/rails-page-caching-and-mime-types.html
<LocationMatch \/(rss)\/?>
ForceType text/xml;charset=utf-8
</LocationMatch>
<LocationMatch \/(ical)\/?>
ForceType text/calendar;charset=utf-8
</LocationMatch>

Resources