Thin + Nginx + Upload Module + Upload Progress Module - ruby-on-rails

I'm using Nginx as a reverse proxy for Thin instances.
My goal is to set up a Rails (3) app to upload large files and do something with them.
For that, I came across the Nginx Upload and Upload Progress modules.
I was reading, for the most part, this post, but that's specifically wrote thinking in Passenger.
If possible, I'm looking for two possible answers:
1) Information an examples of implementing this stack (with Thin instead of Passenger)
2) Specific Information of how could I rewrite this:
location ^~ /progress {
# report uploads tracked in the 'proxied' zone
upload_progress_json_output;
report_uploads proxied;
}
location #fast_upload_endpoint {
passenger_enabled on;
rails_env development;
}
location / {
rails_env development;
passenger_enabled on;
}
I don't know what is Passenger exclusive, and how to write it for a typical 4 workers / 3 thin instances conf.
Thanks.

First, you should install nginx with the upload module. The nginx config for site:
upstream uploader_cluster {
server unix:/tmp/thin.uploader.0.sock;
server unix:/tmp/thin.uploader.1.sock;
server unix:/tmp/thin.uploader.2.sock;
server unix:/tmp/thin.uploader.3.sock;
server unix:/tmp/thin.uploader.4.sock;
}
server {
listen 80;
server_name ***.com;
charset off;
client_max_body_size 1000m;
access_log /var/www/uploader/log/access.log;
error_log /var/www/uploader/log/error.log;
root /var/www/uploader/public;
index index.html;
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (!-f $request_filename) {
proxy_pass http://uploader_cluster;
break;
}
}
location ~*uploads$ {
if ($request_method = GET) {
proxy_pass http://uploader_cluster;
break;
}
# pass request body to here
upload_pass #upload_photos;
# Store files to this directory
# The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
# i.e. make sure to create /vol/uploads/0 /vol/uploads/1 etc.
upload_store /vol/uploads 1;
# set permissions on the uploaded files
upload_store_access user:rw group:rw all:r;
# Set specified fields in request body
# this puts the original filename, new path+filename and content type in the requests params
upload_set_form_field upload[file_name] "$upload_file_name";
upload_set_form_field upload[file_content_type] "$upload_content_type";
upload_set_form_field upload[file_path] "$upload_tmp_path";
upload_aggregate_form_field upload[file_size] "$upload_file_size";
upload_pass_form_field "^fb_sig_user$|^aid$|^batch_id$|^album_name$|^album_visible$|^caption$|^tags$|^complete$";
upload_cleanup 400 404 499 500-505;
}
location #upload_photos {
proxy_pass http://uploader_cluster;
}
}

Related

Nginx + Rails + sendfile : file not found

Been having a problem setting up sendfile for nginx+rails lately. We have one kind of file download that's already handled by nginx, and we wanted to add a second rule to deal with another kind of files at another location, but we have no success so far.
Ruby controller :
def download_file
send_file("/srv/www/myapp/shared/tmp/directory/file.zip")
end
Environment configuration file:
Rails.application.configure do
# ..
config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for nginx
# ..
end
Nginx configuration :
upstream myapp {
server 127.0.0.1:9292;
}
server {
listen 80;
server_name myapp.tld;
client_max_body_size 10M;
root /srv/www/myapp/current/public;
# This first block works perfectly
location /__working_file {
internal;
alias /var/lib/myapp;
}
# This second block does not work at all
location /__new_files {
internal;
alias /srv/www/myapp/shared/tmp/directory;
}
location / {
root /srv/www/myapp/current/public;
try_files $uri #app;
}
location #app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Sendfile-Type X-Accel-Redirect;
# This first rule works perfectly
proxy_set_header X-Accel-Mapping /var/lib/myapp/=/__working_file/;
# This second rule doesn't
proxy_set_header X-Accel-Mapping /srv/www/myapp/shared/tmp/directory/=/__new_files/;
proxy_pass_header Server;
proxy_read_timeout 300;
proxy_pass http://myapp;
}
}
Result
When accessing the controller action, the send_file command is triggered, then we get a "File not found" in the browser, nothing gets downloaded, and the rails log shows this:
Sent file /srv/www/myapp/shared/tmp/directory/file.zip (0.4ms)
Completed 200 OK in 169ms (ActiveRecord: 37.5ms)
Started GET "/srv/www/myapp/shared/tmp/directory/file.zip" for 109.190.197.126 at 2018-11-20 11:34:44 +0100
ActionController::RoutingError (No route matches [GET] "/srv/www/myapp/shared/tmp/directory/file.zip"):
The file does exist and is readable but nginx can't seem to access it. Any idea?
The problem comes from the way the two X-Accel-Mapping are set. Rack is indeed able to deal with several mappings since the merge of this PR #1187.
However, as of today, this PR has been merged on master but not released yet (2.0.6 is currently the latest release).
The only thing is, the correct way of setting several mappings is by using one single proxy_set_header rule, and separate each mapping with a comma, like this:
proxy_set_header X-Accel-Mapping, /var/lib/myapp/=/__working_file/,/srv/www/myapp/shared/tmp/directory/=/__new_files/
while routing it is not getting . you have to provide proper route to work.

Load Balancing - Web Applications with NGINX

I have a web application ruby on rails who is not configured to multithreading.
In nginx config, I set up an upstream block to be load balanced.
Like this :
upstream myapp {
server 127.0.0.1:3075;
server 127.0.0.1:3076;
server 127.0.0.1:3077;
}
I set up also 3 process thin with 3 ports (3075,3076,3077).
I think when my first application '127.0.0.1:3075' is busy, all the request will be balanced automatically to my second application '127.0.0.1:3076' or the third one.
But load balancing is not work, even though my three web applications are running correctly independent.
Please help me to find errors.
------------------- nginx config --------------------
upstream myapp_hosts {
server 127.0.0.1:3075;
server 127.0.0.1:3076;
server 127.0.0.1:3077;
}
server {
listen 80;
server_name myapp.mydomain.com;
rewrite ^(.*)$ https://myapp.mydomain.com$1 permanent; # rewrite for https, i have another bloc server listen 443.
access_log /var/log/nginx/myapp.access.log;
location / {
proxy_pass http://myapp_hosts/;
proxy_connect_timeout 900;
proxy_send_timeout 900;
proxy_read_timeout 900;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_busy_buffers_size 64k;
}
location /public {
root /var/www/nemo/;
}
location /images {
root /var/www/nemo/assets/;
}
location /javascripts {
root /var/www/nemo/assets/;
}
location /stylesheets {
root /var/www/nemo/assets/;
}
client_max_body_size 10m;
client_body_buffer_size 128k;
client_header_buffer_size 64k;
}
What is the purpose of your rewrite?
rewrite ^(.*)$ http://myapp.mydomain.com$1 permanent;
It looks like it's going to constantly redirect anything to itself based off of those rules, resulting in a redirect loop. You may have mixed this line up with an HTTPS redirect configuration you found somewhere else, perhaps?
Try removing that line and see if it works.

Nginx does not cache cross domain requests (CORS)

I have rails application and nginx.
There are parts of the nginx config:
http {
...
proxy_cache_path /var/cache/nginx levels= keys_zone=cache:50m inactive=1d;
...
}
server {
...
location #rails {
proxy_pass http://rails_server;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_read_timeout 300m;
proxy_cache cache;
proxy_cache_valid 1h;
set $proxy_hide_header 'Set-Cookie';
proxy_hide_header $proxy_hide_header;
proxy_ignore_headers "Cache-Control" "Expires";
}
...
}
I have a promblem with caching cross domain requests.
Cache files are created when I open the url my_domain/data.js directly.
Cache files are created when I open the url my_domain/data.js from my_domain/index.html using ajax request.
Cache files are not created when I open the url my_domain/data.js from my_domain2/index.html using ajax request.
Content and http code are ok in all cases.
The problem was in rails (not in nginx).
Rails add "Set-Cookie" header for cross domain requests even if cookie is already set (I don't know why, maybe it's rails bug).
So my solution is:
proxy_ignore_headers "Set-Cookie";

Content-Length Header missing from Nginx-backed Rails app

I've a rails app that serves large static files to registered users. I was able to implement it by following the excellent guide here: Protected downloads with nginx, Rails 3.0, and #send_file. The downloads and everything else is working great, but there is just this problem - The Content-Length header isn't being sent.
It's okay for small files, but it gets really frustrating when large files are downloaded, since download managers and browsers don't show any progress. How can I fix this? Do I have to add something to my nginx configuration or do I have to pass along some other option to the send_file method in my rails controller? I have been searching online for quite some time but have been unsuccessful. Please Help! Thanks!
Here's my nginx.conf:
upstream unicorn {
server unix:/tmp/unicorn.awesomeapp.sock fail_timeout=0;
}
server {
listen 80 default_server deferred;
# server_name example.com;
root /home/deploy/apps/awesomeapp/current/public;
location ~ /downloads/(.*) {
internal;
alias /home/deploy/uploads/$1;
}
location ^~ /assets/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
try_files $uri/index.html $uri #unicorn;
location #unicorn {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_set_header X-Sendfile-Type X-Accel-Redirect;
proxy_set_header X-Accel-Mapping /downloads/=/home/deploy/uploads/;
proxy_pass http://unicorn;
}
error_page 500 502 503 504 /500.html;
client_max_body_size 20M;
keepalive_timeout 10;
}
Okay, here's something. I don't know if it's the right way or not but I was able to fix the issue by manually sending the Content-Length Header from my Rails Controller. Here's what I'm doing:
def download
#file = Attachment.find(params[:id])
response.headers['Content-Length'] = #file.size.to_s
send_file(#file.path, x_sendfile: true)
end
nginx should be automatically able to set the header. There must be something that I'm missing; but until I find a 'proper' solution, I guess this will have to do.
P.S: The Header needs to be a string to work properly with some webservers, hence the .to_s

Rails, favicon.ico not found

This is soo odd, I've been receiving:
ActionController::RoutingError (No route matches "/favicon.ico")
but I have the favicon.ico in my public directory... any ideas how to solve this? Nginx doesn't throw an error at all.
Run
rake assets:precompile
then set
config.serve_static_assets = true
in config\environments\production.rb file.
Then restart your server.
But I think rake assets:precompile is not required.
It seems that nginx doesn't handle your static assets (since this request for static file goes to the ActionController). Check public root in nginx config file nginx.conf.
Here is an example with Capistrano deployments:
server {
listen 80;
root /var/www/my_project/current/public;
}
And do you use a favicon_link_tag helper in your head :) ?
If you want to keep config.serve_static_assets = false, which is recommended if you have nginx or apache, you can tell nginx to statically serve the files directly. This is especially important for performance reasons, as you don't want rails serving these assets.
Below is a sample which also correctly has nginx statically serve the assets directory:
server {
listen 80;
root /var/www/my_project/current/public;
location / {
proxy_pass http://mysite;
proxy_redirect off;
proxy_set_header X_FORWARDED_PROTO https;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
# static resource routing - both assets folder and favicon.ico
location ~* ^/assets/|favicon.ico {
# Per RFC2616 - 1 year maximum expiry
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
expires 1y;
add_header Cache-Control public;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
}
Make sure that the favicon.ico file isn't empty (byte size > 0). For some reason I had an empty favicon.ico file which was triggering the same error, even though the file did exist.
delete slash sign before favicon.ico and try to use something like:
<link rel="shortcut icon" type="image/png" href="favicon.ico" />
I was facing the same problem when I first clone code from git repository and run with RAILS_ENV=production. Since there was no assets directory in my git repository, I needed to run rake assets:precompile.
Also I was run with rails s, so config.serve_static_assets = true worked. Thanks #Jiemurat

Resources