Unicorn Rails stack + Vagrant not serving some assets - ruby-on-rails

I'm using a Rails Stack with nginx + unicorn + rails meant for a production server, but I'm staging it under Vagrant for testing purposes. While doing this, I encountered a strange behavior of the rails application, where very often one or other asset wasn't being served, i.e. application.css isn't being served and therefore the whole page is displayed without any styles applied to it.
I've googled the problem and found that Vagrant's FS driver isn't completely implemented and that would bring some problems while using Apache (haven't found any mentions to nginx). The solution to this problem was to disable sendfile by adding sendfile off; to the configuration file. And... it didn't work.
Further, I went through the logs (Rails, unicorn and nginx) and found that when the file isn't served, there isn't any mention to it in any of the logs. This brought me to the question that the problem can be in the mechanism used by Vagrant to share the rails app folder through the VM. As mentioned in vagrant's website, Vagrant uses Virtual Box's Shared Folders, which is quite slow comparing to other alternatives (as shown here), and the workaround is to set up NFS Shared Folders. So, I decided to give NFS a try and the result was... the same. Unfortunately, some assets are kept from being served.
Does anyone have any ideas on this? I've searched for quite a while but haven't found any pointers additional to those I described here.
I'm using:
Mac OS X 10.6.8 + rbenv (to develop)
Vagrant + nginx + rbenv + unicorn + bundler (to stage)
unicorn.rb
rails_env = ENV['RAILS_ENV'] || 'production'
app_directory = File.expand_path(File.join(File.dirname(__FILE__), ".."))
worker_processes 4
working_directory app_directory
listen "/tmp/appname.sock", :backlog => 64
#listen "#{app_directory}/tmp/sockets/appname.sock", :backlog => 64
timeout 30
pid "#{app_directory}/tmp/pids/unicorn.pid"
stderr_path "#{app_directory}/log/unicorn.stderr.log"
stdout_path "#{app_directory}/log/unicorn.stdout.log"
preload_app true
GC.respond_to?(:copy_on_write_friendly=) and
GC.copy_on_write_friendly = true
before_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
end
after_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end
/etc/nginx/sites-enabled/appname
upstream unicorn_server {
server unix:/tmp/appname.sock fail_timeout=0;
}
server {
listen 80;
client_max_body_size 4G;
server_name _;
keepalive_timeout 5;
# Location of our static files
root /home/appname/www/current/public;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://unicorn_server;
break;
}
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /home/hemauto/www/current/public;
}
}

Related

Assets precompiled but not showing with puma & nginx (Rails 4)

In my local machine when I do:
RAILS_ENV=production bundle exec rake assets:precompile
RAILS_ENV=production bundle exec puma -e production
Everything works fine. However I have the exact same app in a docker container for production that runs with nginx. I can see the application.css & application.js files in the assets folder with the chrome dev tools and they're not empty. But I have a page with no css/js, it should have something to do with nginx but I'm really confused.
/var/etc/nginx/nginx.conf:
user app sudo;
http{
upstream app {
server unix:/home/app/puma.sock fail_timeout=0;
}
server {
listen 80 default;
root /home/app/app/public;
try_files $uri/index.html $uri #app;
location #app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app;
}
}
}
events {worker_connections 1024;}
config/environments/production.rb:
Rails.application.configure do
...
config.consider_all_requests_local = false
config.action_controller.perform_caching = true
config.serve_static_files = true
config.assets.js_compressor = :uglifier
config.assets.compile = false
config.assets.digest = true
...
end
Please do not hesitate to suggest anything. :)
update:
I've just realised that I have these 2 errors in my console:
new:11 Resource interpreted as Stylesheet but transferred with MIME type text/plain: "http://localhost/assets/application-5574b338d88d13681ef38b9b0800bc47.css".
new:12 Resource interpreted as Script but transferred with MIME type text/plain: "http://localhost/assets/application-ea59e17aff7f15a316e3e03d49f3daf4.js".
If you want nginx to serve your static assets, you'll need to set config.serve_static_files = false so Rails doesn't try to do it for you.

Unicorn+Nginx concurrency and data duplication

I have 4 Nginx workers and 4 unicorn workers. We hit a concurrency issue in some of our models that validate unique names. We are getting duplicated names when we send multiple requests at the same time on the same resource. For instance if we send around 10 requests to create Licenses we get duplicated serial_numbers...
Here's some context:
Model (simplified)
class License < ActiveRecord::Base
validates :serial_number, :uniqueness => true
end
Unicorn.rb
APP_PATH = '.../manager'
worker_processes 4
working_directory APP_PATH # available in 0.94.0+
listen ".../manager/tmp/sockets/manager_rails.sock", backlog: 1024
listen 8080, :tcp_nopush => true # uncomment to listen to TCP port as well
timeout 600
pid "#{APP_PATH}/tmp/pids/unicorn.pid"
stderr_path "#{APP_PATH}/log/unicorn.stderr.log"
stdout_path "#{APP_PATH}/log/unicorn.stdout.log"
preload_app true
GC.copy_on_write_friendly = true if GC.respond_to?(:copy_on_write_friendly=)
check_client_connection false
run_once = true
before_fork do |server, worker|
ActiveRecord::Base.connection.disconnect! if defined?(ActiveRecord::Base)
MESSAGE_QUEUE.close
end
after_fork do |server, worker|
ActiveRecord::Base.establish_connection if defined?(ActiveRecord::Base)
end
Nginx.conf (simplified)
worker_processes 4;
events {
multi_accept off;
worker_connections 1024;
use epoll;
accept_mutex off;
}
upstream app_server {
server unix:/home/blueserver/symphony/manager/tmp/sockets/manager_rails_write.sock fail_timeout=0;
}
try_files $uri #app;
location #app {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
proxy_pass http://app_server;
}
Every time I send multiple requests (more than 4+) to create Licenses I get some duplicates. I understand why. It's because each unicorn process doesn't have a resource with the serial_number created yet. So, it allow to create it multiple times...
ActiveRecord is validating the uniqueness of the field at the process level rather than a database level. One workaround could be moving the validations to the database (but it will be very cumbersome and hard to maintain).
Another workaround is to limit the write requests (POST/PUT/DELETE) to only one unicorn and have multiple unicorns to reply to read requests (GET). Something like this in the location in Nginx...
# 4 unicorn workers for GET requests
proxy_pass http://app_read_server;
# 1 unicorn worker for POST/PUT/DELETE requests
limit_except GET {
proxy_pass http://app_write_server;
}
we are currently using that. It fixes the concurrency issue. However, one write server is not enough to reply at peak times and its creating a bottleneck.
Any idea to solve the concurrency and scalability issues with Nginx+Unicorn?
Take a look at transaction isolation. For example, PostgreSQL - http://www.postgresql.org/docs/current/static/transaction-iso.html.
Normally, you can go two ways:
use unique index for unique key column(via migration) and catch appropriate exception;
maintain the database constraints in a way described here and catch appropriate exception as well.
or use PostureSQL transaction with isolation level "serialised" which is basically transforms parallel translations into consecutive ones as it was described by Andrey Kryachkov early.

rails fallback to assets pipeline

I'm trying to deploy a rails4 (ruby-2.0.0) app to my server. Almost all of my assets are precompiled, and served by nginx.
One js.erb, generates a dynamic html-list, by getting models from my database. This asset can't be precompiled, because it must remain dynamic.
I'm excluding this asset from asset.precompile, and turned on
config.assets.compile = true
to fall back to the asset pipeline, for this one asset.
In my local production env, everthing is working, but on my server (nginx, unicorn) the asset pipeline fall back won't work. I get a simple 404 Error
nginx error log:
2013/09/13 08:54:54 [error] 27442#0: *58 open() "/XXX/current/public/assets/rails_admin/rails_admin_switchable-051203ae1d7aca2c08092e5c92bcdf15.js" failed (2: No such file or directory), client: XXX, server: , request: "GET /assets/rails_admin/rails_admin_switchable-051203ae1d7aca2c08092e5c92bcdf15.js HTTP/1.1", host: "XXX", referrer: "http://XXX/admin"
unicorn and rails don't show any errors.
Any ideas, how I can solve this?
best,
Franz
It looks like your nginx server definition isn't properly integrated with your app server. It should be configured to pass a request that doesn't match a physical file on to the app server.
Here is a standard configuration for a rails app living in /app with nginx via a unicorn/UNIX-socket integration:
upstream app_server {
server unix:/tmp/nginx.socket fail_timeout=0;
}
server {
listen <%= ENV["PORT"] %>;
server_name _;
keepalive_timeout 5;
# path for static files
root /app/public;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
# Rails error pages
error_page 500 502 503 504 /500.html;
location = /500.html {
root /app/public;
}
}
If your asset pipeline compiles to /app/public/assets you should be good to go.

unicorn request queuing

We just migrated from passenger to unicorn to host few rails apps.
Everything works great but we notice via New Relic that request are queuing between 100 and 300ms.
Here's the graph :
I have no idea where this is coming from here's our unicorn conf :
current_path = '/data/actor/current'
shared_path = '/data/actor/shared'
shared_bundler_gems_path = "/data/actor/shared/bundled_gems"
working_directory '/data/actor/current/'
worker_processes 6
listen '/var/run/engineyard/unicorn_actor.sock', :backlog => 1024
timeout 60
pid "/var/run/engineyard/unicorn_actor.pid"
logger Logger.new("log/unicorn.log")
stderr_path "log/unicorn.stderr.log"
stdout_path "log/unicorn.stdout.log"
preload_app true
if GC.respond_to?(:copy_on_write_friendly=)
GC.copy_on_write_friendly = true
end
before_fork do |server, worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
end
old_pid = "#{server.config[:pid]}.oldbin"
if File.exists?(old_pid) && server.pid != old_pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :TERM : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
sleep 1
end
if defined?(Bundler.settings)
before_exec do |server|
paths = (ENV["PATH"] || "").split(File::PATH_SEPARATOR)
paths.unshift "#{shared_bundler_gems_path}/bin"
ENV["PATH"] = paths.uniq.join(File::PATH_SEPARATOR)
ENV['GEM_HOME'] = ENV['GEM_PATH'] = shared_bundler_gems_path
ENV['BUNDLE_GEMFILE'] = "#{current_path}/Gemfile"
end
end
after_fork do |server, worker|
worker_pid = File.join(File.dirname(server.config[:pid]), "unicorn_worker_actor_#{worker.nr$
File.open(worker_pid, "w") { |f| f.puts Process.pid }
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
end
end
our nginx.conf :
user deploy deploy;
worker_processes 6;
worker_rlimit_nofile 10240;
pid /var/run/nginx.pid;
events {
worker_connections 8192;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
sendfile on;
tcp_nopush on;
server_names_hash_bucket_size 128;
if_modified_since before;
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_buffers 16 8k;
gzip_types application/json text/plain text/html text/css application/x-javascript t$
# gzip_disable "MSIE [1-6]\.(?!.*SV1)";
# Allow custom settings to be added to the http block
include /etc/nginx/http-custom.conf;
include /etc/nginx/stack.conf;
include /etc/nginx/servers/*.conf;
}
and our app specific nginx conf :
upstream upstream_actor_ssl {
server unix:/var/run/engineyard/unicorn_actor.sock fail_timeout=0;
}
server {
listen 443;
server_name letitcast.com;
ssl on;
ssl_certificate /etc/nginx/ssl/letitcast.crt;
ssl_certificate_key /etc/nginx/ssl/letitcast.key;
ssl_session_cache shared:SSL:10m;
client_max_body_size 100M;
root /data/actor/current/public;
access_log /var/log/engineyard/nginx/actor.access.log main;
error_log /var/log/engineyard/nginx/actor.error.log notice;
location #app_actor {
include /etc/nginx/common/proxy.conf;
proxy_pass http://upstream_actor_ssl;
}
include /etc/nginx/servers/actor/custom.conf;
include /etc/nginx/servers/actor/custom.ssl.conf;
if ($request_filename ~* \.(css|jpg|gif|png)$) {
break;
}
location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}
error_page 404 /404.html;
error_page 500 502 504 /500.html;
error_page 503 /system/maintenance.html;
location = /system/maintenance.html { }
location / {
if (-f $document_root/system/maintenance.html) { return 503; }
try_files $uri $uri/index.html $uri.html #app_actor;
}
include /etc/nginx/servers/actor/custom.locations.conf;
}
We are not under heavy load so I don't understand why requests are stuck in the queue.
As specified in the unicorn conf, we have 6 unicorn workers.
Any idea where this could be coming from ?
Cheers
EDIT:
Average requests per minute: about 15 most of the time, more than 300 in peeks but we didn't experienced one since the migration.
CPU Load average: 0.2-0.3
I tried with 8 workers, it didn't change anything.
I've also used raindrops to look what unicorn workers were up to.
Here's the ruby script :
#!/usr/bin/ruby
# this is used to show or watch the number of active and queued
# connections on any listener socket from the command line
require 'raindrops'
require 'optparse'
require 'ipaddr'
usage = "Usage: #$0 [-d delay] ADDR..."
ARGV.size > 0 or abort usage
delay = false
# "normal" exits when driven on the command-line
trap(:INT) { exit 130 }
trap(:PIPE) { exit 0 }
opts = OptionParser.new('', 24, ' ') do |opts|
opts.banner = usage
opts.on('-d', '--delay=delay') { |nr| delay = nr.to_i }
opts.parse! ARGV
end
socks = []
ARGV.each do |f|
if !File.exists?(f)
puts "#{f} not found"
next
end
if !File.socket?(f)
puts "#{f} ain't a socket"
next
end
socks << f
end
fmt = "% -50s % 10u % 10u\n"
printf fmt.tr('u','s'), *%w(address active queued)
begin
stats = Raindrops::Linux.unix_listener_stats(socks)
stats.each do |addr,stats|
if stats.queued.to_i > 0
printf fmt, addr, stats.active, stats.queued
end
end
end while delay && sleep(delay)
How i've launched it :
./linux-tcp-listener-stats.rb -d 0.1 /var/run/engineyard/unicorn_actor.sock
So it basically check every 1/10s if there are requests in the queue and if there are it outputs :
the socket | the number of requests being processed | the number of requests in the queue
Here's a gist of the result :
https://gist.github.com/f9c9e5209fbbfc611cb1
EDIT2:
I tried to reduce the number of nginx workers to one last night but it didn't change anything.
For information we are hosted on Engine Yard and have a High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each)
We host 4 rails applications, this one has 6 workers, we have one with 4, one with 2 and another with one. They're all experiencing request queuing since we migrated to unicorn.
I don't know if Passenger was cheating but New Relic didn't log any request queuing when we were using it. We also have a node.js app handling file uploads, a mysql database and 2 redis.
EDIT 3:
We're using ruby 1.9.2p290, nginx 1.0.10, unicorn 4.2.1 and newrelic_rpm 3.3.3.
I'll try without newrelic tomorrow and will let you know the results here but for the information we were using passenger with new relic, the same version of ruby and nginx and didnt have any issue.
EDIT 4:
I tried to increase the client_body_buffer_size and proxy_buffers with
client_body_buffer_size 256k;
proxy_buffers 8 256k;
But it didn't do the trick.
EDIT 5:
We finally figured it out ... drumroll ...
The winner was our SSL cypher. When we changed it to RC4 we saw the request queuing droppping from 100-300ms to 30-100ms.
I've just diagnosed a similar looking New relic graph as being entirely the fault of SSL. Try turning it off. We are seeing 400ms request queuing time, which drops to 20ms without SSL.
Some interesting points on why some SSL providers might be slow: http://blog.cloudflare.com/how-cloudflare-is-making-ssl-fast
What version of ruby, unicorn, nginx (shouldn't matter much but worth mentioning) and newrelic_rpm are you using?
Also, I would try running a baseline perf test without newrelic. NewRelic parses the response and there are cases where this can be slow due to the issue with 'rindex' in ruby pre-1.9.3. This is usually only noticeable when you're response is very large and doesn't contain 'body' tags (e.g. AJAX, JSON, etc). I saw an example of this where a 1MB AJAX response was taking 30 seconds for NewRelic to parse.
Are you sure that you are buffering the requests from the clients in nginx and then buffering the responses from the unicorns before sending them back to the clients. From your setup it seems that you do (because this is by default), but I will suggest you double check that.
The config to look at is:
http://wiki.nginx.org/HttpProxyModule#proxy_buffering
This is for the buffering of the response from the unicorns. You definitely need it because you don't want to keep unicorns busy sending data to a slow client.
For the buffering of the request from the client I think that you need to look at:
http://wiki.nginx.org/HttpCoreModule#client_body_buffer_size
I think all this can't explain a delay of 100ms, but I am not familiar with all of your system setup, so it is worth it to have a look at this direction. It seems that your queuing is not caused by a CPU contention, but by some kind of IO blocking.

Thin + Nginx + Upload Module + Upload Progress Module

I'm using Nginx as a reverse proxy for Thin instances.
My goal is to set up a Rails (3) app to upload large files and do something with them.
For that, I came across the Nginx Upload and Upload Progress modules.
I was reading, for the most part, this post, but that's specifically wrote thinking in Passenger.
If possible, I'm looking for two possible answers:
1) Information an examples of implementing this stack (with Thin instead of Passenger)
2) Specific Information of how could I rewrite this:
location ^~ /progress {
# report uploads tracked in the 'proxied' zone
upload_progress_json_output;
report_uploads proxied;
}
location #fast_upload_endpoint {
passenger_enabled on;
rails_env development;
}
location / {
rails_env development;
passenger_enabled on;
}
I don't know what is Passenger exclusive, and how to write it for a typical 4 workers / 3 thin instances conf.
Thanks.
First, you should install nginx with the upload module. The nginx config for site:
upstream uploader_cluster {
server unix:/tmp/thin.uploader.0.sock;
server unix:/tmp/thin.uploader.1.sock;
server unix:/tmp/thin.uploader.2.sock;
server unix:/tmp/thin.uploader.3.sock;
server unix:/tmp/thin.uploader.4.sock;
}
server {
listen 80;
server_name ***.com;
charset off;
client_max_body_size 1000m;
access_log /var/www/uploader/log/access.log;
error_log /var/www/uploader/log/error.log;
root /var/www/uploader/public;
index index.html;
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (!-f $request_filename) {
proxy_pass http://uploader_cluster;
break;
}
}
location ~*uploads$ {
if ($request_method = GET) {
proxy_pass http://uploader_cluster;
break;
}
# pass request body to here
upload_pass #upload_photos;
# Store files to this directory
# The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
# i.e. make sure to create /vol/uploads/0 /vol/uploads/1 etc.
upload_store /vol/uploads 1;
# set permissions on the uploaded files
upload_store_access user:rw group:rw all:r;
# Set specified fields in request body
# this puts the original filename, new path+filename and content type in the requests params
upload_set_form_field upload[file_name] "$upload_file_name";
upload_set_form_field upload[file_content_type] "$upload_content_type";
upload_set_form_field upload[file_path] "$upload_tmp_path";
upload_aggregate_form_field upload[file_size] "$upload_file_size";
upload_pass_form_field "^fb_sig_user$|^aid$|^batch_id$|^album_name$|^album_visible$|^caption$|^tags$|^complete$";
upload_cleanup 400 404 499 500-505;
}
location #upload_photos {
proxy_pass http://uploader_cluster;
}
}

Resources