Rails app keeps crashing when experiencing heavy load - ruby-on-rails

Question

I was testing my app as I am anticipating a heavy load next week. I am currently using BlazeMeter to simulate the load. My server is hosted on AWS Elasticbeanstalk and RDS using m5.large EC2 instance. I have set up my application using Puma, Capistrano and Nginx. My config are as below:
//nginx.conf
upstream app {
server unix: ///home/deploy/apps/appname/shared/tmp/sockets/appname-puma.sock;
}
server {
listen 80;
server_name example.com;
root / home / deploy / apps / appname / current / public;
if ($http_x_forwarded_proto != "https") {
return 301 https: //$server_name$request_uri;
}
}
server {
listen 443 default_server http2;
root / home / deploy / apps / appname / current / public;
try_files / system / maintenance.html $uri / index.html $uri $uri.html #app;
access_log /
var / log / nginx / access.log main;
access_log /
var / log / nginx / healthd / application.log.$year - $month - $day - $hour healthd;
error_log /
var / log / nginx / error.log debug;
location / assets {
alias /
var / app / current / public / assets;
gzip_static on;
gzip on;
expires max;
add_header Cache - Control public;
}
location~ * \.(jpg | jpeg | gif | css | png | js | ico | svg | svgz) $ {
gzip_static on;
gzip on;
expires max;
add_header Cache - Control public;
}
location #app {
proxy_pass http: //app; # match the name of upstream directive which is defined above
proxy_set_header Host $http_host;
proxy_set_header X - Real - IP $remote_addr;
proxy_set_header X - Forwarded - For $proxy_add_x_forwarded_for;
proxy_set_header X - Forwarded - Proto $scheme;
proxy_set_header HTTP_CLIENT_IP $remote_addr;
}
error_page 500 502 503 504 / 500. html;
location / 500. html {}
error_page 404 / 404. html;
location / 404. html {}
error_page 422 / 422. html;
location / 422. html {}
}
##config/deploy.rb
# config valid only for current version of Capistrano
lock "3.8.2"
set :application, "appname"
set :repo_url, #"censored"
set :user, 'deploy'
set :puma_threads, [16, 64]
set :puma_workers, 2
set :pty, true
set :use_sudo, false
set :stage, :production
set :deploy_via, :remote_cache
set :deploy_to, "/home/#{fetch(:user)}/apps/#{fetch(:application)}"#{}"/var/app/current"
set :puma_bind, "unix://#{shared_path}/tmp/sockets/#{fetch(:application)}-puma.sock"
set :puma_state, "#{shared_path}/tmp/pids/puma.state"
set :puma_pid, "#{shared_path}/tmp/pids/puma.pid"
set :puma_access_log, "#{release_path}/log/puma.error.log"
set :puma_error_log, "#{release_path}/log/puma.access.log"
set :ssh_options, { forward_agent: true, user: fetch(:user), keys: %w(~/.ssh/id_rsa.pub) }
set :puma_preload_app, true
set :puma_worker_timeout, nil
set :puma_init_active_record, true # Change to false when not using ActiveRecord
append :linked_files, %w{config/database.yml}
append :linked_dirs, ".bundle", "tmp", "public/system"#, %w{bin log tmp/pids tmp/cache tmp/store vendor/bundle public/system}
namespace :puma do
desc 'Create Directories for Puma Pids and Socket'
task :make_dirs do
on roles(:app) do
execute "mkdir #{shared_path}/tmp/sockets -p"
execute "mkdir #{shared_path}/tmp/pids -p"
end
end
before :start, :make_dirs
end
namespace :deploy do
desc "Make sure local git is in sync with remote."
task :check_revision do
on roles(:app) do
unless `git rev-parse HEAD` == `git rev-parse origin/master`
puts "WARNING: HEAD is not the same as origin/master"
puts "Run `git push` to sync changes."
exit
end
end
end
desc 'Initial Deploy'
task :initial do
on roles(:app) do
before 'deploy:restart', 'puma:start'
invoke 'deploy'
end
end
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
invoke 'puma:restart'
end
end
before :starting, :check_revision
after :finishing, :compile_assets
end
There are many tests which failed and the error percentage is high. According to my logs from elasticbeanstalk,
[error] 2465#0: *2262 testing "/var/app/current/public/assets" existence failed (2: No such file or directory)
[crit] 2465#0: *2216 connect() to unix:///var/run/puma/my_app.sock failed (2: No such file or directory) while connecting to upstream
I am a newbie to this and I have no idea why this is happening! Any help is appreciated! Thanks!
Update 1: My website also shows 502 Bad Gateway Nginx after reaching a certain number of users simulated.
Update 2:
Like what Myst has pointed out, I am using a database as well.
default: &default
adapter: sqlite3
pool: 5
timeout: 5000
development:
<<: *default
database: db/development.sqlite3
test:
<<: *default
adapter: mysql2
encoding: utf8
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
production:
<<: *default
adapter: mysql2
encoding: utf8
database: <%= ENV['RDS_DB_NAME'] %>
username: <%= ENV['RDS_USERNAME'] %>
password: <%= ENV['RDS_PASSWORD'] %>
host: <%= ENV['RDS_HOSTNAME'] %>
port: <%= ENV['RDS_PORT'] %>
pool: 50 #20
timeout: 10000
Versions:
capistrano3-puma (3.1.0)
ruby (2.3.0)
rails (4.2.8)
mysql2 (0.4.9)

Answer 1

In general - when dealing with high load at first it's good to measure machine metrics - how much load can it take in terms of RPS/simultaneous clients while being stable. And during this test - LA, memory usage, IO to determine what resource is current bottleneck.
Set upstream connections limit in nginx (server unix:///... max_conns=20;), it is unlimited by default and under stress test this may cause workers to swell and get Out-of-memory errors.
Once workers are dead - nginx cannot connect to socket and reports error 502.
See if your puma_error_log (it looks that it is confused with access log in config) has exceptions.
Also since you only have 2 cpu cores - I doubt there's need for 64 threads times 2 workers, unless most of your requests result in waiting for calls to an external api.