Heavy load web service because of blocked Phusion Passenger queues - ruby-on-rails

Question

We are developing a Web service using Ruby 2 on Rails 4, Mongoid 4, MongoDB 2.6. It uses Sidekiq 3.3.0 and Redis 2.8 and is running on Phusion Passenger 5.0.4 + Nginx 1.7.10. It only serves mobile clients & AngularJS web client via JSON APIs.
Normally everything works fine, APIs are processed and responded under 1s. But in rush hours, the service is heavy load (APIs are rendered as 503 Service Unavailable). Below are our Nginx & Mongoid config:
Nginx config
passenger_root /home/deployer/.rvm/gems/ruby-2.1.3/gems/passenger-4.0.53;
#passenger_ruby /usr/bin/ruby;
passenger_max_pool_size 70;
passenger_min_instances 1;
passenger_max_requests 20; # A workaround if apps are mem-leaking
passenger_pool_idle_time 300;
passenger_max_instances_per_app 30;
passenger_pre_start http://production_domain/;
## Note: there're 2 apps with the same config
server {
listen 80;
server_name production_domain;
passenger_enabled on;
root /home/deployer/app_name-production/current/public;
more_set_headers 'Access-Control-Allow-Origin: *'
more_set_headers 'Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE, HEAD';
more_set_headers 'Access-Control-Allow-Headers: DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
if ($request_method = 'OPTIONS') {
# more_set_headers 'Access-Control-Allow-Origin: *';
# add_header 'Access-Control-Allow-Origin' '*';
# add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
# add_header 'Access-Control-Allow-Headers' 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-FooA$
# add_header 'Access-Control-Max-Age' 1728000;
# add_header 'Content-Type' 'text/plain charset=UTF-8';
# add_header 'Content-Length' 0;
return 200;
}
access_log /var/log/nginx/app_name-production.access.log;
error_log /var/log/nginx/app_name-production.error.log;
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /etc/nginx/html/;
}
rails_env production;
}
Mongoid config
development:
sessions:
default:
another:
uri: mongodb://127.0.0.1:27017/database_name
test:
sessions:
default:
another:
uri: mongodb://127.0.0.1:27017/database_name
options:
pool_size: 10
pool_timeout: 15
retry_interval: 1
max_retries: 30
refresh_interval: 10
timeout: 15
staging:
sessions:
default:
another:
uri: mongodb://staging_domain/staging_database
options:
pool_size: 10
pool_timeout: 15
retry_interval: 1
max_retries: 30
refresh_interval: 10
timeout: 15
production:
sessions:
default:
another:
uri: mongodb://production_domain/production_database
options:
pool_size: 30
pool_timeout: 15
retry_interval: 1
max_retries: 30
refresh_interval: 10
timeout: 15
Sidekiq config
and Passenger logs when heavy load:
Version : 5.0.4
Date : 2015-04-04 09:31:14 +0700
Instance: MxPcaaBy (nginx/1.7.10 Phusion_Passenger/5.0.4)
----------- General information -----------
Max pool size : 120
Processes : 62
Requests in top-level queue : 0
----------- Application groups -----------
/home/deployer/memo_rails-staging/current/public (staging)#default:
App root: /home/deployer/memo_rails-staging/current
Requests in queue: 0
* PID: 20453 Sessions: 0 Processed: 639 Uptime: 14h 34m 26s
CPU: 0% Memory : 184M Last used: 14s ago
* PID: 402 Sessions: 0 Processed: 5 Uptime: 13h 0m 42s
CPU: 0% Memory : 171M Last used: 23m 35s
* PID: 16081 Sessions: 0 Processed: 3 Uptime: 10h 26m 9s
CPU: 0% Memory : 163M Last used: 24m 9s a
* PID: 30300 Sessions: 0 Processed: 1 Uptime: 4h 19m 43s
CPU: 0% Memory : 164M Last used: 24m 15s
/home/deployer/memo_rails-production/current/public (production)#default:
App root: /home/deployer/memo_rails-production/current
Requests in queue: 150
* PID: 25924 Sessions: 1 Processed: 841 Uptime: 20m 49s
CPU: 3% Memory : 182M Last used: 7m 58s ago
* PID: 25935 Sessions: 1 Processed: 498 Uptime: 20m 49s
CPU: 2% Memory : 199M Last used: 5m 40s ago
* PID: 25948 Sessions: 1 Processed: 322 Uptime: 20m 49s
CPU: 1% Memory : 200M Last used: 7m 57s ago
* PID: 25960 Sessions: 1 Processed: 177 Uptime: 20m 49s
CPU: 0% Memory : 158M Last used: 19s ago
* PID: 25972 Sessions: 1 Processed: 115 Uptime: 20m 48s
CPU: 0% Memory : 151M Last used: 7m 56s ago
* PID: 25987 Sessions: 1 Processed: 98 Uptime: 20m 48s
CPU: 0% Memory : 179M Last used: 7m 56s ago
* PID: 25998 Sessions: 1 Processed: 77 Uptime: 20m 48s
CPU: 0% Memory : 145M Last used: 7m 2s ago
* PID: 26012 Sessions: 1 Processed: 97 Uptime: 20m 48s
CPU: 0% Memory : 167M Last used: 19s ago
* PID: 26024 Sessions: 1 Processed: 42 Uptime: 20m 47s
CPU: 0% Memory : 148M Last used: 7m 55s ago
* PID: 26038 Sessions: 1 Processed: 44 Uptime: 20m 47s
CPU: 0% Memory : 164M Last used: 1m 0s ago
* PID: 26050 Sessions: 1 Processed: 29 Uptime: 20m 47s
CPU: 0% Memory : 142M Last used: 7m 54s ago
* PID: 26063 Sessions: 1 Processed: 41 Uptime: 20m 47s
CPU: 0% Memory : 168M Last used: 1m 1s ago
* PID: 26075 Sessions: 1 Processed: 23 Uptime: 20m 47s
CPU: 0% Memory : 126M Last used: 7m 51s ago
* PID: 26087 Sessions: 1 Processed: 19 Uptime: 20m 46s
CPU: 0% Memory : 120M Last used: 7m 50s ago
* PID: 26099 Sessions: 1 Processed: 37 Uptime: 20m 46s
CPU: 0% Memory : 131M Last used: 7m 3s ago
* PID: 26111 Sessions: 1 Processed: 20 Uptime: 20m 46s
CPU: 0% Memory : 110M Last used: 7m 49s ago
* PID: 26126 Sessions: 1 Processed: 28 Uptime: 20m 46s
CPU: 0% Memory : 172M Last used: 1m 56s ago
* PID: 26141 Sessions: 1 Processed: 20 Uptime: 20m 45s
CPU: 0% Memory : 107M Last used: 7m 19s ago
* PID: 26229 Sessions: 1 Processed: 20 Uptime: 20m 21s
CPU: 0% Memory : 110M Last used: 11s ago
* PID: 26241 Sessions: 1 Processed: 9 Uptime: 20m 21s
CPU: 0% Memory : 105M Last used: 7m 47s ago
* PID: 26548 Sessions: 1 Processed: 23 Uptime: 19m 14s
CPU: 0% Memory : 125M Last used: 7m 44s ago
* PID: 27465 Sessions: 1 Processed: 30 Uptime: 15m 23s
CPU: 0% Memory : 109M Last used: 2m 22s ago
* PID: 27501 Sessions: 1 Processed: 28 Uptime: 15m 18s
CPU: 0% Memory : 117M Last used: 7m 15s ago
* PID: 27511 Sessions: 1 Processed: 34 Uptime: 15m 17s
CPU: 0% Memory : 144M Last used: 5m 40s ago
* PID: 27522 Sessions: 1 Processed: 30 Uptime: 15m 17s
CPU: 0% Memory : 110M Last used: 26s ago
* PID: 27533 Sessions: 1 Processed: 38 Uptime: 15m 17s
CPU: 0% Memory : 110M Last used:"4m 44s ago
* PID: 27555 Sessions: 1 Processed: 27 Uptime: 15m 15s
CPU: 0% Memory : 120M Last used: 1m 29s ago
* PID: 27570 Sessions: 1 Processed: 21 Uptime: 15m 14s
CPU: 0% Memory : 107M Last used: 7m 1s ago
* PID: 27590 Sessions: 1 Processed: 8 Uptime: 15m 13s
CPU: 0% Memory : 105M Last used: 7m 34s ago
* PID: 27599 Sessions: 1 Processed: 13 Uptime: 15m 13s
CPU: 0% Memory : 107M Last used: 7m 0s ago
* PID: 27617 Sessions: 1 Processed: 26 Uptime: 15m 12s
CPU: 0% Memory : 114M Last used: 4m 49s ago
* PID: 27633 Sessions: 1 Processed: 19 Uptime: 15m 11s
CPU: 0% Memory : 137M Last used: 1m 14s ago
* PID: 27643 Sessions: 1 Processed: 15 Uptime: 15m 11s
CPU: 0% Memory : 132M Last used: 6m 19s ago
* PID: 27661 Sessions: 1 Processed: 23 Uptime: 15m 10s
CPU: 0% Memory : 112M Last used: 9s ago
* PID: 27678 Sessions: 1 Processed: 24 Uptime: 15m 9s
CPU: 0% Memory : 108M Last used: 6m 53s ago
* PID: 27692 Sessions: 1 Processed: 9 Uptime: 15m 9s
CPU: 0% Memory : 105M Last used: 7m 22s ago
* PID: 28400 Sessions: 1 Processed: 19 Uptime: 12m 45s
CPU: 0% Memory : 111M Last used: 1m 25s ago
* PID: 28415 Sessions: 1 Processed: 26 Uptime: 12m 45s
CPU: 0% Memory : 149M Last used: 3m 45s ago
* PID: 28439 Sessions: 1 Processed: 14 Uptime: 12m 44s
CPU: 0% Memory : 106M Last used: 59s ago
* PID: 28477 Sessions: 1 Processed: 12 Uptime: 12m 42s
CPU: 0% Memory : 108M Last used: 1m 34s ago
* PID: 28495 Sessions: 1 Processed: 14 Uptime: 12m 41s
CPU: 0% Memory : 108M Last used: 18s ago
* PID: 29315 Sessions: 1 Processed: 7 Uptime: 10m 1s
CPU: 0% Memory : 107M Last used: 7m 0s ago
* PID: 29332 Sessions: 1 Processed: 13 Uptime: 10m 0s
CPU: 0% Memory : 108M Last used: 5m 39s ago
* PID: 29341 Sessions: 1 Processed: 7 Uptime: 10m 0s
CPU: 0% Memory : 105M Last used: 6m 53s ago
* PID: 29353 Sessions: 1 Processed: 11 Uptime: 10m 0s
CPU: 0% Memory : 119M Last used: 5m 4s ago
* PID: 29366 Sessions: 1 Processed: 16 Uptime: 9m 59s
CPU: 0% Memory : 119M Last used: 3m 13s ago
* PID: 29377 Sessions: 1 Processed: 10 Uptime: 9m 59s
CPU: 0% Memory : 113M Last used: 1m 34s ago
* PID: 29388 Sessions: 1 Processed: 2 Uptime: 9m 59s
CPU: 0% Memory : 97M Last used: 7m 28s ago
* PID: 29400 Sessions: 1 Processed: 6 Uptime: 9m 59s
CPU: 0% Memory : 103M Last used: 6m 53s ago
* PID: 29422 Sessions: 1 Processed: 17 Uptime: 9m 58s
CPU: 0% Memory : 132M Last used: 1m 24s ago
* PID: 29438 Sessions: 1 Processed: 1 Uptime: 9m 57s
CPU: 0% Memory : 96M Last used: 6m 52s ago
* PID: 29451 Sessions: 1 Processed: 21 Uptime: 9m 56s
CPU: 0% Memory : 133M Last used: 2m 10s ago
* PID: 29463 Sessions: 1 Processed: 19 Uptime: 9m 56s
CPU: 0% Memory : 111M Last used: 27s ago
* PID: 29477 Sessions: 1 Processed: 23 Uptime: 9m 56s
CPU: 0% Memory : 117M Last used: 14s ago
* PID: 30625 Sessions: 1 Processed: 7 Uptime: 6m 49s
CPU: 0% Memory : 106M Last used: 1m 21s ago
* PID: 30668 Sessions: 1 Processed: 2 Uptime: 6m 44s
CPU: 0% Memory : 105M Last used: 1m 13s ago
* PID: 30706 Sessions: 1 Processed: 16 Uptime: 6m 43s
CPU: 0% Memory : 148M Last used: 1m 11s ago
* PID: 30718 Sessions: 1 Processed: 12 Uptime: 6m 43s
CPU: 0% Memory : 112M Last used: 1m 16s ago
I have some questions:
It seems to be that someone with slow internet connection is requesting our service, leading to Passenger processes is blocked. We have to restart Nginx to get the web service works again. Anyone has experiences with this?
We also use Sidekiq as worker queues. Most of our Workers are implemented without hitting MongoDB. They work fine.
But there are 2 Workers we use to update User's data, which query, update & insert data into the database. We've tried to optimize these all tasks using MongoDB bulk commands (update & insert).
Normally, when a small amount of users request the web service, the Workers work fine, busy queues are processed in about 1 minutes, but when it receives more requests, busy queues block the whole system. We have to restart the Nginx, again, to get it works. Below are Sidekiq config:
development:
:concurrency: 5
:logfile: ./log/sidekiq_development.log
:pidfile: ./log/sidekiq.pid
staging:
:concurrency: 5
:logfile: ./log/sidekiq_staging.log
:pidfile: ./log/sidekiq.pid
production:
:concurrency: 15
:logfile: ./log/sidekiq_production.log
:pidfile: ./log/sidekiq.pid
:queues:
- ...
We don't have any experiences with those problems. Anyone has any ideas?
Update 1:
After some monitorings when server is heavy load, we got this result: the MongoDB processes have many faults & stacked read queues, below are what mongostat logged during the downtime:
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
*0 2 *0 *0 0 4|0 0 79g 160g 3.36g 137 memo_v2:2.6% 0 17|0 8|0 36k 8k 61 15:05:22
*0 6 *0 *0 0 1|0 0 79g 160g 3.38g 144 memo_v2:2.1% 0 30|0 3|0 722b 11k 61 15:05:23
1595 15 1 *0 0 5|0 0 79g 160g 3.41g 139 memo_v2:19.7% 0 20|0 8|0 164k 179k 61 15:05:25
1 18 2 *0 1 6|0 0 79g 160g 3.38g 198 memo_v2:14.4% 0 31|0 1|0 3k 122k 61 15:05:26
2 20 4 *0 0 7|0 0 79g 160g 3.38g 169 memo_v2:8.6% 0 29|0 1|0 3k 157k 61 15:05:27
1 6 23 *0 0 4|0 0 79g 160g 3.39g 190 memo_v2:18.7% 0 32|0 1|0 1k 63k 61 15:05:28
1 4 42 *0 0 4|0 0 79g 160g 3.1g 115 memo_v2:35.9% 0 30|0 0|1 1k 20k 61 15:05:29
1 5 51 *0 0 4|0 0 79g 160g 3.11g 177 memo_v2:30.0% 0 28|0 1|0 1k 23k 61 15:05:30
*0 6 20 *0 0 2|0 0 79g 160g 3.12g 174 memo_v2:40.9% 0 28|0 1|0 15k 7k 61 15:05:31
2 9 *0 *0 1 7|0 0 79g 160g 3.1g 236 memo_v2:4.4% 0 26|0 2|0 2k 31k 61 15:05:32
Anyone faced this before?

Answer 1

I don't have enough reputation to post a comment, so I'll have to add a very lacklustre answer.
I don't have any experience with the stack, but if you are correct in that slow clients are the cause of the passenger issues then I'd suggest that you ensure there is adequate buffering in front of the passenger processes.
For nginx, the important setting looks to be proxy_buffers. The section titled "Using Buffers to Free Up Backend Servers" in following article talks about the nginx module: https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching
For the mongodb issue, it sounds like you just need to dig a little deeper. If you can find where the issue is happening in the code, the solution will probably present itself. The article linked by Hongli looks very good for that.