nginx + passenger + rails 3.1 = 502 bad gateway? - ruby-on-rails

Question

I have the latest Nginx running with Passenger, SQLite and Rails 3.1. Somehow, when I have Passenger running for a while, I start getting "502 bad gateway" errors when visiting my website.
Here is a snippet from my Nginx error log:
2011/06/27 08:55:33 [error] 20331#0: *11270 upstream prematurely closed connection while reading response header from upstream, client: xxx.xxx.xx.x, server: www.example.com, request: "GET / HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "example.com"
2011/06/27 08:55:47 [info] 20331#0: *11273 client closed prematurely connection, so upstream connection is closed too while sending request to upstream, client: xxx.xxx.xx.x, server: www.example.com, request: "GET / HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "example.com"
Here is my passenger-status --show=backtraces output:
Thread 'Client thread 7':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 10':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 11':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 12':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 13':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 14':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 15':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 16':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 17':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 18':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 19':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 20':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 21':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 22':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 23':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'Client thread 24':
in 'Passenger::FileDescriptor Client::acceptConnection()' (HelperAgent.cpp:160)
in 'void Client::threadMain()' (HelperAgent.cpp:603)
Thread 'MessageServer thread':
in 'void Passenger::MessageServer::mainLoop()' (MessageServer.h:537)
Thread 'MessageServer client thread 35':
in 'virtual bool Passenger::BacktracesServer::processMessage(Passenger::MessageServer::CommonClientContext&, boost::shared_ptr<Passenger::MessageServer::ClientContext>&, const std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)' (BacktracesServer.h:47)
in 'void Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor&)' (MessageServer.h:470)
This is what my passenger-memory-stats shows:
---------- Nginx processes ----------
PID PPID VMSize Private Name
-------------------------------------
16291 1 35.4 MB 0.1 MB nginx: master process /home/apps/.nginx/sbin/nginx
16292 16291 36.0 MB 0.8 MB nginx: worker process
16293 16291 35.8 MB 0.5 MB nginx: worker process
16294 16291 35.8 MB 0.5 MB nginx: worker process
16295 16291 35.8 MB 0.5 MB nginx: worker process
### Processes: 5
### Total private dirty RSS: 2.46 MB
----- Passenger processes ------
PID VMSize Private Name
--------------------------------
16251 87.0 MB 0.3 MB PassengerWatchdog
16254 100.4 MB 1.3 MB PassengerHelperAgent
16256 41.6 MB 5.7 MB Passenger spawn server
16259 134.8 MB 0.8 MB PassengerLoggingAgent
18390 770.4 MB 17.1 MB Passenger ApplicationSpawner: /home/apps/manager/current
18415 853.3 MB 147.7 MB Rack: /home/apps/manager/current
18424 790.5 MB 57.2 MB Rack: /home/apps/manager/current
18431 774.7 MB 18.7 MB Rack: /home/apps/manager/current
### Processes: 8
### Total private dirty RSS: 248.85 MB
It seems there is an issue with my the communication between Passenger and Nginx?
Also, looking at the Rails logs, it is clear that the request never reaches Rails at all, as there are no log entries for visits that get the 502 error. So my initial thought of something being wrong with any Rack middleware should not be possible.

Answer 1

The "V" in VM is for Virtual. See also answers on other SO questions, e.g. Virtual Memory Usage from Java under Linux, too much memory used.
That top 147 MB does not hint of anything unusual whatsoever. Your 502 errors mean something else is wrong with the worker processes from Passenger's point of view. You should check your Rails & Nginx log files for clues, and perhaps passenger-status --show=backtraces.

Answer 2

Try setting passenger_spawn_method conservative -- apparently there are issues with Passenger default forking settings and Rails 3.1

Answer 3

I just meet such deadly "502 Bad Gateway error" reported by nginx, web stack is Ubuntu 12.04 + Rails 3.2.9 + Passenger 3.0.18 + nginx 1.2.4, it spent me 2 hours to found the root cause:
My rails application no need database support, so I just remove the gem 'sqlite3' in the Gemfile, it works fine in development mode, but will lead 502 Bad Gateway in production mode.
So after add back gem 'sqlite3' in Gemfile, such 502 Bad Gateway error disappear....

Answer 4

I had the same problem and in my case it helped to increase the passenger_max_pool_size setting in the Nginx configuration file.
Maybe you can also take a look on the following postings which also helped me finding this solution:
http://whowish-programming.blogspot.com/2011/10/nginx-502-bad-gateway.html
https://groups.google.com/forum/?fromgroups#!topic/phusion-passenger/fgj6_4VdlLo and
https://groups.google.com/forum/?fromgroups#!topic/phusion-passenger/wAQDCrFHHgE

Answer 5

It was the same for me in Rails 4, but I have added a "SECRETKEYBASE" in /confirg/secrets.yml
production:
secretkeybase: # add yours here