My Rails app currently runs on Passenger. We have six machines in production with each machine having 45 instances of passenger. Each instance of Passenger seems to be having a open connection with MySQL. So there are about 270 open connections with MySQL.
A few minutes after the application is started MySQL seems to at 400% CPU and the application almost becomes non-responsive with a lot of pending requests in the global queue.
Prior to this release the application was running on Mongrel (six machines with each machine running 5 instances of mongrel). We did not see any issues there.
Will reducing the number of Passenger instances solve the issue? Is there a way we can have a connection pool for the passenger instances (Instead of each instance having a separate connection).
Thanks,
Sivakumar
Passenger shares db connections among all processes. Make sure your database.yml has a high pool size.
passenger-hosted-rails-app-painfully
Related
I have an app currently on a 48GB (12 vCPU) DO server, which is struggling with connection sizes. We have an API that gets hit ever 5 minutes by 350~ IoT devices and it gets added to a background queue for log processing.
I have tried using both delayed_job and sidekiq for background processing, however both create so many database connections that it exceeds the postgresql default 100 connection limit and causes downtime. I have had to temporarily remove this background processing. Without the background processing, we use around 60-65 postgresql connections.
I am running Dokku 0.6.5
Postgresql was installed using this dokku plugin
App is Ruby on
Rails
My database.yml pool size is set to 5, my puma worker threads are 5 so I followed herokus recommendation that these should match.t
I am not conclusive in how PgBouncer can be installed on a dokku app that is running postgresql through a plugin. Currently, my only idea is to upgrade my dokku to 0.15.0, which then has buildpack support and I can install the heroku PgBouncer buildpack.
I would greatly appreciate any insight on the best way to install PgBouncer with this set-up.
Hard issue happening in production for a long time, we have no clue about where it's coming from. Can sometimes reproduces it on localhost, Heroku Enterprise support has been clue-less about this.
On our production database, we currently have the following setup:
Passenger Standalone, threading disabled, limited to 25 processes MAX. No min setup.
3 web dynos
a SELECT * FROM pg_stat_activity GROUP BY client_addr and count the number of connections per instance shows that more than 1 PSQL connection is opened for one passenger process during our peak days.
Assumptions:
A single address is about a single Dyno (Confirmed by Heroku staff)
Passenger does not spawn more than 25 processes at the time (confirmed with passenger-status during those peaks)
Here is a screenshot of what looks the SELECT * FROM pg_stat_activity;:
In the screenshot, we can see that there are 45 psql connections coming from the same dyno that runs passenger. If we followed our previous logic, it should not have more than 1 connection per Passenger process, so 25.
The logs doesn't look unusual, nothing mentioning either a dyno crash / process crash.
Here is a screenshot of our passenger status for the same dyno (different time, just to prove that there are not more processes than 25 created for one dyno):
And finally one of the response we got from the Heroku support (Amazing support btw)
I have also seen previous reports of Passenger utilising more connections than expected, but most were closed due to difficulty reproducing, unfortunately.
In the Passenger documentation, it's explained that Passenger handle itself the ActiveRecord connections.
Any leads appreciated. Thanks!
Various information:
Ruby Version: 2.4.x
Rails Version: 5.1.x
Passenger Version: 5.3.x
PG Version: 10.x
ActiveRecord Version: 5.1.x
If you need any more info, just let me know in the comments, I will happily update this post.
One last thing: We use ActionCable. I've read somewhere that passenger is handling weirdly the socket connections (Opens a somewhat hidden process to keep the connection alive). This is one of our leads, but so far no luck in reproducing it on localhost. If anyone can confirm how Passenger handles ActionCable connections, it would be much appreciated.
Update 1 (01/10/2018):
Experimented:
Disable NewRelic Auto-Explain feature as explained here: https://devcenter.heroku.com/articles/forked-pg-connections#disabling-new-relic-explain
Run locally a Passenger server with min and max pool size set to 3 (more makes my computer burn), then kill process with various signals (SIGKILL, SIGTERM) to try to see if connections are closed properly. They are.
We finally managed to fix the issue we had on Passenger. We have had this issue for a very long time actually.
The fix
If you use ActionCable, and your default cable route is /cable, then change the Procfile from:
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE
to
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE --unlimited-concurrency-path /cable
Explanation
Before the change, each socket connection (ActionCable) would take one single process in Passenger.
But a Socket is actually something that should not take a whole process. A process can handle many many open socket connection. (Many is more than 10thousands at the same time for some big names). Fortunately, we have much lower socket connections, but still.
After the change, we basically told Passenger to not take a whole process to handle one socket connection, but rather dedicate a whole process to handle all the socket connections.
Documentation
The in-depth documentation on how to do Sockets with Passenger: https://www.phusionpassenger.com/library/config/standalone/tuning_sse_and_websockets/
The flag to pass to Passenger: https://www.phusionpassenger.com/library/config/standalone/reference/#--unlimited-concurrency-path-unlimited_concurrency_paths
Some metrics, after 3 weeks with the fix
Number of forked processes on Passenger dramatically decreased (from 75 processes to ~ 15 processes)
Global memory usage on the web dynos dramatically decreased (related to previous point on forked Passenger processes)
The global number of PSQL connections dramatically decreased and has been steady for two days (even after deployment). (from 150 to ~30 connections)
Number of PSQL connections per dyno dramatically decreased, (from ~50 per dyno to less than 10 per dyno)
The number of Redis connections decreased and has been steady for two days (even after deployment)
Average memory usage on PostgreSQL dramatically decreased and has been steady for two days.
The overall throughput is a bit higher than usual (Throughput is the number of requests handled per minute)
I am using Nginx with Phusion Passenger with a single-threaded Rails application. Here's the catch. Within that application, I am using multi-threaded sidekiq to perfrom some background jobs. Typically in my database.yml, I would only need to set the pool value to 1. Here's an example:
default: &default
adapter: mysql2
encoding: utf8
collation: utf8_unicode_ci
pool: 1
username: username
password: password
host: localhost
The reason is because for each tcp socket connection opened, when an http request comes in through that socket, nginx will take the request and pass the information to passenger. Passenger detects its a Rails app, and it spawns a Rails instance, which converts the response to html, which is sent back to nginx, which is then passed back to the client (browser) So for each passenger instance, I will only need one database connection, with a single-threaded Rails app.
But in my sidekiq.yml, I have set concurrency to 5:
:concurrency: 5
This means for each passenger rack instance, I will have 5 concurrent threads handled by sidekiq plus the one connection for the main app, that is a total of 6 database connections for one passenger instance.
When I look at passenger-status, I notice that max_pool_size is set to 6:
----------- General information -----------
Max pool size : 6
So does that mean passenger will never spawn more than 6 Rails instances concurrently? And if that's the case, does that mean my math is correct: 6 (instances) * 6 (database connections: 5 for sidekiq and 1 for main app) = 36 (total database connections possible for my rails app to handle concurrently).
Right now my mysql database is configured to handle 151 max concurrent connections.
SHOW VARIABLES LIKE "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
I just want to make sure my math is correct regarding passenger, rails and sidekiq.
First of all, your Sidekiq processes and your web server (in your case, Passenger) are separate. Passenger's thread pool size has no effect on your Sidekiq concurrency; instead, your Sidekiq configuration specifies a separate concurrency. So, we'll consider the two separately:
Passenger
The ActiveRecord database pool value is the number of database connections that your web process will use, in total across all threads. If your Passenger server is set up in multi-process mode, then your max connections from your web processes is db pool size * passenger pool size. On the other hand, if you set it up in multi-threaded mode (which I'd recommend if possible), your max connections is just db pool size (multiplied by however many processes are running; Puma, for example, runs by default two processes with up to fifteen threads or so, so the max connections in that case would be 30).
So, if you're using multi-threaded mode, a pool size of 1 is absolutely not sufficient -- you'll want at least as big a pool as you expect to have threads. In multi-process mode, 1 might work but I doubt it's really worth straying from the default of 5, until you encounter issues.
Sidekiq
Sidekiq always runs in multi-threaded mode (you can technically run multiple processes as well, but I'll assume you aren't). So, like above, you want your connection pool to be at least as big as the number of threads. This might mean that you technically need two different values for your db pool value depending on whether the Rails env is spinning up for Passenger, or for Sidekiq -- see this issue on the Sidekiq repo or this helpful Heroku guide for more information on how to address that.
In summary
Don't forget that, aside from all the above, you may easily have multiple servers all running the same Rails app, but only one database with one connection limit. If you're running Passenger in multi-instance mode with a max of 6 processes, set your db pool size to 5, then each web server node will use up to 30 connections. If it's running a Sidekiq server, then add 5 to that. You will probably not need more than one Sidekiq server, so 4 web nodes # 30 connections + one Sidekiq process # 5 connections = 125 maximum connections, well within your MySQL connection limit.
I reviewed the Passenger documentation again, and while the answer above answers the question, I want to add a little more detail:
HTTP client via TCP sends a request to Nginx
Phusion Passenger loaded into Nginx checks if request should be handled by Passenger. If so, request is sent to Passenger Core.
Passenger core, using load balancing rules, determines which process a request should be forwarded to.
Passenger core also takes care of application spawning: if it determines that having more application processes is necessary or beneficial, then it will make that happen subject to user-configured limits: the core will never spawn more processes than a user-configured maximum.
Passenger core also has monitoring and statistics: passenger-memory-stats and passenger-status
Passenger core restarts an application process if it crashes.
UstRouter sits idle and does not consume resources if you did not configure it to send data to Union Station, a monitoring web service
Watchdog monitors Passenger Core and UstRouter. If either of them crash, they are restarted by the Watchdog.
passenger-memory-stats will verify the three aforementioned processes as well as the spawned rack apps:
------ Passenger processes ------
PID VMSize Private Name
---------------------------------
18355 419.1 MB ? Passenger watchdog
18358 1096.5 MB ? Passenger core
18363 427.2 MB ? Passenger ust-router
18700 818.9 MB 256.2 MB Passenger RubyApp: myapp_rack_rails
24783 686.9 MB 180.2 MB Passenger RubyApp: myapp_rack_rails
passenger-status shows that the max_pool_size is 6. That is, at most there will be 6 rack apps spawned by Passenger Core:
----------- General information -----------
Max pool size : 6
App groups : 2
Processes : 3
As stated in another answer, the ActiveRecord database pool value is the number of database connections that your web process will use, in total across all threads.
But since I am using the free Passenger server, which is set up in multi-process mode, then my max connections from my web processes is db pool size * passenger pool size. So since Passenger pool size is 6, and if my db pool size is 1, that is 6 * 1 = 6. That will be 6 maximum database connections.
Sidekiq always runs in multi-threaded mode.
If someone wants to use sidekiq they must configure the number of threads they want to run on or use the default (25). If they are using a database (likely) then to not hit a connection timeout error they will need to have at least as many connections in their database pool as sidekiq threads. Currently they must configure these two values in two different places, database pool in database.yml for ActiveRecord, and sidekiq connections either via command line or the sidekiq yml file. This is a problem as it is difficult to remember when you are modifying one value that you need to modify both.
I have a server with about 30 Ruby On Rails applications.
When I (re)start 1 of the 30 apps, then all other apps are not accessible and they seem to be waiting for the 1 app to (re)start.
Even the command: passenger-status, seems to wait untill the 1 app is (re)started.
Is this normal behavior?
Or how can this be fixed?
(Virtual) Server specifications:
CPU: 3 cores x 2.2ghz
Memory: 4GB
Hardisk: 40GB
Server software:
CentOS release 6.3 (Final)
Nginx version: nginx/1.0.12
Ruby 1.9.3p125 (2012-02-16 revision 34643)[x86_64-linux]
Phusion Passenger version 3.0.18
Nginx/Passenger config:
passenger_max_pool_size 50;
passenger_min_instances 1;
passenger_max_instances_per_app 2;
I'm happy to help with more details if needed.
Update **
Installed passenger enterprise and nginx, now the apps don't bother each other anymore with starting.. so I think the problem got fixed with passenger_rolling_restarts on;
Hmm... it looks like this may be a "feature" of Passenger's open-source version. From http://phusionpassenger.com/enterprise :
In the open source version of Phusion Passenger restarting an application involves shutting down all application processes and spawnining new ones. Because starting a new process can take a long time (depending on the application), visitors may experience slow responses while the restart is in progress. With rolling restarts, Phusion Passenger Enterprise restarts your application processes in the background.
So, the options would appear to be:
1) upgrade to the enterprise edition of Passenger
2) switch to some other server.
Yuck.
Actually, two issues are at work here:
When rolling restarts are not being used, your visitors will have to wait until the restart is finished. Just as what drosboro says.
In addition, Phusion Passenger 3 locks the entire application pool when spawning the first process for an application. During this time, no requests can be handled. Subsequent process spawns are done in the background, and rolling restart spawns are also done in the background, so they don't affect requests. This locking limitation has been entirely lifted in Phusion Passenger 4 (and of course, Phusion Passenger Enterprise 4): everything has been made asynchronous.
Can anybody provide a brief explanation of the differences between mongrel and mongrel cluster?
Mongrel is a web server that can handle one request at a time. In order to handle multiple requests, you want to run multiple mongrels. A proxy server (i.e. apache) will sit in front of the servers and listen on port 80 and then relay the web requests to an available mongrel. Mongrel cluster is a gem that manages the launching of the mongrels, stopping, restarting and running it in the right environment with the right user. It abstracts the individual mongrels as workers so you don't need to worry about them (until things go wrong). All of that is managed by a configuration file usually located with the application.
Tass and Larry K are correct though. If you are looking at a new setup, think about passenger or unicorn. Both are great, unicorn is a bit more complicated and I would not recommend it to a beginner.
Mongrel cluster is multiple mongrel instances. Then the web server rotates amongst them to handle incoming calls.
But these days the cool kids tend to use Passenger (and often the related Enterprise Ruby too)
Mongrel cluster is somewhat of outdated, today you use unicorn. The github guys switched too.