Question
What is the ideal puma config in development mode for optimizing performance?
Background
Our rails app runs annoyingly slow in development mode; the server will frequently freeze up and have to be manually terminated using kill -9 [process number], and refreshing the app in the browser on code change can take a long time. We are a json api on rails 6 using puma, and our puma config is as follows. We also use active_admin, which I mention because it is the only part of our app that really uses the asset pipeline and is the slowest part of our app to do development on (refreshing admin app after code change takes the longest).
threads_count = ENV.fetch('RAILS_MAX_THREADS') { 5 }
threads threads_count, threads_count
# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch('PORT') { 3000 }
# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch('RAILS_ENV') { 'development' }
We have done puma -w 5 to increase the number of workers, which in theory should increase throughput and the potential number of db connections (correct me if this is wrong), but it hasn't had a noticeable impact on performance of the app. From what I've read doing puma -w 5 is also not a great idea during development because it means every time you change some code each worker has to reload the entire app, which would mean it is better to just run one worker locally so the app only has to reload one time on code change. It's possible that this slowness does not have to do with our puma config and that it is perhaps another issue, but regardless I would love to hear thoughts on how to best configure puma to run optimally in development mode.
Thanks in advance for the help.
Related
Hard issue happening in production for a long time, we have no clue about where it's coming from. Can sometimes reproduces it on localhost, Heroku Enterprise support has been clue-less about this.
On our production database, we currently have the following setup:
Passenger Standalone, threading disabled, limited to 25 processes MAX. No min setup.
3 web dynos
a SELECT * FROM pg_stat_activity GROUP BY client_addr and count the number of connections per instance shows that more than 1 PSQL connection is opened for one passenger process during our peak days.
Assumptions:
A single address is about a single Dyno (Confirmed by Heroku staff)
Passenger does not spawn more than 25 processes at the time (confirmed with passenger-status during those peaks)
Here is a screenshot of what looks the SELECT * FROM pg_stat_activity;:
In the screenshot, we can see that there are 45 psql connections coming from the same dyno that runs passenger. If we followed our previous logic, it should not have more than 1 connection per Passenger process, so 25.
The logs doesn't look unusual, nothing mentioning either a dyno crash / process crash.
Here is a screenshot of our passenger status for the same dyno (different time, just to prove that there are not more processes than 25 created for one dyno):
And finally one of the response we got from the Heroku support (Amazing support btw)
I have also seen previous reports of Passenger utilising more connections than expected, but most were closed due to difficulty reproducing, unfortunately.
In the Passenger documentation, it's explained that Passenger handle itself the ActiveRecord connections.
Any leads appreciated. Thanks!
Various information:
Ruby Version: 2.4.x
Rails Version: 5.1.x
Passenger Version: 5.3.x
PG Version: 10.x
ActiveRecord Version: 5.1.x
If you need any more info, just let me know in the comments, I will happily update this post.
One last thing: We use ActionCable. I've read somewhere that passenger is handling weirdly the socket connections (Opens a somewhat hidden process to keep the connection alive). This is one of our leads, but so far no luck in reproducing it on localhost. If anyone can confirm how Passenger handles ActionCable connections, it would be much appreciated.
Update 1 (01/10/2018):
Experimented:
Disable NewRelic Auto-Explain feature as explained here: https://devcenter.heroku.com/articles/forked-pg-connections#disabling-new-relic-explain
Run locally a Passenger server with min and max pool size set to 3 (more makes my computer burn), then kill process with various signals (SIGKILL, SIGTERM) to try to see if connections are closed properly. They are.
We finally managed to fix the issue we had on Passenger. We have had this issue for a very long time actually.
The fix
If you use ActionCable, and your default cable route is /cable, then change the Procfile from:
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE
to
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE --unlimited-concurrency-path /cable
Explanation
Before the change, each socket connection (ActionCable) would take one single process in Passenger.
But a Socket is actually something that should not take a whole process. A process can handle many many open socket connection. (Many is more than 10thousands at the same time for some big names). Fortunately, we have much lower socket connections, but still.
After the change, we basically told Passenger to not take a whole process to handle one socket connection, but rather dedicate a whole process to handle all the socket connections.
Documentation
The in-depth documentation on how to do Sockets with Passenger: https://www.phusionpassenger.com/library/config/standalone/tuning_sse_and_websockets/
The flag to pass to Passenger: https://www.phusionpassenger.com/library/config/standalone/reference/#--unlimited-concurrency-path-unlimited_concurrency_paths
Some metrics, after 3 weeks with the fix
Number of forked processes on Passenger dramatically decreased (from 75 processes to ~ 15 processes)
Global memory usage on the web dynos dramatically decreased (related to previous point on forked Passenger processes)
The global number of PSQL connections dramatically decreased and has been steady for two days (even after deployment). (from 150 to ~30 connections)
Number of PSQL connections per dyno dramatically decreased, (from ~50 per dyno to less than 10 per dyno)
The number of Redis connections decreased and has been steady for two days (even after deployment)
Average memory usage on PostgreSQL dramatically decreased and has been steady for two days.
The overall throughput is a bit higher than usual (Throughput is the number of requests handled per minute)
I am using Nginx with Phusion Passenger with a single-threaded Rails application. Here's the catch. Within that application, I am using multi-threaded sidekiq to perfrom some background jobs. Typically in my database.yml, I would only need to set the pool value to 1. Here's an example:
default: &default
adapter: mysql2
encoding: utf8
collation: utf8_unicode_ci
pool: 1
username: username
password: password
host: localhost
The reason is because for each tcp socket connection opened, when an http request comes in through that socket, nginx will take the request and pass the information to passenger. Passenger detects its a Rails app, and it spawns a Rails instance, which converts the response to html, which is sent back to nginx, which is then passed back to the client (browser) So for each passenger instance, I will only need one database connection, with a single-threaded Rails app.
But in my sidekiq.yml, I have set concurrency to 5:
:concurrency: 5
This means for each passenger rack instance, I will have 5 concurrent threads handled by sidekiq plus the one connection for the main app, that is a total of 6 database connections for one passenger instance.
When I look at passenger-status, I notice that max_pool_size is set to 6:
----------- General information -----------
Max pool size : 6
So does that mean passenger will never spawn more than 6 Rails instances concurrently? And if that's the case, does that mean my math is correct: 6 (instances) * 6 (database connections: 5 for sidekiq and 1 for main app) = 36 (total database connections possible for my rails app to handle concurrently).
Right now my mysql database is configured to handle 151 max concurrent connections.
SHOW VARIABLES LIKE "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
I just want to make sure my math is correct regarding passenger, rails and sidekiq.
First of all, your Sidekiq processes and your web server (in your case, Passenger) are separate. Passenger's thread pool size has no effect on your Sidekiq concurrency; instead, your Sidekiq configuration specifies a separate concurrency. So, we'll consider the two separately:
Passenger
The ActiveRecord database pool value is the number of database connections that your web process will use, in total across all threads. If your Passenger server is set up in multi-process mode, then your max connections from your web processes is db pool size * passenger pool size. On the other hand, if you set it up in multi-threaded mode (which I'd recommend if possible), your max connections is just db pool size (multiplied by however many processes are running; Puma, for example, runs by default two processes with up to fifteen threads or so, so the max connections in that case would be 30).
So, if you're using multi-threaded mode, a pool size of 1 is absolutely not sufficient -- you'll want at least as big a pool as you expect to have threads. In multi-process mode, 1 might work but I doubt it's really worth straying from the default of 5, until you encounter issues.
Sidekiq
Sidekiq always runs in multi-threaded mode (you can technically run multiple processes as well, but I'll assume you aren't). So, like above, you want your connection pool to be at least as big as the number of threads. This might mean that you technically need two different values for your db pool value depending on whether the Rails env is spinning up for Passenger, or for Sidekiq -- see this issue on the Sidekiq repo or this helpful Heroku guide for more information on how to address that.
In summary
Don't forget that, aside from all the above, you may easily have multiple servers all running the same Rails app, but only one database with one connection limit. If you're running Passenger in multi-instance mode with a max of 6 processes, set your db pool size to 5, then each web server node will use up to 30 connections. If it's running a Sidekiq server, then add 5 to that. You will probably not need more than one Sidekiq server, so 4 web nodes # 30 connections + one Sidekiq process # 5 connections = 125 maximum connections, well within your MySQL connection limit.
I reviewed the Passenger documentation again, and while the answer above answers the question, I want to add a little more detail:
HTTP client via TCP sends a request to Nginx
Phusion Passenger loaded into Nginx checks if request should be handled by Passenger. If so, request is sent to Passenger Core.
Passenger core, using load balancing rules, determines which process a request should be forwarded to.
Passenger core also takes care of application spawning: if it determines that having more application processes is necessary or beneficial, then it will make that happen subject to user-configured limits: the core will never spawn more processes than a user-configured maximum.
Passenger core also has monitoring and statistics: passenger-memory-stats and passenger-status
Passenger core restarts an application process if it crashes.
UstRouter sits idle and does not consume resources if you did not configure it to send data to Union Station, a monitoring web service
Watchdog monitors Passenger Core and UstRouter. If either of them crash, they are restarted by the Watchdog.
passenger-memory-stats will verify the three aforementioned processes as well as the spawned rack apps:
------ Passenger processes ------
PID VMSize Private Name
---------------------------------
18355 419.1 MB ? Passenger watchdog
18358 1096.5 MB ? Passenger core
18363 427.2 MB ? Passenger ust-router
18700 818.9 MB 256.2 MB Passenger RubyApp: myapp_rack_rails
24783 686.9 MB 180.2 MB Passenger RubyApp: myapp_rack_rails
passenger-status shows that the max_pool_size is 6. That is, at most there will be 6 rack apps spawned by Passenger Core:
----------- General information -----------
Max pool size : 6
App groups : 2
Processes : 3
As stated in another answer, the ActiveRecord database pool value is the number of database connections that your web process will use, in total across all threads.
But since I am using the free Passenger server, which is set up in multi-process mode, then my max connections from my web processes is db pool size * passenger pool size. So since Passenger pool size is 6, and if my db pool size is 1, that is 6 * 1 = 6. That will be 6 maximum database connections.
Sidekiq always runs in multi-threaded mode.
If someone wants to use sidekiq they must configure the number of threads they want to run on or use the default (25). If they are using a database (likely) then to not hit a connection timeout error they will need to have at least as many connections in their database pool as sidekiq threads. Currently they must configure these two values in two different places, database pool in database.yml for ActiveRecord, and sidekiq connections either via command line or the sidekiq yml file. This is a problem as it is difficult to remember when you are modifying one value that you need to modify both.
I'm running my Rails application through thin on Windows OS.
thin start -e production
Since the number of users grew, now around 10 people using the app simultaneously, there are times when a same page takes a while longer to load.
Are there other configurations that I need to set when running the server on production?
I'm quite sure that it has to do with the server since the slow down happens on pages that normally loads fast.
The Thin webserver is not meant to production environment. Instead of this you should use a different webserver and application server like Nginx/Unicorn, Nginx/Passenger.
I would recommend Passenger to run your rails app as fast as possible in production mode.
The thin webserver is very fast for few requests, but if there are simultaneously requests, thin gets very slow.
The following document describes about how to deploy rails application in windows. I haven't done this personally but, believe the latest versions should allow that. Please check the below link to see how it can be done
http://weblog.rubyonrails.org/2006/5/11/deploying-rails-on-windows-servers/
I found this question before:
How to change Rails 3 server default port in develoment?
However what I really want is to change my port number for the production environment only. I am using RoR 4. It would be very nice if I could type something on production.rbin config/environments. Is there a way to do that?
The answer in brief
The rails stack has your application and then the server that runs your application (aka the "application server"). This server could be webrick (not a good idea in production), thin, gunicorn, passenger etc etc.
You should be telling that server which port to run under. You'll (likely) need to specify this outside Rails - not in config/production.rb because by the time Rails boots it's already running inside some application server.
A deeper dive with an example:
Let's use Heroku for an example, because port numbers there are essentially randomized (at least from the view of us looking in).
Heroku will pick a random port for us, then tell us through the PORT environmental variable. With Heroku you need a Procfile to tell it what services to launch, and your Procfile may look something like this:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
See here, we use -p $PORT to tell unicorn - our application server in this example - to run on some port Heroku gave us.
Back to your question:
However you kick off your application serving process in your production environment, you should tell it to specify the port number to your web server. There's a bunch of ways to kick off your application serving process at a production level: from upstart (built into Ubuntu), to supervisord to god... all these methods run commands and make sure the process stays up (an important part of production level deployment ;) )
Is it possible to hot deploy Ruby just like PHP?
Normally I used FTP to upload the PHP file, then it will be available automatically.
Can Ruby hot deploy its file like this?
Your comment welcome.
Are you talking about a ruby on rails application ?
If so, when deploying a rails application in production mode, the all application gets loaded in memory. So changing the files won't affect the running application.
For hot restarting a rails application you will need to use solution such as:
Unicorn
Puma
Passenger
For a first time, Puma is the easiest way.
However if you are looking for a zero-downtime, either Unicorn or Passenger enterprise are what you are looking for.
EDIT
Unicorn
Free
Complex configuration
zero-downtime when hot restarting. when hot-restarting unicorn, it keeps the old threads working until the new ones are fully functionnal. So if the new ones fail to start, nothing happens. The old ones just keep going.
Puma
Free
Simple configuration
hot restart but no zero-downtime. When hot-restarting puma, it shuts down the old threads and starts the new ones. Puma keeps the sockets open, so the client are not disconnected, but are waiting to get a response while the new threads restart. However if the new threads fail to start, Puma can't restart the old ones. So connections are lost and the server is down.
Passenger
Free edition
Free
The configuration is easier than unicorn
hot-restart, but no zero-downtime. Like Puma.
Enterprise edition
$29/mo
The configuration is easier than unicorn
zero-downtime when hot restarting. Like Unicorn.