I have an app currently on a 48GB (12 vCPU) DO server, which is struggling with connection sizes. We have an API that gets hit ever 5 minutes by 350~ IoT devices and it gets added to a background queue for log processing.
I have tried using both delayed_job and sidekiq for background processing, however both create so many database connections that it exceeds the postgresql default 100 connection limit and causes downtime. I have had to temporarily remove this background processing. Without the background processing, we use around 60-65 postgresql connections.
I am running Dokku 0.6.5
Postgresql was installed using this dokku plugin
App is Ruby on
Rails
My database.yml pool size is set to 5, my puma worker threads are 5 so I followed herokus recommendation that these should match.t
I am not conclusive in how PgBouncer can be installed on a dokku app that is running postgresql through a plugin. Currently, my only idea is to upgrade my dokku to 0.15.0, which then has buildpack support and I can install the heroku PgBouncer buildpack.
I would greatly appreciate any insight on the best way to install PgBouncer with this set-up.
Related
Hard issue happening in production for a long time, we have no clue about where it's coming from. Can sometimes reproduces it on localhost, Heroku Enterprise support has been clue-less about this.
On our production database, we currently have the following setup:
Passenger Standalone, threading disabled, limited to 25 processes MAX. No min setup.
3 web dynos
a SELECT * FROM pg_stat_activity GROUP BY client_addr and count the number of connections per instance shows that more than 1 PSQL connection is opened for one passenger process during our peak days.
Assumptions:
A single address is about a single Dyno (Confirmed by Heroku staff)
Passenger does not spawn more than 25 processes at the time (confirmed with passenger-status during those peaks)
Here is a screenshot of what looks the SELECT * FROM pg_stat_activity;:
In the screenshot, we can see that there are 45 psql connections coming from the same dyno that runs passenger. If we followed our previous logic, it should not have more than 1 connection per Passenger process, so 25.
The logs doesn't look unusual, nothing mentioning either a dyno crash / process crash.
Here is a screenshot of our passenger status for the same dyno (different time, just to prove that there are not more processes than 25 created for one dyno):
And finally one of the response we got from the Heroku support (Amazing support btw)
I have also seen previous reports of Passenger utilising more connections than expected, but most were closed due to difficulty reproducing, unfortunately.
In the Passenger documentation, it's explained that Passenger handle itself the ActiveRecord connections.
Any leads appreciated. Thanks!
Various information:
Ruby Version: 2.4.x
Rails Version: 5.1.x
Passenger Version: 5.3.x
PG Version: 10.x
ActiveRecord Version: 5.1.x
If you need any more info, just let me know in the comments, I will happily update this post.
One last thing: We use ActionCable. I've read somewhere that passenger is handling weirdly the socket connections (Opens a somewhat hidden process to keep the connection alive). This is one of our leads, but so far no luck in reproducing it on localhost. If anyone can confirm how Passenger handles ActionCable connections, it would be much appreciated.
Update 1 (01/10/2018):
Experimented:
Disable NewRelic Auto-Explain feature as explained here: https://devcenter.heroku.com/articles/forked-pg-connections#disabling-new-relic-explain
Run locally a Passenger server with min and max pool size set to 3 (more makes my computer burn), then kill process with various signals (SIGKILL, SIGTERM) to try to see if connections are closed properly. They are.
We finally managed to fix the issue we had on Passenger. We have had this issue for a very long time actually.
The fix
If you use ActionCable, and your default cable route is /cable, then change the Procfile from:
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE
to
web: bundle exec passenger start -p $PORT --max-pool-size $PASSENGER_MAX_POOL_SIZE --unlimited-concurrency-path /cable
Explanation
Before the change, each socket connection (ActionCable) would take one single process in Passenger.
But a Socket is actually something that should not take a whole process. A process can handle many many open socket connection. (Many is more than 10thousands at the same time for some big names). Fortunately, we have much lower socket connections, but still.
After the change, we basically told Passenger to not take a whole process to handle one socket connection, but rather dedicate a whole process to handle all the socket connections.
Documentation
The in-depth documentation on how to do Sockets with Passenger: https://www.phusionpassenger.com/library/config/standalone/tuning_sse_and_websockets/
The flag to pass to Passenger: https://www.phusionpassenger.com/library/config/standalone/reference/#--unlimited-concurrency-path-unlimited_concurrency_paths
Some metrics, after 3 weeks with the fix
Number of forked processes on Passenger dramatically decreased (from 75 processes to ~ 15 processes)
Global memory usage on the web dynos dramatically decreased (related to previous point on forked Passenger processes)
The global number of PSQL connections dramatically decreased and has been steady for two days (even after deployment). (from 150 to ~30 connections)
Number of PSQL connections per dyno dramatically decreased, (from ~50 per dyno to less than 10 per dyno)
The number of Redis connections decreased and has been steady for two days (even after deployment)
Average memory usage on PostgreSQL dramatically decreased and has been steady for two days.
The overall throughput is a bit higher than usual (Throughput is the number of requests handled per minute)
Is it possible to hot deploy Ruby just like PHP?
Normally I used FTP to upload the PHP file, then it will be available automatically.
Can Ruby hot deploy its file like this?
Your comment welcome.
Are you talking about a ruby on rails application ?
If so, when deploying a rails application in production mode, the all application gets loaded in memory. So changing the files won't affect the running application.
For hot restarting a rails application you will need to use solution such as:
Unicorn
Puma
Passenger
For a first time, Puma is the easiest way.
However if you are looking for a zero-downtime, either Unicorn or Passenger enterprise are what you are looking for.
EDIT
Unicorn
Free
Complex configuration
zero-downtime when hot restarting. when hot-restarting unicorn, it keeps the old threads working until the new ones are fully functionnal. So if the new ones fail to start, nothing happens. The old ones just keep going.
Puma
Free
Simple configuration
hot restart but no zero-downtime. When hot-restarting puma, it shuts down the old threads and starts the new ones. Puma keeps the sockets open, so the client are not disconnected, but are waiting to get a response while the new threads restart. However if the new threads fail to start, Puma can't restart the old ones. So connections are lost and the server is down.
Passenger
Free edition
Free
The configuration is easier than unicorn
hot-restart, but no zero-downtime. Like Puma.
Enterprise edition
$29/mo
The configuration is easier than unicorn
zero-downtime when hot restarting. Like Unicorn.
We're running a rails app via heroku that connects to a windows azure VM, where I've set up a redis master/slave to act as a cache (slash quick reference data store). The problem is, we sporadically get redis timeouts. This is with a timeout of 10 seconds (which I know is more than it needs), and reestablishing redis connections on fork. And using hiredis as a driver.
Anyone have a clue why this might be happening? I know heroku and the azure vm are hosted on different coasts, so there's a bit of latency; could there be TCP request drops? I'm fairly out of ideas.
I've been struggling with this error for quite some time:
Redis::ProtocolError: Got 'i' as initial reply byte.
If you're running in a multi-threaded environment, make sure you pass the :thread_safe
option when initializing the connection. If you're in a forking environment, such as
Unicorn, you need to connect to Redis after forking.
It happens intermittently in apps that use Unicorn and Redis. From this redis-rb Github issue it looks the :thread_safe option is now enabled by default. I am using redis 2.2.2 because redis 3.0.1 is not compatible with the latest version of resque.
In my Unicorn config I'm using Redis.current.quit after the fork.
I'm also connecting to Redis with a gem called ruote-redis which is a storage implementation for the workflow engine Ruote.
How can I ensure that all of my Redis connections are stable and that I don't get this error anymore which is disrupting to the normal use of our app?
Unicorn is not multi-threaded. Are you using threads yourself?
As stated in the docs, the problem you're hitting is that multiple Unicorn workers are sharing the same connection (ie the same underlying file descriptor).
This change, included in version redis-rb 3.0, makes it even clearer.
If you're still hitting this error please post your Unicorn configuration.
My Rails app currently runs on Passenger. We have six machines in production with each machine having 45 instances of passenger. Each instance of Passenger seems to be having a open connection with MySQL. So there are about 270 open connections with MySQL.
A few minutes after the application is started MySQL seems to at 400% CPU and the application almost becomes non-responsive with a lot of pending requests in the global queue.
Prior to this release the application was running on Mongrel (six machines with each machine running 5 instances of mongrel). We did not see any issues there.
Will reducing the number of Passenger instances solve the issue? Is there a way we can have a connection pool for the passenger instances (Instead of each instance having a separate connection).
Thanks,
Sivakumar
Passenger shares db connections among all processes. Make sure your database.yml has a high pool size.
passenger-hosted-rails-app-painfully