I am using web2py which creates a connection pool and retains it till something else knocks them off. When conducting performance tests I can see the PostgreSQL connection going up and once the test is over, immediately dropping to 1 or 2. Since the pool_size is set at 10 on web2py, why would this be happening? It has to be something on PostgreSQL server. which setting?
Related
Background:
On production we have a poorly understood error that occurs sporadically (more frequently by the week) and may take down our whole application at times – or sometimes just all of our background processes. Unfortunately, I am not certain what causes the issue, below is my working theory – could you please validate its logic?
The error preceding the downtime (occurring a couple of hundred times in matters of seconds) is the PostgreSQL error FATAL: sorry, too many clients already.
Working theory:
Various parts of an API can request connections with the database. In our Ruby on Rails application for example, we have 12 puma workers with 16 threads (12 * 16 = 192 possible db connections). Also, we have 10 background workers, each being allowed a single db connection. If we also account for a single SSH session with 1 database connection, the maximum amount of db connections we would have to anticipate is 192 + 10 + 1 = 203 PostgreSQL connections, set with the max_connections in the postgresql.conf config file.
Our max_connections however is still set to the PostgreSQL default of 100. My understanding is that this is problematic: when the application thinks more db connections are possible (looking at the application side settings for puma and our background workers) it allows for new db connections to be made. But when those connections with PostgreSQL are initiated, PostgreSQL looks at its own set maximum of 100 connections and breaks the connection.
When instead the amount of "requestable" connections (in this case 203) would either be lower than or equal to the PostgreSQL max_connections, it would utilise the pool timeout to queue to requested db connection until a db socket becomes available.
This is desirable since too many connections could be resolved within the pool timeout. Thus the solution to our problem is to make the "requestable" database connections =< possible database connections. If that is still not enough, I should increase the 100 possible connections.
Does this make sense...?
Any ideas or criticism would be very much appreciated!
Your app threads does not need to map 1-1 to database connections. You can use a connection pool for the database connections. See https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
There is also lots of good info on this subject at https://devcenter.heroku.com/articles/concurrency-and-database-connections
We are unable to scale the frequency of our crons as we've would've liked and the thing holding us back is the number of database connection issues.
We have a primary server which has the master db, and 3 slaves. We run sidekiq on all our machines.
Our postgresql.conf -: max_connections = 200
Our pool option is also set at pool: 200 on all our rails app - database.yml in our servers.
We are running 2 sidekiq processes on each of our servers
In the green machine, if we change our concurrency from 6 to 7, we start getting a steam of errors -: Sidekiq - could not obtain a database connection within 5.042 seconds. Where am I messing up? :-(
Could it be something else inside our app? The numbers just don't add up.
Also does the number of active record connections have any association with pg_stat_activity?
Thanks in advance
Just figured it out
We're using replication, and in shards.yml, we had not set the pool size :-(
It was picking up 5 by default.
We have recently been having issues with postgres running out of connection slots, and after a lot of debugging and shrugging of shoulders we have pretty much tracked it down to the fact that we understood Connection pools wrong.
We use Rails, Postgres and Unicorn, and Delayed Job
Are we correct to assume that the connection pool is process specific, i.e each process has its own 10 (our connection pool limit) connections to the db in the pool?
And If there are no threads anywhere in the app, are we correct to assume that for the most part each process will use 1 connection, since noone ever needs a second one?
Based on these assumptions we tracked it down to the number of processes
Web server - 4x unicorn
Delayed job 3x server - 30 processes = 90 connections
That's 94 connections, and a couple connections for rails:consoles and a couple of rails runner or rake tasks would explain why we were hitting the limit often right? It has been particularly often this week after I converted a ruby script into a rails runner script.
We are planning to increase the max from 100 -> 200 or 250 to relieve this but is there a trivial way to implement inter process connection pooling in rails?
You probably want to take a look at pgbouncer. It's a purpose-built PostgreSQL connection pooler. There are some notes on the wiki too. It's packaged for most linux distros too.
Whenever I have an application using ActiveRecord I get this ConnectionTimeoutError - but always after a certain unknown period of time
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 30; consider increasing it.):
It was previously set to 5, we have already increased it, and there is no way it can be using 30 connections at the same time. The only thing we use ActiveRecord for is our session store.
Our database.yml file looks like:
development:
adapter: sqlite3
database: db/development.sqlite3
pool: 30
timeout: 5000
(Test and production settings are the same)
I have been googling this occurrence, and just came across this posting:
https://groups.google.com/forum/#!msg/copenhagen-ruby-user-group/GEHgi_WudmM/gnCiwWqmVfMJ
Which mentions that ActiveRecord does not check a connection back into the pool once it is done with it?? Is that true? Do I need to manually manage the connections?
I appreciate any advice!!
edit I should probably mention I am running Rails 3.1.3
Rails has a middleware called ActiveRecord::ConnectionAdapters::ConnectionManagement which clears active connections every request so they do not stick around. Check your middleware to make sure you have this (which is there by default), run "rake middleware". You should not have to manage the connections manually to answer your last question.
Run this in your console
ActiveRecord::Base.clear_active_connections!
I used this code on my Sinatra app
after do
ActiveRecord::Base.clear_active_connections!
end
This solve my problem
Applies also to Rails 5, since Puma is default server.
If you are using Threaded Servers like Puma, Phushion Passenger, they create multiple threads of the same application. Thereby making your application run faster, by concurrently executing each incoming requests.
Make sure that the pool size is equal or more than the number of threads. I was having an issue when few of my threads were giving me ActiveRecord::ConnectionTimeoutError, and the problem was vague since it occurs once in a while not very often.
I was also experiencing a similar problem with a Sinatra App, I added
after do
ActiveRecord::Base.clear_active_connections!
end
To my application controller and it solved my problem.
This construct is known as a filter and it evaluates after each request.
I'm not sure what was actually happening with the application, but I would suspect that connections weren't being closed after each request.
What are the benefits to using an external connection pool?
I've heard that most other applications will open up a connection for each unit of work. In Rails, for example, I'd take that to mean that each request could open a new connection. I'm assuming a connection pool would make that possible.
The only benefit I can think of is that it allows you to have 1,000 frontend processes without having 1,000 postgres processes running.
Are there any other benefits?
Rails has connection pooling built in:
Simply use ActiveRecord::Base.connection as with Active Record 2.1 and earlier (pre-connection-pooling). Eventually, when you’re done with the connection(s) and wish it to be returned to the pool, you call ActiveRecord::Base.clear_active_connections!. This will be the default behavior for Active Record when used in conjunction with Action Pack’s request handling cycle.
Manually check out a connection from the pool with ActiveRecord::Base.connection_pool.checkout. You are responsible for returning this connection to the pool when finished by calling ActiveRecord::Base.connection_pool.checkin(connection).
Use ActiveRecord::Base.connection_pool.with_connection(&block), which obtains a connection, yields it as the sole argument to the block, and returns it to the pool after the block completes.
This has been available since version 2.2. You'll see a pool parameter in your database.yml for controlling it:
pool: number indicating size of connection pool (default 5)
I don't think there would be much point in layering another pooling system underneath it and it could even confuse AR's pooling if you tried it.