Is there a way to increase the connection timeout for ActiveRecord?
I keep getting this error when I have more than 25 threads with a pool size of 5.
(ActiveRecord::ConnectionTimeoutError) "could not obtain a database connection within 5 seconds (waited 5.000144774 seconds). The max pool size is currently 3; consider increasing it."
If there is not a way to increase the connection timeout, what is the best way to ensure a thread is using a connection as quick as possible?
According to docs you should set the "checkout_timeout" option in your database configuration file.
checkout_timeout: number of seconds to block and wait for a connection
before giving up and raising a timeout error (default 5 seconds).
You can add
pool: 5
in your database.yml.
you can also set the checkout_timeout value, but I do not recomend it, because your application might take more seconds to answer it. If the error is throw when the system is under many requests, it is probably better to just give more possible simultaneous connections instead of making each request wait longer to finish.
Related
Background:
On production we have a poorly understood error that occurs sporadically (more frequently by the week) and may take down our whole application at times – or sometimes just all of our background processes. Unfortunately, I am not certain what causes the issue, below is my working theory – could you please validate its logic?
The error preceding the downtime (occurring a couple of hundred times in matters of seconds) is the PostgreSQL error FATAL: sorry, too many clients already.
Working theory:
Various parts of an API can request connections with the database. In our Ruby on Rails application for example, we have 12 puma workers with 16 threads (12 * 16 = 192 possible db connections). Also, we have 10 background workers, each being allowed a single db connection. If we also account for a single SSH session with 1 database connection, the maximum amount of db connections we would have to anticipate is 192 + 10 + 1 = 203 PostgreSQL connections, set with the max_connections in the postgresql.conf config file.
Our max_connections however is still set to the PostgreSQL default of 100. My understanding is that this is problematic: when the application thinks more db connections are possible (looking at the application side settings for puma and our background workers) it allows for new db connections to be made. But when those connections with PostgreSQL are initiated, PostgreSQL looks at its own set maximum of 100 connections and breaks the connection.
When instead the amount of "requestable" connections (in this case 203) would either be lower than or equal to the PostgreSQL max_connections, it would utilise the pool timeout to queue to requested db connection until a db socket becomes available.
This is desirable since too many connections could be resolved within the pool timeout. Thus the solution to our problem is to make the "requestable" database connections =< possible database connections. If that is still not enough, I should increase the 100 possible connections.
Does this make sense...?
Any ideas or criticism would be very much appreciated!
Your app threads does not need to map 1-1 to database connections. You can use a connection pool for the database connections. See https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html
There is also lots of good info on this subject at https://devcenter.heroku.com/articles/concurrency-and-database-connections
I'm getting this error sporadically on my Prod server.
ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds
I see there is not high CPU usage for DB, but still this error happened once a day maybe twice.
Puma.rb
threads 2, 100
workers 2
database.yml
pool: 15
Ruby
ruby:2.3
Puma
puma (3.11.2)
DB size
db.m5.large
In your current configuration, each puma worker has its own connection pool with 15 available database connections. And each worker is allowed to scale between 2 and 100 threads depending on the server load.
This means when the load goes up or there are more long-running requests then your server will run create more than 15 threads and at that point, your database connection pool will be empty and the new threads have to wait for other threads to return database connections. This might take a while and after 5 seconds of waiting you will observe an ActiveRecord::ConnectionTimeoutError exception.
To solve this issue you have to make sure that the database connection pool is big enough for all your threads. But at the same time, you have to make sure that the total number of all connections in all pools – in the workers plus in sidekiq and other tools (database management tools or Rails consoles) – is below the maximum number of connections available by your database.
My advice is: First, figure out the maximum number of connections in your database. You might find this information in your database's config or in the docs of your database provider. Then split that number overall workers and tools like Sidekiq. Once you know the max number of connections per worker set the max number of thread to that number.
Example: Imaging your database supports 64 connections. Then you might want to run two servers with 2 workers each, a Sidekiq instance with 4 workers and you want to have a buffer to be able to connect a Rails console or a backup system to the database too.
2 servers with 2 workers 48
8 sidekiq workers 8
buffer 8
With those numbers, I would set the connection pool in Rails' database config to 12 and would set the number of threads in the puma config to 2, 12
Read more about this topic at Concurrency and Database Connections in Ruby with ActiveRecord in the Heroku devcenter.
I am reading DBConnection documentation. And I don't quite understand following quote:
Our goal is to wait at most :queue_target for a connection.
If all connections checked out during a :queue_interval takes more than
:queue_target, then we double the :queue_target. If checking out
connections take longer than the new target, then we start dropping
messages.
Could you please explain me on examples?
In my app I have very huge operation that is executed by periodic worker. I would like to have timeout for it 1minute, or don't have timeout at all. Which queue_target and queue_interval should I set to avoid: Elixir.DBConnection.ConnectionError',message => <<"tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)"
In regular case I would like me queue timeout to be 5 seconds. How could I achieve this with queue_target and queue_interval?
The timeouts you're referring to are set with the :timeout option in execution functions (i.e. execute/4), :queue_target and :queue_interval are only meant to affect the pool's ability to begin new requests (for requests to checkout connections from the pool), not requests that have already checked out connections and are already being processed.
Keep in mind that all attempts to checkout connections during a :queue_interval must take longer than :queue_target in order for these values to affect anything. Normally you'd test different values and monitor your database's ability to keep up in order to find optimal values for your environment.
I understand from reading HikariCP's documentation (see below) that idle connections should be retired from a connection pool.
My question is: why and when should an idle database connection be retired from the connection pool?
This is the part of HikariCP documentation that sparked my question:
idleTimeout:
This property controls the maximum amount of time (in milliseconds)
that a connection is allowed to sit idle in the pool. Whether a
connection is retired as idle or not is subject to a maximum variation
of +30 seconds, and average variation of +15 seconds. A connection
will never be retired as idle before this timeout. A value of 0 means
that idle connections are never removed from the pool. Default: 600000
(10 minutes)
Two main reasons:
a) they take up resources on the server (not terribly much since the connection is idle)
b) sometimes connections timeout themselves after periods of inactivity. You want to either close them before that, or run some periodic "ping" SQL to make sure they are still alive. Otherwise you'd get an error on the next SQL you want to execute.
I am using SessionStore to store my jruby Rails sessions in a postgres database. The connection pool configuration is:
pool: 10
timeout: 5000
In my testing, I found that with around 15 connections, I start getting the following error:
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 10; consider increasing it.)
It seems like this is failing too early. Surely 10 DB connections can support more than 15 concurrent users? I am only using this database to access sessions, so I assume that each request only uses the connection for a small period of time. Are these failures happening at a reasonable time?
How does Rails use DB connections for session store? Does it hold the DB connection for the length of the request or does it simply load it once at beginning of each request? Any ideas about what could be causing this poor DB connection usage?