I understand from reading HikariCP's documentation (see below) that idle connections should be retired from a connection pool.
My question is: why and when should an idle database connection be retired from the connection pool?
This is the part of HikariCP documentation that sparked my question:
idleTimeout:
This property controls the maximum amount of time (in milliseconds)
that a connection is allowed to sit idle in the pool. Whether a
connection is retired as idle or not is subject to a maximum variation
of +30 seconds, and average variation of +15 seconds. A connection
will never be retired as idle before this timeout. A value of 0 means
that idle connections are never removed from the pool. Default: 600000
(10 minutes)
Two main reasons:
a) they take up resources on the server (not terribly much since the connection is idle)
b) sometimes connections timeout themselves after periods of inactivity. You want to either close them before that, or run some periodic "ping" SQL to make sure they are still alive. Otherwise you'd get an error on the next SQL you want to execute.
Related
I observed that Hikari validates the connection before giving it to the requester. At that point, if the connection is found to be dead, a new connection is created.
Is there a way that idle connections are regularly validated (even if no one is asking for a new connection)?
Something similar is present in C3P0 as 'idle_test_period'.
You can define keepaliveTime to validate idle connections
"keepalive" will only occur on an idle connection. When the time arrives for a "keepalive" against a given connection, that connection will be removed from the pool, "pinged", and then returned to the pool. The 'ping' is one of either: invocation of the JDBC4 isValid() method, or execution of the connectionTestQuery.
I am reading DBConnection documentation. And I don't quite understand following quote:
Our goal is to wait at most :queue_target for a connection.
If all connections checked out during a :queue_interval takes more than
:queue_target, then we double the :queue_target. If checking out
connections take longer than the new target, then we start dropping
messages.
Could you please explain me on examples?
In my app I have very huge operation that is executed by periodic worker. I would like to have timeout for it 1minute, or don't have timeout at all. Which queue_target and queue_interval should I set to avoid: Elixir.DBConnection.ConnectionError',message => <<"tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)"
In regular case I would like me queue timeout to be 5 seconds. How could I achieve this with queue_target and queue_interval?
The timeouts you're referring to are set with the :timeout option in execution functions (i.e. execute/4), :queue_target and :queue_interval are only meant to affect the pool's ability to begin new requests (for requests to checkout connections from the pool), not requests that have already checked out connections and are already being processed.
Keep in mind that all attempts to checkout connections during a :queue_interval must take longer than :queue_target in order for these values to affect anything. Normally you'd test different values and monitor your database's ability to keep up in order to find optimal values for your environment.
We have a connection pool for an embedded derby database. We are setting
max wait time to 5 secs
max connection in pool 100
We are getting org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object very frequently. When this exception is coming, the connections owned by the application is always 1, this is evident from the logs.
The above exception states that the pool manager cannot produce a viable connection to a waiting requester and the maxWait has passed therefore triggering a timeout.
Ref: Cannot get a connection, pool error Timeout waiting for idle object in PutSQL?
There is 1 application using derby, the Derby database, and 2 other applications.
As per my understanding, the following are the main reason for not getting a connection
There is a network issue
Connection pool has been exhausted, because of connection leak
Connection pool getting exhausted, because of long-running queries
In our case, it's an embedded derby database which is local to the application.
So, network issue is ruled out. There are no long-running queries.
I am not able to figure out what is causing wait timeout. Could it be related to OS, Filesystem, server utilization going high etc?
Any help is appreciated.
Is there a way to increase the connection timeout for ActiveRecord?
I keep getting this error when I have more than 25 threads with a pool size of 5.
(ActiveRecord::ConnectionTimeoutError) "could not obtain a database connection within 5 seconds (waited 5.000144774 seconds). The max pool size is currently 3; consider increasing it."
If there is not a way to increase the connection timeout, what is the best way to ensure a thread is using a connection as quick as possible?
According to docs you should set the "checkout_timeout" option in your database configuration file.
checkout_timeout: number of seconds to block and wait for a connection
before giving up and raising a timeout error (default 5 seconds).
You can add
pool: 5
in your database.yml.
you can also set the checkout_timeout value, but I do not recomend it, because your application might take more seconds to answer it. If the error is throw when the system is under many requests, it is probably better to just give more possible simultaneous connections instead of making each request wait longer to finish.
What is the advantage and disadvantage of connection timeout=0?
And what is the use of Connection Lifetime=0?
e.g
(Database=TestDB;
port=3306;
Uid=usernameID;
Pwd=myPassword;
Server=192.168.10.1;
Pooling=false;
Connection Lifetime=0;
Connection Timeout=0)
and what is the use of Connection Pooling?
Timeout is how long you wait for a response from a request before you give up. TimeOut=0 means you will keep waiting for the connection to occur forever. Good I guess if you are connecting to a really slow server that it is normal if it takes 12 hours to respond :-). Generally a bad thing. You want to put some kind of reasonable timeout on a request, so that you can realize your target is down and move on with your life.
Connection Lifetime = how long a connection lives before it is killed and recreated. A lifetime of 0 means never kill and recreate. Normally not a bad thing, because killing and recreating a connection is slow. Through various bugs your connections may get stuck in an unstable state (like when dealing with weird 3 way transactions).. but 99% of the time it is good to keep connection lifetime as infinite.
Connection pooling is a way to deal with the fact that creating a connection is very slow. So rather than make a new connection for every request, instead have a pool of say, 10, premade connections. When you need one, you borrow one, use it, and return in. You can adjust the size of the pool to change how your app behaves. Bigger pool = more connections = more threads doing stuff at a time, but this could also overwhelm whatever you are doing.
In summary:
ConnectionTimeout=0 is bad, make it something reasonable like 30 seconds.
ConnectionLifetime=0 is okay
ConnectionPooling=disabled is bad, you will likely want to use it.
I know this is an old thread but I think it is important to point out an instance in which you may want to disable Connection Pooling or use Connection Lifetime.
In some environments (especially when using Oracle, or at least in my experience) the web application is designed so that it connects to the database using the user's credentials vs a fixed connection string located in the server's configuration file. In this case enabling connection pooling will cause the server to create a connection pool for each user accessing the website (See Pool Fragmentation). Depending on the scenario this could either be good or bad.
However, connection pooling becomes a problem when the database server is configured to kill database connections that exceed a maximum idle time due to the fact that the database server could kill connections that may still reside in the connection pool. In this scenario the Connection Lifetime may come in handy to throw away these connections since they have been closed by the server anyway.