We are currently using Pgbouncer(installed on database server) for database connection pooling. At the same time we use Npgsql Library which has its own connection pool. I have read recommendations that we should disable pooling in Npgsql and use only Pgbouncer.
There is a performance problem when we disable connection pooling in Npgsql.
According to my test, it takes 100 ms to connect to pgbouncer. Latency to server with PgBouncer is <1ms.
Executing 5 queries with 5 connections will take more than 500ms, which is too much.
Are we using it correct? That connection latency is killing my performance.
I tried to connect to pgbouncer from different server in network and it took from 8 to 22 ms. I am assuming, it is some network issue.
There is no reason to disable connection pooling in Npgsql unless you have errors or compatibility issues.
PGBouncer helps with scalability by handling many more concurrent connections at once without overloading Postgres (which creates a process for each new connection). This does not mean that creating new connections is any faster, so it's better to reuse an existing pool.
Npgsql would maintain a pool of connections from your application to pgbouncer, and pgbouncer would have a pool of connections from itself to Postgres. This works fine and will ensure both network hops are as efficient as possible.
Related
We are using loopback-connector-mongodb#4.2.0 version to connect to mongo db . Our database admins are reporting that our application is using more connections and it is constantly growing. When they report issues , we just restart our applications and connection use count drops drastically (from 800 active connections to 500 connections on mongodb).
We need to implement connection pooling in loopback, but not sure on what settings needs to be done on the datasource.json file. Can you please give some suggestions.
thanks
Arun S.
I'm building an advanced RoR application with several instances of database connections running concurrently. However I'm still looking for answers for the following,
Does the connection handler automatically add a connection into the pool once its established? Like the following way,
ActiveRecord::Base.establish_connection(spec)
When would it disconnect the active connection if there's no activity for a while? Or should it be manually disconnected.
Are there any other alternatives to disconnecting the database connection like the following,
ActiveRecord::Base.remove_connection(connection_object)
How can I increase the pool size which could be for different database adapters.
Looking forward to some tips.
I am running a tonne of jobs in parallel using Sidekiq and a lot of them are failing to connect to the database because I've only got a connection pool size of 5.
I'd like to just bump that up to 15 (at least on localhost) but was wondering what the possible negative consequences of that might be.
Setup is Ruby on Rails, default poolsize is 5.
It depends on many factors like:
how much memory you want to allocate to your database pool
how long your connections last
the timeout on the connections
the locality of your database server compared to your app/web server
There are other tweaks that some connection pools have also such as the minimum number of connections to have open (even if not used), and the maximum open connections which looks like what you are trying to set.
I have heard that you can potentially saturate your network card with as little as 10 open connections.
I think the only answer is to monitor your cpu/memorry/io usage based on what you have so you have some sort of a baseline, and then bump the connection count up and compare.
Personally I think you should be fine with 15 connections assuming you arent' already pushing your server to the limit or have a tiny VM with 256MB of ram :)
Setting the value too high could saturate the # of allowable open connections to postgres (check the default but it may be around 100). This could especially be a problem if you prematurely shut your services down without allowing it to gracefully close the connections. Then when you try and restart your app server it will error out saying that postgres is not allowing any more connections. This isn't a problem of setting it too high as this would happen in either case but it would def. accelerate the issue.
Can some tell me how to use connection pooling with MonetDb? Do I just append these args to the URL "poolminimum=1;poolmaximum=5;"?
No, by itself, the driver does not pool connections. You could use c3p0 (http://www.mchange.com/projects/c3p0/) to create a connection pool for MonetDB JDBC connections. However, please be aware that MonetDB uses optimistic concurrency control and table-level locking, so firing updates at a single table from multiple connections will have an increasing risk of transaction aborts. For reading however, this should be fine.
So I have a connection pool setup. Which is great and all since I have an application that really needs it. However what I would like to know is if it is possible to share this connection pool with other J2SE apps? Would this even be worth it, as opposed to creating a connection pool based on each apps needs? If it would be prudent, how can I accomplish this?
It is not hard having connection pools in a single JVM doing multiple things - that is what applications servers do everyday (using JNDI to throw objects across classloaders)
The interesting part is when you have the connection pool in a separate JVM from the client code needing it, as this does not immediately allow simply asking for and getting a connection from the pool and returning it afterwards.
Basically you have two options:
Doing remote requests for all your JDBC commands over the network. This will most likely mean that the data will travel over the network twice, from the database to the connection pool, and then from the connection pool to your application. If the database connections are very expensive objects then this might be a viable solution.
Use RMI to get the connection object from the connection pool JVM to your own machine. This is a very expensive operation, but can as far as I know include the actual driver classes, allowing your connection pool to provide connections to databases not known to your application JVM. To me this would only make sense if the database connections were ridiculoulusly expensive or it was a requirement to be able to support additional databases after deployment without changing the original deployments.
Note that the primary reason for having connection pools at all is because connections are expensive to create, use shortly and then discard. Some databases more than others, e.g. MySQl is (or was when I tried) very cheap so it might be the simplest just to do that.
So. First of all: Measure what your connection pool buys you in time, and then consider if it is worth your while to centralize this further.