I am using DBCP connection pool and I have created a MBean to show the connection pool stats in Jconsole. What I have observed is that even if there is no activity on the application (means no page click, etc) the connection pool stats shows active connections.
The 16 NumActive connections we see above doesn't drop to zero. I am not sure why?
Can someone explain?
Thanks!
So I figured out the issue it was due to connection leakages. The connections were obtained from the pool but were never returned to the pool.
Related
I have a web application running on an App Service on Azure cloud.
On the back-end I'm using a tcp connection to our database (Neo4j graph db), the best practice is to open the tcp connection and keep it alive in order to be more reactive when we perform queries.
The issue I encountered is that the database is logging the exception "Connection reset by peer";
reading on the web I found out that maybe Azure has a TCP timeout configured by default, I read it to be set up to 4 minutes, which could be my issue root cause.
Someone knows how to configure the tcp KEEP ALIVE to always for App Services on Azure?
I found on the web how to do it in Google cloud but nothing about Azure cloud.
Thank you in advance.
OaicStef
From everything I can find that is not an adjustable setting. Here is the forum link that says it will not be changing and that is a couple years old at this point. https://social.msdn.microsoft.com/Forums/en-US/32b76114-67a4-4e6b-ac45-61b0f0a0829f/changing-the-4-minute-request-time-out-for-app-services?forum=windowsazurewebsitespreview
I think you are going to have to add logic to your app that tests the connection, if it has been closed then either reopen it or create a new one. I don't know what language you are using to make any suggestions there.
Edit
I will add that the total number of TCP connections that can be open on a single App Service is about 6k, at least on the S1. Keep that in mind because if you don't have pooling on the server side or you are not disposing of those then you will exhaust that the TCP pool and you will start getting errors. I recommend you configure an alert for that.
I don't understand the application of expiryTimeout field in Activemq PooledConnectionFactory. The java doc said "allow connections to expire, irrespective of load or idle time. This is useful with failover to force a reconnect from the pool, to reestablish load balancing or use of the master post recovery". please give me an example, a real scenario which expiryTimeout field effect in it.
The expiry timeout option is a bit of a legacy feature of the Pool that isn't all that useful in most applications these days. The way it works is that if you configure an expiration time then the Connection that is loaned out and is later closed will be completely closed and dropped should there be no other active users of the Connection, otherwise it stays alive until all active instances are closed, then the underlying Connection object is closed.
This works slightly differently than the Idle timeout which applies to Connection instances that are sitting unused in the pool and are closed after some length of time to release resources on the Broker side.
These days you are better off using a failover URI in the PooledConnectionFactory with broker support for rebalance of cluster clients enabled which would then dynamically redistribute the load in the broker cluster as opposed to the expiry timeout which only closes down Connection instances once everyone that is currently actively using them has released them by calling close on them.
If a connection is 'inactive' I guess the Weblogic internal data source manager should recover the connection. Why should 'Inactive Connection Timeout' be a configurable parameter. Is there any use case which requires WL to wait for certain period before an inactive connection is recovered?
Thanks in advance.
This variation could occur depending on what downstream systems are doing, or where a system is sometimes under load and not able to respond in an adequate period of time
A leaked connection is a connection that was not properly returned to the connection pool in the data source. To automatically recover leaked connections, you can specify a value for Inactive Connection Timeout on the JDBC Data Source. ( Configuration: Connection Pool page in the Administration Console.) When you set a value for Inactive Connection Timeout, WebLogic Server forcibly returns a connection to the data source when there is no activity on a reserved connection for the number of seconds that you specify. When set to 0 (the default value), this feature is turned off.
After digging into this.. specific to question "Why should it be configurable..", because by default this feature is turned off, you can turn it on if you believe that the application for some reason is not returning connections back to the pool, i.e.
leaking connections. Depending on the application use cases and the amount of resource leakage, one could set the duration. IOW, you don't want to timeout connections too quickly, because it will force the app server to have to recreate those on next need or cause unnecessary errors. But if the application is leaking a lot of connections, then you can tighten the duration so it keeps cleaning up.
We have our application deployed to WAS 6 cluster. And recently it is throring following exception.
javax.resource.ResourceException: The back-end resource is currently unavailable. Stuck connections have been detected.
......
Can somebody explain me why db connection was not released by the app and came back to free pool? How can I detect what is blocking connection to be released? I am planning to take thread dump every fee secs.
Everything was working fine and all of a sudden we started getting this exception, which is causing an issue with new user who is trying to login into the app.
Any input will be greatly appreciated. I have very little knowledge about WAS admin.
Thanks
Try using the PMI within the WAS console under Monitoring and Tuning, this will allow you to trace both the JDBC and thread pool usage in real time, I would definitely pay close attention to the WebContainer pool and see if the size of the pool tracks with the JDBC connection.
If the pools themselves are becoming exhausted you can increase the size to provide some legroom by upping the Maximum Connection Settings for the JDBC connection under Resources -> Data Sources -> $NAME -> Connection Pool, and the other connection pool settings under Server -> $SERVERNAME -> Additional Properties -> Thread Pool
Ensuring that the database your connecting to also has sufficent free connections would also be an idea! :)
If you are leaking pool connections, then its likely the code is missing a close connection somewhere.
We notice something strange in our struts web application which was hosted on sun app server enterprise edition 8.1.
The NumConnUsed for Monitoring of JDBC resources stays at 100 over connections even though there was relatively very low user activities.
I try to do some research and found the following links
http://j2ee-performance.blogspot.com/
http://www.ibm.com/developerworks/websphere/library/techarticles/0506_johnsen/0506_johnsen.html
"When the application closes a shareable connection, the connection is not truly closed, nor is it returned to the free pool. Rather, it remains in the Shared connection pool, ready for another request within the same LTC for a connection to the same resource."
Base on the above comments, it is true that if my web.xml resource ref scope is set to shareable, when application side close conneciton, it remains in the shared conneciton pool thus the numconnused is always so high?
If I interpret the links in my own special way (;)), the shared vs. unshared connections is based on different connections in the same page.
java.sql.Connection connectionOne = DriverManager.getConnection(...);
...
java.sql.Connection connectionTwo = DriverManager.getConnection(...);
These two, at a glance, seem to be individual - but if your AS is set to shareable connections, the second one will be created with a pointer to the first connection instead of returning a new connection. When the page finishes the connection should be sent back to the pool.
The AS is probably keeping the pool filled with connections to enhance performance.
This is not fact, only my own iterpretation of the links.