I'm using for a web project JSF2 with Oracle Glassfish Server Open Source Edition 4.0 and Oracle Database 11g (Version 11.2.0-1.0).
The server and database are running on the same windows machine.
A connection pool managed the connections to the database.
Does anybody know why I sometimes get the following exception:
java.sql.SQLException: Connection closed
at com.sun.gjc.spi.base.ConnectionHolder.checkValidity(ConnectionHolder.java:766)
at com.sun.gjc.spi.base.ConnectionHolder.commit(ConnectionHolder.java:243)
at de.mydomain.myproject.Hl7MessageHandler.run(Hl7MessageHandler.java:123)
...
Or sometimes this one:
java.sql.SQLRecoverableException: Closed connection
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:5675)
at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:5735)
at com.sun.gjc.spi.base.ConnectionHolder.commit(ConnectionHolder.java:244)
at de.mydomain.myproject.Hl7MessageHandler.run(Hl7MessageHandler.java:123)
...
The Database Class:
public static Connection getConnection() throws NamingException, SQLException {
Context initContext = new InitialContext();
DataSource datasSource = (DataSource)initContext.lookup("jdbc/Oracle");
Connection connnection = datasSource.getConnection();
return connnection;
}
Request handling in the servlet:
public IResponseSendable<String> run(String hl7MsgString, boolean publishErrorToDB) {
// ... do something
try {
con = Database.getConnection();
} catch (NamingException | SQLException conExc) {
return generateAck(true, conExc.getMessage(), hl7MsgString);
}
try {
con.setAutoCommit(false);
process();
con.commit();
} catch (HL7Exception | SQLException pe) {
logger.error(...);
// Exceptionhandling...
try {
con.rollback();
} catch (SQLException rollbackExc) {
logger.error(...);
}
return generateAck(true, pe.getMessage(),hl7MsgString, _log);
}
finally {
try {
con.setAutoCommit(true);
con.close();
} catch (SQLException e) {
logger.error(...);
}
}
return generateAck(false, "", hl7MsgString);
}
The process-Methode:
private void process() throws HL7Exception, SQLException {
// Do something...
String sql = "BEGIN save_patient_data(?,?,?,?,?,?,?); END;";
CallableStatement stmt = (CallableStatement) con.prepareCall(sql);
stmt.setString(1, ...);
// ...
stmt.registerOutParameter(6, java.sql.Types.VARCHAR);
stmt.registerOutParameter(7, java.sql.Types.NUMERIC);
stmt.execute();
// ...
stmt.close();
// More databse stored procedure can be called ...
}
Connection Pool Settingts:
Initial and Minimum Pool Size: 10 Connections
Maximum Pool Size: 60 Connections
Pool Resize Quantity: 2 Connections
Idle Timeout: 600 Seconds
Max Wait Time: 0 Milliseconds
Validate At Most Once: 0 Seconds
Connection Leak Timeout: 10 Seconds
Connection Leak Reclaim: enabled
Statement Leak Timeout: 6 Seconds
Statement Leak Reclaim: enabled
Creation Retry Attempts: 0
Retry Interval: 10 Seconds
Connection Validation: Required
Validation Method: meta-data
The database IDLE-Timeout setting is "UNLIMITED".
Notcie:
The exception occurred either when to call "con.prepareCall(sql);" (must not be at the first time) or when I try to commit the connection or later when to try to turn autocommit on.
Does any body know the reason or what is the best way to debug the application to find it out?
Thank you.
Edit:
Maybe it's important:
I can find in the server log many warnings about connection leaks:
2014-07-28T14:49:17.961+0200|Warnung: A potential connection leak detected for connection pool OraclePool. The stack trace of the thread is provided below :
com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324)
com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:758)
com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632)
com.sun.enterprise.resource.pool.AssocWithThreadResourcePool.getUnenlistedResource(AssocWithThreadResourcePool.java:200)
com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526)
com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:360)
com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:307)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:196)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:171)
com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:166)
com.sun.gjc.spi.base.AbstractDataSource.getConnection(AbstractDataSource.java:114)
de.mydomain.myproject.utilities.Database.getConnection(Database.java:17)
...
You have connection leak reclaim enabled and the connection leak timeout is 10 seconds. This means that if you hold onto a logical connection for longer than 10 seconds, it is forcibly revoked and closed by the connection pool manager (and the physical connection is returned to the connection pool). Subsequent attempts to use the logical connection will result in a SQLException as the connection is closed.
Find out which operation takes longer than 10 seconds and try to reduce the time it takes or configure a longer connection leak timeout (10 seconds is IMHO a bit short for connection leak detection). The same BTW applies to your statement leak detection (6 seconds is also pretty short).
Related
We are using r2dbc-pool for our application, along with Jooq.
The ConnectionFactory is as follows
ConnectionFactoryOptions.Builder connectionFactoryBuilder = ConnectionFactoryOptions.builder();
connectionFactoryBuilder.option(ConnectionFactoryOptions.HOST, ....)
.option(ConnectionFactoryOptions.DRIVER, "pool")
.option(ConnectionFactoryOptions.PROTOCOL, "postgres")
.option(ConnectionFactoryOptions.DATABASE, ....)
.option(ConnectionFactoryOptions.USER, username)
.option(ConnectionFactoryOptions.PASSWORD, password);
return ConnectionFactories.get(connectionFactoryBuilder.build());
The ConnectionPoolConfiguration looks something like this
ConnectionPoolConfiguration configuration = ConnectionPoolConfiguration.builder(<connection-factory>)
.initialSize(10)
.maxCreateConnectionTime(Duration.ofSeconds(30))
.maxAcquireTime(Duration.ofSeconds(30))
.acquireRetry(3)
.maxSize(20)
.build();
We were constantly getting Connection acquisition timed out after 30000ms. We suspected that maybe the load was too much for the connection and decided to log the PoolMetrics exposed by ConnectionPool.getMetrics()
The code we had to get connection looks something like this
public Single<Connection> getConnection(ConnectionPool connectionPool) {
Optional<PoolMetrics> poolMetricsOptional = connectionPool.getMetrics();
poolMetricsOptional.ifPresent(
poolMetrics -> log.info("Connection Pool before acquiring connection: {}", poolMetrics)
);
Single<Connection> connectionSingle = Single.fromPublisher(connectionPool.create());
Optional<PoolMetrics> poolMetricsOptional = connectionPool.getMetrics();
poolMetricsOptional.ifPresent(
poolMetrics -> log.info("Connection Pool after acquiring connection: {}", poolMetrics)
);
return connectionSingle;
When we started hitting the timeout the logs looked something like this
Connection Pool before acquiring connection: Acquire Size: 1, Allocated Size: 20, Idle Size: 9, Pending Acquire Size: 0
Connection Pool after acquiring connection: Acquire Size: 1, Allocated Size: 20, Idle Size: 9, Pending Acquire Size: 0
I have two doubts:
Shouldn't acquire size + idle size be equal to allocated size?
Any idea why isn't the connection being acquired despite the idle connections?
Version details are as follows
r2dbc-pool: 0.9.1.RELEASE
r2dbc-postgresql: 0.9.0.RELEASE
I am working on dotnet core 5.0 and I am facing this error message:
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I set these parameters to my connection string
"DefaultConnection": "Data Source=.;Initial Catalog=TestDB;User Id=sa;password=admin123;Pooling=true;Max Pool Size=100;MultipleActiveResultSets=true"
This is how I deal with DB:
using (CD_DataToolContext objCDContext = new CD_DataToolContext())
{
List<CdStandardFile> standardFiles = new List<CdStandardFile>();
foreach (var item in data) {
CdStandardFile cdStandardFile = new CdStandardFile();
standardFiles.Add(cdStandardFile);
}
objCDContext.AddRange(standardFiles); objCDContext.SaveChanges();
}
I am trying to execute the following code on a 4 core machine. I have 5 threads in the pool and within the map operator, I put the executing thread to sleep for a few seconds.
I expect that the core would put the executing thread on sleep and when the next event is available, should perform the map operation on the next available thread from the thread pool, BUT is not the behavior I see.
I see that the 4 threads from the pool go on wait for 13 seconds and process the next event only after the wait is complete.
Why is the runOn() method not executing the map operator on the next available thread from the pool when the threads go to wait state?
I am using reactor-core version '3.0.7.RELEASE'
CountDownLatch latch = new CountDownLatch(10);
ExecutorService executorService = Executors.newFixedThreadPool(5);
Flux<Integer> flux = Flux.just(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
flux.parallel()
.runOn(Schedulers.fromExecutorService(executorService))
.map(l -> {
Logger.log(ReactorParallelTest.class, "map1", "inside run waiting for 13 seconds");
try {
Thread.sleep(13000);
} catch (InterruptedException e) {
e.printStackTrace();
}
Logger.log(ReactorParallelTest.class, "map1", "l=" + l);
latch.countDown();
return l;
}).subscribe(l -> {
Logger.log(ReactorParallelTest.class, "onNext", "l=" + l);
}, error -> System.err.println(error),
() -> {
Logger.log(ReactorParallelTest.class, "onComplete", "inside complete.");
executorService.shutdown();
});
try {
latch.await(60, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
You are blocking all your rails with this code. 4 rails will be started (number of CPUs), and they will immediately request 1 element each from the source. Since you immediately block in the map when this is done, the rail cannot request more from upstream, so in effect you only get 4 elements at a time, block, get more, block... The parallelism is more limited than the capacity of the thread pool. If you want to put all threads to good use, do .parallel(5) (same configuration as the thread pool).
On a side note, the subscribe(lambda) from ParallelFlux will invoke the onComplete callback for each rail. If you want to merge back to a single sequence (and single complete), use a .sequential() just before .subscribe.
Due to Heartbleed, our Gateway Server was updated and this problem presented itself.
Due to POODLE, SSLv3 is no longer supported.
Note, the problem is only present on Win7+ boxes; WinXP boxes work without issue (same code, different OS = problem); granted WinXP is no longer a valid OS, just wanted to make note of functionality.
Client application (.NET 2.0) sits on a Windows 7 (or 8) box. Server runs within a DMZ behind a Gateway Server. Just to note, I found that this problem is no longer present on .NET 4.0+ - however due to legacy code, I do not have the luxury of updating.
Gateway Server is a pass through box on which Apache HTTP Server with SSL run. Its location is outside the DMZ, and it is used to access the Server which is inside the DMZ. Versions of software running on the Gateway server are Apache/2.2.25 (Win32), mod_jk/1.2.39, mod_ssl/2.2.25, OpenSSL/1.0.1g
Here is the code used on the Client application (with an exorbitant amount of logging added) ... note, 'serverName' typically contains a value such as "https://some.url.com"
private bool ConnectAndAuthenicate(string serverName, out TcpClient client, out SslStream sslStream)
{
client = null;
sslStream = null;
try
{
client = new TcpClient(serverName, 443); // Create a TCP/IP client; ctor attempts connection
Log("ConnectAndAuthenicate: Client CONNECTED"));
sslStream = new SslStream(client.GetStream(), false, ValidateServerCertificate, null);
Log("ConnectAndAuthenicate: SSL Stream CREATED"));
}
catch (Exception x)
{
Log("ConnectAndAuthenicate: EXCEPTION >> CONNECTING to server: {0}", x.ToString()));
if (x is SocketException)
{
SocketException s = x as SocketException;
Log("ConnectAndAuthenicate: EXCEPTION >> CONNECTING to server: Socket.ErrorCode: {0}", s.ErrorCode));
}
if (client != null) { client.Close(); client = null; }
if (sslStream != null) { sslStream.Close(); sslStream = null; }
}
if (sslStream == null) return false;
try
{
sslStream.ReadTimeout = 10000; // wait 10 seconds for a response ...
Log("ConnectAndAuthenicate: AuthenticateAsClient CALLED ({0})", serverName));
sslStream.AuthenticateAsClient(serverName);
Log("ConnectAndAuthenicate: AuthenticateAsClient COMPLETED SUCCESSFULLY"));
return true;
}
catch (Exception x)
{
Log("ConnectAndAuthenicate: EXCEPTION >> AuthenticateAsClient: {0}", x.ToString()));
client.Close(); client = null;
sslStream.Close(); sslStream = null;
}
return false;
}
Note - answers posted pertaining to ServicePointManager have absolutely no effect on the outcome of this application.
Every time that AuthenicateAsClient() is called when application is run on Win 7+ box, the exception occurs - if application is run on WinXP box, code works properly without exceptions.
Any ideas for solutions are very welcome.
Following the trail of setting the ServicePointManager.SecurityProtocol static ctor with a SecurityProtocolType, I found mention of another enum called SslPolicy -- further research found that AuthenicateAsClient has an overload that takes SslPolicy as an argument.
Changing this line in the above code fixed this problem:
sslStream.AuthenticateAsClient(serverName, null, SslPolicy.Tls, false);
I will try to explain the problem in shortest possible words. I am using c++ builder 2010.
I am using TIdTCPServer and sending voice packets to a list of connected clients. Everything works ok untill any client is disconnected abnormally, For example power failure etc. I can reproduce similar disconnect by cutting the ethernet connection of a connected client.
So now we have a disconnected socket but as you know it is not yet detected at server side so server will continue to try to send data to that client too.
But when server try to write data to that disconnected client ...... Write() or WriteLn() HANGS there in trying to write, It is like it is wating for somekind of Write timeout. This hangs the hole packet distribution process as a result creating a lag in data transmission to all other clients. After few seconds "Socket Connection Closed" Exception is raised and data flow continues.
Here is the code
try
{
EnterCriticalSection(&SlotListenersCriticalSection);
for(int i=0;i<SlotListeners->Count;i++)
{
try
{
//Here the process will HANG for several seconds on a disconnected socket
((TIdContext*) SlotListeners->Objects[i])->Connection->IOHandler->WriteLn("Some DATA");
}catch(Exception &e)
{
SlotListeners->Delete(i);
}
}
}__finally
{
LeaveCriticalSection(&SlotListenersCriticalSection);
}
Ok i already have a keep alive mechanism which disconnect the socket after n seconds of inactivity. But as you can imagine, still this mechnism cant sync exactly with this braodcasting loop because this braodcasting loop is running almost all the time.
So is there any Write timeouts i can specify may be through iohandler or something ? I have seen many many threads about "Detecting disconnected tcp socket" but my problem is little different, i need to avoid that hangup for few seconds during the write attempt.
So is there any solution ?
Or should i consider using some different mechanism for such data broadcasting for example the broadcasting loop put the data packet in some kind of FIFO buffer and client threads continuously check for available data and pick and deliver it to themselves ? This way if one thread hangs it will not stop/delay the over all distribution thread.
Any ideas please ? Thanks for your time and help.
Regards
Jams
There are no write timeouts implemented in Indy. For that, you will have to use the TIdSocketHandle.SetSockOpt() method to set the socket-level timeouts directly.
The FIFO buffer is a better option (and a better design in general). For example:
void __fastcall TForm1::IdTCPServer1Connect(TIdContext *AContext)
{
...
AContext->Data = new TIdThreadSafeStringList;
...
}
void __fastcall TForm1::IdTCPServer1Disconnect(TIdContext *AContext)
{
...
delete AContext->Data;
AContext->Data = NULL;
...
}
void __fastcall TForm1::IdTCPServer1Execute(TIdContext *AContext)
{
TIdThreadSafeStringList *Queue = (TIdThreadSafeStringList*) AContext->Data;
TStringList *Outbound = NULL;
TStringList *List = Queue->Lock();
try
{
if( List->Count > 0 )
{
Outbound = new TStringList;
Outbound->Assign(List);
List->Clear();
}
}
__finally
{
Queue->Unlock();
}
if( Outbound )
{
try
{
AContext->Connection->IOHandler->Write(Outbound);
}
__finally
{
delete Outbound;
}
}
...
}
...
try
{
EnterCriticalSection(&SlotListenersCriticalSection);
int i = 0;
while( i < SlotListeners->Count )
{
try
{
TIdContext *Ctx = (TIdContext*) SlotListeners->Objects[i];
TIdThreadSafeStringList *Queue = (TIdThreadSafeStringList*) Ctx->Data;
Queue->Add("Some DATA");
++i;
}
catch(const Exception &e)
{
SlotListeners->Delete(i);
}
}
}
__finally
{
LeaveCriticalSection(&SlotListenersCriticalSection);
}