I am testing global transaction and timeout in atomikos. I created test where I am inserting data to first db and then call procedure in second db so I am using global transaction. Here is my test service code:
#Transactional(timeout = 5)
public void rollbackTest2x() {
SqlSession sqlLogSession = getSession(LOGRWX);
wsZadostDao.insert(sqlLogSession, new Record().withUUID(UUID.randomUUID().toString()));
SqlSession sqlSession = getSession(REFRWX);
procDelayDao.procDelay(sqlSession, new ProcDelay().withSecond(10));
// procDelayDao.procDelay(sqlSession, new ProcDelay().withSecond(10));
}
This test throw no exception but the data are rollback. There is just exception in Atomikos PooledAlarmTimer which is not propagated to main thread. So my test ends OK but data are not inserted. Any idea how should I fix it ?
2018-11-12 11:57:39.948 [main] DEBUG o.sqlproc.engine.spring.SpringQuery - callUpdate, query=call PROC_DELAY(?)
2018-11-12 11:57:40.021 [main] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): registering listener atomikos connection pool 'REFRW1X'
2018-11-12 11:57:40.023 [main] TRACE com.atomikos.thread.TaskManager - ThreadFactory: creating new thread: Atomikos:6
2018-11-12 11:57:40.025 [main] TRACE c.a.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'REFRW1X': init done.
2018-11-12 11:57:40.025 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.025 [main] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): updating last time acquired
2018-11-12 11:57:40.025 [main] TRACE c.a.jdbc.AtomikosXAPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): testing connection with query [SELECT 1 FROM DUAL]
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AtomikosXAPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): connection tested OK
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AtomikosXAPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): creating connection proxy...
2018-11-12 11:57:40.026 [main] DEBUG c.a.jdbc.JdbcConnectionProxyHelper - setting isolation level to 2
2018-11-12 11:57:40.026 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 0 context(s): notifySessionBorrowed
2018-11-12 11:57:40.026 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.NotInBranchStateHandler#6f07d414
2018-11-12 11:57:40.026 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.026 [main] DEBUG c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: calling toString...
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: toString returning oracle.jdbc.driver.LogicalConnection#40faff12
2018-11-12 11:57:40.026 [main] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 1 context(s): returning proxy oracle.jdbc.driver.LogicalConnection#40faff12
2018-11-12 11:57:40.026 [main] TRACE c.a.datasource.pool.ConnectionPool - atomikos connection pool 'REFRW1X': current size: 0/1
2018-11-12 11:57:40.026 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.026 [main] DEBUG c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: calling toString...
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: toString returning oracle.jdbc.driver.LogicalConnection#40faff12
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AbstractDataSourceBean - AtomikosDataSoureBean 'REFRW1X': returning oracle.jdbc.driver.LogicalConnection#40faff12
2018-11-12 11:57:40.026 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.026 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: notifyBeforeUse a SessionHandleState with 1 context(s)
2018-11-12 11:57:40.026 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.027 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 1 context(s): checking XA context for transaction com.atomikos.icatch.imp.CompositeTransactionImp#f881a272
2018-11-12 11:57:40.027 [main] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 entering state: ACTIVE
2018-11-12 11:57:40.027 [main] DEBUG c.a.i.imp.CompositeTransactionImp - addParticipant ( XAResourceTransaction: 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343432 ) for transaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.027 [main] TRACE c.a.d.xa.XAResourceTransaction - XAResourceTransaction: 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343432: about to switch to XAResource oracle.jdbc.driver.T4CXAResource#223967ea
2018-11-12 11:57:40.027 [main] TRACE c.a.d.xa.XAResourceTransaction - XAResourceTransaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001my-app-7879c2825b02389836785ecff7ca2c442: switched to XAResource oracle.jdbc.driver.T4CXAResource#223967ea
2018-11-12 11:57:40.027 [main] DEBUG c.a.d.xa.XAResourceTransaction - XAResource.start ( 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343432 , XAResource.TMNOFLAGS ) on resource REFRW1X represented by XAResource instance oracle.jdbc.driver.T4CXAResource#223967ea
2018-11-12 11:57:40.027 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.BranchEnlistedStateHandler#4a6a6a69
2018-11-12 11:57:40.027 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: detected transaction com.atomikos.icatch.imp.CompositeTransactionImp#f881a272
2018-11-12 11:57:40.027 [main] DEBUG c.a.i.imp.CompositeTransactionImp - registerSynchronization ( com.atomikos.jdbc.AtomikosConnectionProxy$JdbcRequeueSynchronization#f881a272 ) for transaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:40.027 [main] DEBUG c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: calling prepareCall({ call PROC_DELAY(?)})...
2018-11-12 11:57:40.048 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: prepareCall returning oracle.jdbc.driver.OracleCallableStatementWrapper#16ac4d3d
2018-11-12 11:57:44.703 [Atomikos:4] WARN c.a.icatch.imp.ActiveStateHandler - Transaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 has timed out - rolling back...
2018-11-12 11:57:44.704 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 entering state: ABORTING
2018-11-12 11:57:44.708 [Atomikos:4] TRACE com.atomikos.thread.TaskManager - ThreadFactory: creating new thread: Atomikos:7
2018-11-12 11:57:44.708 [Atomikos:4] TRACE com.atomikos.thread.TaskManager - ThreadFactory: creating new thread: Atomikos:8
2018-11-12 11:57:44.710 [Atomikos:8] DEBUG c.a.d.xa.XAResourceTransaction - XAResource.end ( 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343432 , XAResource.TMSUCCESS ) on resource REFRW1X represented by XAResource instance oracle.jdbc.driver.T4CXAResource#223967ea
2018-11-12 11:57:44.711 [Atomikos:7] DEBUG c.a.d.xa.XAResourceTransaction - XAResource.end ( 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343431 , XAResource.TMSUCCESS ) on resource LOGRW1X represented by XAResource instance oracle.jdbc.driver.T4CXAResource#446c8c72
2018-11-12 11:57:50.055 [Atomikos:7] DEBUG c.a.d.xa.XAResourceTransaction - XAResource.rollback ( 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343431 ) on resource LOGRW1X represented by XAResource instance oracle.jdbc.driver.T4CXAResource#446c8c72
2018-11-12 11:57:50.058 [Atomikos:8] DEBUG c.a.d.xa.XAResourceTransaction - XAResource.rollback ( 726F622D6170702D3738373963323832356230323338393833363738356563666637636132633434313534323032303235393538363030303031:726F622D6170702D373837396332383235623032333839383336373835656366663763613263343432 ) on resource REFRW1X represented by XAResource instance oracle.jdbc.driver.T4CXAResource#223967ea
2018-11-12 11:57:50.059 [main] DEBUG o.sqlproc.engine.spring.SpringQuery - callUpdate, number of updated rows=-1
2018-11-12 11:57:50.059 [main] DEBUG o.sqlproc.engine.SqlProcedureEngine - << callUpdate, result=-1
2018-11-12 11:57:50.059 [main] TRACE org.my.app.db.ProcDelayDao - sql procDelay result: -1
2018-11-12 11:57:50.060 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.060 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 entering state: TERMINATED
2018-11-12 11:57:50.060 [main] DEBUG c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#72eed547: close()...
2018-11-12 11:57:50.060 [main] TRACE c.a.jdbc.AbstractConnectionProxy - Forcing close of pending statement: oracle.jdbc.driver.OraclePreparedStatementWrapper#37b1149b
2018-11-12 11:57:50.060 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 1 context(s): entering notifySessionClosed
2018-11-12 11:57:50.060 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 1 context(s): delegating session close to a TransactionContext
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing state to com.atomikos.datasource.xa.session.BranchEndedStateHandler#5409dfdd
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.BranchEndedStateHandler#5409dfdd
2018-11-12 11:57:50.061 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#72eed547: closed.
2018-11-12 11:57:50.061 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.061 [main] DEBUG c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: close()...
2018-11-12 11:57:50.061 [main] TRACE c.a.jdbc.AbstractConnectionProxy - Forcing close of pending statement: oracle.jdbc.driver.OracleCallableStatementWrapper#16ac4d3d
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 1 context(s): entering notifySessionClosed
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 1 context(s): delegating session close to a TransactionContext
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing state to com.atomikos.datasource.xa.session.BranchEndedStateHandler#45f9d394
2018-11-12 11:57:50.061 [main] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.BranchEndedStateHandler#45f9d394
2018-11-12 11:57:50.061 [main] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: closed.
2018-11-12 11:57:50.061 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.062 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 : stopping timer...
2018-11-12 11:57:50.062 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 : disposing statehandler TERMINATED...
2018-11-12 11:57:50.062 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 : disposed.
2018-11-12 11:57:50.062 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.062 [main] TRACE c.a.i.i.CompositeTransactionManagerImp - getCompositeTransaction() returning instance with id my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.064 [main] DEBUG c.a.i.imp.CompositeTransactionImp - commit() done (by application) of transaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.064 [main] TRACE c.atomikos.icatch.imp.CoordinatorImp - prepare() of Coordinator my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 returning READONLY
2018-11-12 11:57:50.066 [Atomikos:4] TRACE c.atomikos.icatch.imp.CoordinatorImp - Error during setRollbackOnly
java.lang.IllegalStateException: my-app-7879c2825b02389836785ecff7ca2c44154202025958600001 is no longer active but in state TERMINATED
at com.atomikos.icatch.imp.CoordinatorImp.addParticipant(CoordinatorImp.java:383)
at com.atomikos.icatch.imp.CoordinatorImp.setRollbackOnly(CoordinatorImp.java:703)
at com.atomikos.icatch.imp.TransactionStateHandler.setRollbackOnly(TransactionStateHandler.java:260)
at com.atomikos.icatch.imp.CompositeTransactionImp.setRollbackOnly(CompositeTransactionImp.java:262)
at com.atomikos.icatch.imp.CompositeTransactionImp.entered(CompositeTransactionImp.java:343)
at com.atomikos.finitestates.FSMImp.notifyListeners(FSMImp.java:166)
at com.atomikos.finitestates.FSMImp.setState(FSMImp.java:256)
at com.atomikos.icatch.imp.CoordinatorImp.setState(CoordinatorImp.java:312)
at com.atomikos.icatch.imp.CoordinatorImp.setStateHandler(CoordinatorImp.java:222)
at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackFromWithinCallback(CoordinatorStateHandler.java:506)
at com.atomikos.icatch.imp.ActiveStateHandler$1.doRollback(ActiveStateHandler.java:85)
at com.atomikos.icatch.imp.CoordinatorStateHandler.rollbackWithAfterCompletionNotification(CoordinatorStateHandler.java:563)
at com.atomikos.icatch.imp.ActiveStateHandler.onTimeout(ActiveStateHandler.java:80)
at com.atomikos.icatch.imp.CoordinatorImp.alarm(CoordinatorImp.java:650)
at com.atomikos.timing.PooledAlarmTimer.notifyListeners(PooledAlarmTimer.java:95)
at com.atomikos.timing.PooledAlarmTimer.run(PooledAlarmTimer.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-12 11:57:50.067 [Atomikos:4] DEBUG c.a.i.imp.CompositeTransactionImp - setRollbackOnly() called for transaction my-app-7879c2825b02389836785ecff7ca2c44154202025958600001
2018-11-12 11:57:50.068 [Atomikos:4] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#72eed547: detected termination of transaction com.atomikos.icatch.imp.CompositeTransactionImp#f881a272
2018-11-12 11:57:50.069 [Atomikos:4] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.TerminatedStateHandler#3a06f087
2018-11-12 11:57:50.069 [Atomikos:4] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 0 context(s): all contexts terminated, firing TerminatedEvent for a SessionHandleState with 0 context(s)
2018-11-12 11:57:50.069 [Atomikos:4] TRACE c.a.jdbc.AtomikosXAPooledConnection - SessionHandleState terminated, firing XPooledConnectionTerminated event for an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s)
2018-11-12 11:57:50.069 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): updating last time released
2018-11-12 11:57:50.069 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): notifying listener: atomikos connection pool 'LOGRW1X'
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.datasource.pool.ConnectionPool - atomikos connection pool 'LOGRW1X': connection an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s) became available, notifying potentially waiting threads
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): updating last time released
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.jdbc.AtomikosConnectionProxy - atomikos connection proxy for oracle.jdbc.driver.LogicalConnection#40faff12: detected termination of transaction com.atomikos.icatch.imp.CompositeTransactionImp#f881a272
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.xa.session.TransactionContext - a TransactionContext: changing to state com.atomikos.datasource.xa.session.TerminatedStateHandler#38415c0d
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.xa.session.SessionHandleState - a SessionHandleState with 0 context(s): all contexts terminated, firing TerminatedEvent for a SessionHandleState with 0 context(s)
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.jdbc.AtomikosXAPooledConnection - SessionHandleState terminated, firing XPooledConnectionTerminated event for an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s)
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): updating last time released
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): notifying listener: atomikos connection pool 'REFRW1X'
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.datasource.pool.ConnectionPool - atomikos connection pool 'REFRW1X': connection an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s) became available, notifying potentially waiting threads
2018-11-12 11:57:50.070 [Atomikos:4] TRACE c.a.d.pool.AbstractXPooledConnection - an AtomikosXAPooledConnection with a SessionHandleState with 0 context(s): updating last time released
UPDATE:
I dont understand this atomikos code. I have two participants in transaction with state: TerminatedState which return READ_ONLY in prepare method so there is no commit and no exception is thrown
protected void terminate ( boolean commit ) throws HeurRollbackException,
HeurMixedException, SysException, java.lang.SecurityException,
HeurCommitException, HeurHazardException, RollbackException,
IllegalStateException
{
synchronized ( fsm_ ) {
if ( commit ) {
if ( participants_.size () <= 1 ) {
commit ( true );
} else {
int prepareResult = prepare ();
// make sure to only do commit if NOT read only
if ( prepareResult != Participant.READ_ONLY )
commit ( false );
}
} else {
rollback ();
}
}
}
If all the work is READ_ONLY then there is nothing to commit. It means your SQL did not change any records.
Related
We are using testcontainers to instantiate Kafka containers for testing an app that creates a kstream StateStore and kafka consumer to process some messages. The test goes fine and we could both produce and consume messages. But these tests continuously print annoying WARN messages -
(kafka-admin-client-thread)([]) WARN - NetworkClient - [AdminClient
clientId=test_clientId] Connection to node 1
(localhost/127.0.0.1:55304) could not be established. Broker may not
be available.
It seems the AdminClient is scheduled to poll every second and this creates substantial log files.
On enabling DEBUG logs, I observe the error
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Node 1 disconnected.
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) WARN - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Connection to node 1 (localhost/127.0.0.1:55304) could not be established. Broker may not be available.
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - ConsumerNetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=test_x-global-consumer, correlationId=11) due to node 1 being disconnected
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) INFO - FetchSessionHandler - [Consumer clientId=test_x-global-consumer, groupId=null] Error sending fetch request (sessionId=1421844635, epoch=INITIAL) to node 1:
org.apache.kafka.common.errors.DisconnectException: null
[2022-09-16T22:48:11,792Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Initialize connection to node localhost:55304 (id: 1 rack: null) for sending metadata request
[2022-09-16T22:48:11,792Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - NetworkClient - [AdminClient clientId=adminclient-1] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,792Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,793Z](kafka-producer-network-thread | producer-1)([]) DEBUG - Selector - [Producer clientId=producer-1] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
at java.base/java.lang.Thread.run(Thread.java:833)
[2022-09-16T22:48:11,793Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - Selector - [AdminClient clientId=adminclient-1] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1293)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
at java.base/java.lang.Thread.run(Thread.java:833)
[2022-09-16T22:48:11,793Z](kafka-producer-network-thread | producer-1)([]) DEBUG - NetworkClient - [Producer clientId=producer-1] Node 1 disconnected.
[2022-09-16T22:48:11,793Z](kafka-admin-client-thread | adminclient-1)([]) DEBUG - NetworkClient - [AdminClient clientId=adminclient-1] Node 1 disconnected.
Similar logs are also coming from GlobalStreamThread
[2022-09-16T22:48:11,746Z](test_x-GlobalStreamThread)([]) DEBUG - NetworkClient - [Consumer clientId=test_x-global-consumer, groupId=null] Initiating connection to node localhost:55304 (id: 1 rack: null) using address localhost/127.0.0.1
[2022-09-16T22:48:11,747Z](test_x-GlobalStreamThread)([]) DEBUG - Selector - [Consumer clientId=test_x-global-consumer, groupId=null] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:219)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:530)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1296)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1237)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1210)
at org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.pollAndUpdate(GlobalStreamThread.java:237)
at org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:284)
Docker port mapping looks like
0.0.0.0:55306->2181/tcp, 0.0.0.0:55305->9092/tcp, 0.0.0.0:55304->9093/tcp
Testcontainer is started as
KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
.withNetwork(Network.SHARED)
Testcontainer maps KAFKA_LISTENERS here
https://github.com/testcontainers/testcontainers-java/blob/master/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L49
Could you help me understand what is happening here and how to provide the correct port mappings via testcontainers for these threads to work fine?
I install kylin by https://github.com/cas-packone/ambari-kylin-service/
2020-01-13 01:52:16,673 INFO [main] utils.Compatibility:41 : Running in ZooKeeper 3.4.x compatibility mode
2020-01-13 01:52:16,710 INFO [main] imps.CuratorFrameworkImpl:284 : Starting
2020-01-13 01:52:16,715 INFO [main] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=node1:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState#6c000e0c
2020-01-13 01:52:16,717 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1013 : Opening socket connection to server node1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-01-13 01:52:16,719 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:856 : Socket connection established, initiating session, client: /127.0.0.1:49768, server: node1/127.0.0.1:2181
2020-01-13 01:52:16,721 INFO [main-SendThread(node1:2181)] zookeeper.ClientCnxn:1273 : Session establishment complete on server node1/127.0.0.1:2181, sessionid = 0x16f798087cc03af, negotiated timeout = 60000
2020-01-13 01:52:16,722 INFO [main] imps.CuratorFrameworkImpl:326 : Default schema
2020-01-13 01:52:16,725 DEBUG [main] util.ZookeeperDistributedLock:142 : 6521#node1 trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2020-01-13 01:52:16,730 INFO [main-EventThread] state.ConnectionStateManager:237 : State change: CONNECTED
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata#hbase
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:83)
... 3 more
Caused by: java.lang.NoSuchMethodError: org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModePathAndBytesable;
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:145)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:166)
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:305)
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:110)
at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:91)
... 8 more
2020-01-13 01:52:16,772 INFO [Curator-Framework-0] imps.CuratorFrameworkImpl:924 : backgroundOperationsLoop exiting
2020-01-13 01:52:16,773 INFO [Thread-1] zookeeper.ReadOnlyZKClient:344 : Close zookeeper connection 0x7b94089b to node1:2181
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03ae closed
2020-01-13 01:52:16,775 INFO [ReadOnlyZKClient-node1:2181#0x7b94089b-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
2020-01-13 01:52:16,777 INFO [Thread-4] zookeeper.ZooKeeper:692 : Session: 0x16f798087cc03af closed
2020-01-13 01:52:16,777 INFO [main-EventThread] zookeeper.ClientCnxn:517 : EventThread shut down
ERROR: Unknown error. Please check full log.
I have a question regarding Flink. I am running an application in a local cluster, with 1 TaskManager and 4 Taskslots.
After some time of running the application, I got an Timeout error:
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id feea6a6702a0cf960ae2847b5bd25665 timed out.
I have seen some posts with this topic but any answer to it. Could you help me to see the root cause, or a posible troubleshooting?
I am using flink version 1.5.3
It seems that the docker container of taskmanagers and JobManager are stopped when this happens.
Let me add the error trace from the JobManager container logs:
2019-06-09 13:31:06,300 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Socket Window NgsiEvent (ef3a860de48d54544d973754c6170d8b) switched from state FAILING to FAILED.
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 63dbab620797b84da023b33578478238 timed out.
at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1609)
at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:339)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:154)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-06-09 13:31:06,308 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Could not restart the job Socket Window NgsiEvent (ef3a860de48d54544d973754c6170d8b) because the restart strategy prevented it.
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id 63dbab620797b84da023b33578478238 timed out.
at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1609)
at org.apache.flink.runtime.heartbeat.HeartbeatManagerImpl$HeartbeatMonitor.run(HeartbeatManagerImpl.java:339)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:154)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2019-06-09 13:31:06,317 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping checkpoint coordinator for job ef3a860de48d54544d973754c6170d8b.
2019-06-09 13:31:06,322 INFO org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore - Shutting down
2019-06-09 13:31:06,331 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f]
2019-06-09 13:31:06,351 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job ef3a860de48d54544d973754c6170d8b reached globally terminal state FAILED.
2019-06-09 13:31:06,434 INFO org.apache.flink.runtime.jobmaster.JobMaster - Stopping the JobMaster for job Socket Window NgsiEvent(ef3a860de48d54544d973754c6170d8b).
2019-06-09 13:31:06,447 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Suspending SlotPool.
2019-06-09 13:31:06,448 INFO org.apache.flink.runtime.jobmaster.JobMaster - Close ResourceManager connection 883e842633b0fd9a2e53ab45778581fe: JobManager is shutting down..
2019-06-09 13:31:06,449 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcActor - The rpc endpoint org.apache.flink.runtime.jobmaster.slotpool.SlotPool has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.LocalRpcInvocation until processing is started.
2019-06-09 13:31:06,457 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Disconnect job manager 00000000000000000000000000000000#akka.tcp://flink#jobmanager:6123/user/jobmanager_2 for job ef3a860de48d54544d973754c6170d8b from the resource manager.
2019-06-09 13:31:06,459 INFO org.apache.flink.runtime.jobmaster.slotpool.SlotPool - Stopping SlotPool.
2019-06-09 13:31:06,460 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManagerRunner already shutdown.
2019-06-09 13:31:16,304 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f: Name or service not known]
2019-06-09 13:31:26,320 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f: Name or service not known]
2019-06-09 13:31:36,286 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink#16363182f31f:36715] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink#16363182f31f:36715]] Caused by: [16363182f31f]
Thanks in advance!
my spring boot app with amqp rabbitMQ hangs without errors in one of the environments. The same config works ok in another.
The thread dump shows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <e62a7fa> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <5d300728> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1c4259b2> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
and then all the AMQP consumer threads are in BLOCKED state,
"AMQPConsumerThread_20" - Thread t#100
java.lang.Thread.State: BLOCKED
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.isActive(SimpleMessageListenerContainer.java:870)
- waiting to lock <5d300728> (a java.lang.Object) owned by "main" t#32
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1000(SimpleMessageListenerContainer.java:95)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1310)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:268)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
The SpringBoot app then waits with the last line on console,
2017-02-08T17:48:26.91-0500 [APP/0] OUT 2017-02-08 17:48:26 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory -
Created new connection: SimpleConnection#64f05682
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#x.y.z.w:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 57509]
EDIT
when I drop the consumer count down to 1, the BLOCKED state goes away and the main thread state is as follows,
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:815)
- locked <4f3dcc9> (a java.lang.Object)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:550)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:173)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:346)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:149)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:112)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:874)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:144)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544)
- locked <1a60fd96> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
But the app still does not start.
the rabbitmq webconsole shows the consumer connected with the last line in logs,
2017-02-08T18:36:09.99-0500 [APP/0] OUT 2017-02-08 18:36:09 [AMQPConsumerThread_1] INFO o.s.a.r.c.CachingConnectionFactory - Created new connection: SimpleConnection#39271079
[delegate=amqp://b489be6b-b6e5-41b5-8f31-3be3acac4518#10.146.54.74:5672/daad99cf-fa24-40fe-b0d1-f9312fb583be, localPort= 32880]
what am I missing?
thanks
This...
"main" - Thread t#32
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <79908042> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at org.springframework.util.ConcurrencyThrottleSupport.beforeAccess(ConcurrencyThrottleSupport.java:124)
at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottleAdapter.beforeAccess(SimpleAsyncTaskExecutor.java:243)
at org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:184)
Indicates the SimpleAsyncTaskExecutor is at its thread limit (setConcurrencyLimit). This check is only performed if you have set a limit.
This won't proceed until some task terminates. Since the consumer threads are long-lived, that will likely not happen.
From last two years on tomcat I am using this configuration for database connection pooling in tomcat
<Resource auth="Container"
driverClassName="com.mysql.jdbc.Driver"
logAbandoned="true"
maxActive="100"
maxIdle="30"
maxWait="10000"
name="jdbc/maindb"
password="xxxxx"
removeAbandoned="true"
removeAbandonedTimeout="60"
type="javax.sql.DataSource"
url="jdbc:mysql://localhost:3306/maindb?zeroDateTimeBehavior=convertToNull"
connectionProperties="useEncoding=true;"
username="sqladmin" validationQuery="select 1"/>
On production server, from last one month, with this configuration - suddenly tomcat stop responding to any requests. And results in timeout. There is no errors in logs, but as soon as I shutdown tomcat a huge amount of error log comes, which seems to show some kind of deadlock in database connections.
To rectify it, I used database connection pooling configuration from http://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html . After using this configuration, now I face two problems, on production, either a table lock occurs even after using INNODB engine, or some queries start returning empty result set even when query is perfectly fine.
<Resource name="jdbc/maindb"
auth="Container"
type="javax.sql.DataSource"
factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
testWhileIdle="true"
testOnBorrow="true"
testOnReturn="false"
validationQuery="SELECT 1"
validationInterval="30000"
timeBetweenEvictionRunsMillis="30000"
maxActive="100"
minIdle="10"
maxWait="10000"
initialSize="10"
removeAbandonedTimeout="60"
removeAbandoned="true"
logAbandoned="true"
minEvictableIdleTimeMillis="30000"
jmxEnabled="true"
jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer"
username="sqladmin"
password="xxxxx"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/maindb"/>
In case of first configuration, after shutting down tomcat following error logs start coming
04-Feb-2015 20:44:46.048 INFO [main] org.apache.catalina.core.StandardServer.await A valid shutdown command was received via the shutdown port. Stopping the Server instance.
04-Feb-2015 20:44:46.049 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
04-Feb-2015 20:44:46.100 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-apr-8009"]
04-Feb-2015 20:44:46.151 INFO [main] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
04-Feb-2015 20:44:46.157 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:47.158 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:48.160 INFO [localhost-startStop-2] org.apache.catalina.core.StandardWrapper.unload Waiting for 81 instance(s) to be deallocated for Servlet [dispatcher]
04-Feb-2015 20:44:48.260 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing WebApplicationContext for namespace 'dispatcher-servlet': startup date [Tue Feb 03 18:26:26 UTC 2015]; parent: Root WebApplicationContext
04-Feb-2015 20:44:48.307 INFO [localhost-startStop-2] org.springframework.context.support.AbstractApplicationContext.doClose Closing Root WebApplicationContext: startup date [Tue Feb 03 18:26:24 UTC 2015]; root of context hierarchy
04-Feb-2015 20:44:48.310 INFO [localhost-startStop-2] org.springframework.scheduling.concurrent.ExecutorConfigurationSupport.shutdown Shutting down ExecutorService 'taskExecutor'
04-Feb-2015 20:44:48.329 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
com.myproj.dao.ConnectionPool.getConnection(ConnectionPool.java:41)
And in second case following error logs comes, while doing certain operation
22-Jan-2015 16:36:04.077 SEVERE [http-apr-8080-exec-2] com.myproj.dao.cart.impl.VisitorCartDaoImpl.addCartItem Lock wait timeout exceeded; try restarting transaction
java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2077)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2062)
at com.myproj.dao.cart.impl.VisitorCartDaoImpl.addCartItem(VisitorCartDaoImpl.java:96)
Is there anything wrong with the configuration or with the database? I am using MySQL 5.6 as the database and on production, mySQL is running on amazon RDS. And I am using tomcat 8.0.15