Ehcache + Terracotta: ConnectionClosedException - terracotta

Question

I'm setting up Terracotta cluster to support Ehcache.
I'm following the basic example here
I downloaded ehcache-clustered-kit version 3.8.0 from Maven Central and untarred it on my machine. I didn't modifiy anything, except for a fix to the sh script to set the JAVA_HOME correctly.
I executed it using the default OOB configuration (ie: omitting the -f parameter):
./start-tc-server.sh
from the [ehcache root]/server/bin folder.
I'm initializing the CacheManager like that:
private CacheManager setup() {
CacheManagerBuilder<PersistentCacheManager> clusteredCacheManagerBuilder =
CacheManagerBuilder.newCacheManagerBuilder()
.with(ClusteringServiceConfigurationBuilder.cluster(URI.create("terracotta://localhost:9410/my-application"))
.autoCreate(s -> s));
PersistentCacheManager cacheManager = clusteredCacheManagerBuilder.build(true);
return cacheManager;
}
I receive this exception:
org.terracotta.exception.ConnectionClosedException: Entity: org.terracotta.lease.LeaseAcquirer:SystemLeaseAcquirer Connection closed before sending message
at com.tc.object.ClientEntityManagerImpl.throwClosedExceptionOnMessage(ClientEntityManagerImpl.java:509)
at com.tc.object.ClientEntityManagerImpl.queueInFlightMessage(ClientEntityManagerImpl.java:651)
at com.tc.object.ClientEntityManagerImpl.queueInFlightMessage(ClientEntityManagerImpl.java:621)
at com.tc.object.ClientEntityManagerImpl.sendMessageWhileBusy(ClientEntityManagerImpl.java:592)
at com.tc.object.ClientEntityManagerImpl.internalRelease(ClientEntityManagerImpl.java:572)
at com.tc.object.ClientEntityManagerImpl.internalLookup(ClientEntityManagerImpl.java:558)
at com.tc.object.ClientEntityManagerImpl.fetchEntity(ClientEntityManagerImpl.java:182)
at com.terracotta.connection.entity.TerracottaEntityRef.fetchEntity(TerracottaEntityRef.java:82)
at org.terracotta.lease.LeaseMaintainerFactory.getLeaseAcquirer(LeaseMaintainerFactory.java:63)
at org.terracotta.lease.LeaseMaintainerFactory.createLeaseMaintainer(LeaseMaintainerFactory.java:46)
at org.terracotta.lease.connection.BasicLeasedConnection.create(BasicLeasedConnection.java:40)
at org.terracotta.lease.connection.LeasedConnectionServiceImpl.createLeasedConnection(LeasedConnectionServiceImpl.java:58)
at org.terracotta.lease.connection.LeasedConnectionServiceImpl.connect(LeasedConnectionServiceImpl.java:47)
at org.terracotta.lease.connection.LeasedConnectionFactory.getLeasedConnection(LeasedConnectionFactory.java:71)
at org.terracotta.lease.connection.LeasedConnectionFactory.connect(LeasedConnectionFactory.java:47)
at org.ehcache.clustered.client.internal.ConnectionSource$ClusterUri.connect(ConnectionSource.java:70)
at org.ehcache.clustered.client.internal.service.ConnectionState.connect(ConnectionState.java:151)
at org.ehcache.clustered.client.internal.service.ConnectionState.initClusterConnection(ConnectionState.java:128)
at org.ehcache.clustered.client.internal.service.DefaultClusteringService.start(DefaultClusteringService.java:122)
at org.ehcache.core.spi.ServiceLocator.startAllServices(ServiceLocator.java:128)
at org.ehcache.core.EhcacheManager.init(EhcacheManager.java:577)
at com.togather.web.webapp.www.mvc.cache.CacheWrapper.<init>(CacheWrapper.java:63)
at com.togather.web.webapp.www.config.CacheConfig.cacheWrapper(CacheConfig.java:37)
at com.togather.web.webapp.www.config.CacheConfig$$EnhancerBySpringCGLIB$$d8860419.CGLIB$cacheWrapper$1(<generated>)
at com.togather.web.webapp.www.config.CacheConfig$$EnhancerBySpringCGLIB$$d8860419$$FastClassBySpringCGLIB$$dfa29d4c.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:355)
at com.togather.web.webapp.www.config.CacheConfig$$EnhancerBySpringCGLIB$$d8860419.cacheWrapper(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:545)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:368)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1192)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1068)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1014)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:618)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:331)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:446)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:328)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:215)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:184)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:249)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
while in Terracotta log I can see
2020-09-17 11:52:14,413 [WorkerThread(l2_state_change_stage, 0)] INFO org.terracotta.console - Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9410 successfully, and is now ready for work.
2020-09-17 11:52:14,413 [WorkerThread(l2_state_change_stage, 0)] INFO org.terracotta.console - Moved to State[ ACTIVE-COORDINATOR ]
2020-09-17 11:52:35,144 [L2_L1:TCWorkerComm # 0_R] ERROR com.tc.net.protocol.TCNetworkMessage - Hydrate Error - Message Class: com.tc.object.msg.ClientHandshakeMessageImpl
Sealed: true, Header Length: 8, Data Length: 105, Total Length: 113
2020-09-17 11:52:35,145 [L2_L1:TCWorkerComm # 0_R] ERROR com.tc.net.protocol.tcm.TCMessageHydrateSink - Error hydrating message of type CLIENT_HANDSHAKE_MESSAGE
com.tc.net.protocol.tcm.UnknownNameException: unknown name: 10 for message class com.tc.object.msg.ClientHandshakeMessageImpl
at com.tc.net.protocol.tcm.TCMessageImpl.hydrate(TCMessageImpl.java:179)
at com.tc.net.protocol.tcm.TCMessageHydrateSink.putMessage(TCMessageHydrateSink.java:36)
at com.tc.net.protocol.tcm.TCMessageRouterImpl.putMessage(TCMessageRouterImpl.java:64)
at com.tc.net.protocol.tcm.AbstractMessageChannel.receive(AbstractMessageChannel.java:215)
at com.tc.net.protocol.transport.MessageTransportBase.receiveToReceiveLayer(MessageTransportBase.java:138)
at com.tc.net.protocol.transport.ServerMessageTransport.receiveTransportMessageImpl(ServerMessageTransport.java:101)
at com.tc.net.protocol.transport.MessageTransportBase.receiveTransportMessage(MessageTransportBase.java:113)
at com.tc.net.protocol.transport.ServerStackProvider$MessageSink.putMessage(ServerStackProvider.java:267)
at com.tc.net.protocol.transport.WireProtocolAdaptorImpl.addReadData(WireProtocolAdaptorImpl.java:75)
at com.tc.net.core.TCConnectionImpl.addNetworkData(TCConnectionImpl.java:827)
at com.tc.net.core.TCConnectionImpl.doReadFromBufferInternal(TCConnectionImpl.java:488)
at com.tc.net.core.TCConnectionImpl.doReadFromBuffer(TCConnectionImpl.java:342)
at com.tc.net.core.TCConnectionImpl.doReadInternal(TCConnectionImpl.java:325)
at com.tc.net.core.TCConnectionImpl.doRead(TCConnectionImpl.java:309)
at com.tc.net.core.CoreNIOServices$CommThread.selectLoop(CoreNIOServices.java:667)
at com.tc.net.core.CoreNIOServices$CommThread.run(CoreNIOServices.java:371)
2020-09-17 11:52:35,158 [L2_L1:TCComm Main Selector Thread_R (listen 0:0:0:0:0:0:0:0:9410)] INFO com.tc.net.protocol.transport.ServerStackProvider - "Client Cannot Reconnect. Connection attempts from the Terracotta node at 127.0.0.1:50936 are being rejected by the Terracotta server array. Reason: Stack for ConnectionID(10.e25596a1cb1b459f89bf97f2cc9f500d.80b726cf-fe23-437a-953a-42b2b3a3333c-1749b7ac242.Permanent) not found."
What am I doing wrong?

Answer 1

My mistake.
For a typo, I included in my Java project the wrong dependencies:
<dependency>
<groupId>org.ehcache</groupId>
<artifactId>ehcache</artifactId>
<version>3.9.0</version>
</dependency>
<dependency>
<groupId>org.ehcache</groupId>
<artifactId>ehcache-clustered</artifactId>
<version>3.9.0</version>
</dependency>
Instead of version 3.8.0.
This generated a conflict with the Ehcache cluster I installed (3.8.0). Once corrected the dependencies version, I could connect successfully.