How to fix RemoteException in server regarding ehcache cache replication? (Tomcat 8.5.84 | Grails 2.4.5 | grails-cache 1.1.8 | grails-cache-ehcache - grails

Question

I have a Grails 2.4.5 application deployed on a server with Tomcat 8.5.84 and JDK_1.8.0_341.
I have an issue with the cache replication from one server (node 2) to the other (node 1).
When a new deployment is done, sometimes the cache replicates without any issues automatically, and the caches are also well synchronized after each deployment if somehow caches happen to not be synchronized at some point.
I'm looking for reliable cache replication, which will help with reliable data and avoid inconsistent data after each refresh because one node might not have all the data or only some of the data.
Here is the common error I keep seeing in the server logs below. I'm hoping to at least understand what is triggering that error and how to fix it. I'm mostly interested in the root cause.
The original cache configuration worked well with Grails 2.2.0, Tomcat 7.0.109 and jdk1.7.0_331.
2023-02-13 13:17:50,287 [http-nio-16280-exec-4] INFO filters.LoggingFilters - [EXIT] - controller: chat, action: createMessage
2023-02-13 13:17:50,549 [Replication Thread] DEBUG distribution.RMICacheManagerPeerProvider - Lookup URL //cacheBaseUrl:cachePort/Chat(cache name)
2023-02-13 13:17:50,552 [Replication Thread] DEBUG distribution.RMICacheManagerPeerProvider - Lookup URL //cacheBaseUrl:cachePort/Chat.channel.chat.general(cache name)
2023-02-13 13:17:50,555 [Replication Thread] WARN distribution.RMIAsynchronousCacheReplicator - Unable to send message to remote peer. Message was: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.InvalidClassException: net.sf.ehcache.distribution.EventMessage; local class incompatible: stream classdesc serialVersionUID = -293616939110963630, local class serialVersionUID = -293616939110963629
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.InvalidClassException: net.sf.ehcache.distribution.EventMessage; local class incompatible: stream classdesc serialVersionUID = -293616939110963630, local class serialVersionUID = -293616939110963629
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:389)
at sun.rmi.transport.Transport$2.run(Transport.java:202)
at sun.rmi.transport.Transport$2.run(Transport.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:572)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:836)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:624)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:689)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:686)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:686)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:163)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
java.io.InvalidClassException: net.sf.ehcache.distribution.EventMessage; local class incompatible: stream classdesc serialVersionUID = -293616939110963630, local class serialVersionUID = -293616939110963629
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:348)
at sun.rmi.transport.Transport$2.run(Transport.java:202)
at sun.rmi.transport.Transport$2.run(Transport.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:198)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:572)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:836)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:624)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:689)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:686)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:686)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.InvalidClassException: net.sf.ehcache.distribution.EventMessage; local class incompatible: stream classdesc serialVersionUID = -293616939110963630, local class serialVersionUID = -293616939110963629
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:694)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2006)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1853)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:504)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:462)
at java.util.ArrayList.readObject(ArrayList.java:780)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1186)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2322)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1670)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:504)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:462)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:325)
at sun.rmi.server.UnicastServerRef.unmarshalParametersUnchecked(UnicastServerRef.java:629)
at sun.rmi.server.UnicastServerRef.unmarshalParameters(UnicastServerRef.java:615)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:338)
... 14 more
2023-02-13 13:17:50,556 [Replication Thread] WARN distribution.RMIAsynchronousCacheReplicator - Unable to send message to remote peer. Message was: error marshalling arguments; nested exception is:
java.net.SocketException: Broken pipe (Write failed)
java.rmi.MarshalException: error marshalling arguments; nested exception is:
java.net.SocketException: Broken pipe (Write failed)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:159)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1915)
at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:292)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:154)
... 5 more
It seems from the error that a somehow different class net.sf.ehcache.distribution.EventMessage is being used at some point, creating a class cast exception, but I'm not sure what causes that and why. I'll need to find out more details about how Grails uses that class underneath.
The class that seems to cause the exception:
Class net.sf.ehcache.distribution.EventMessage
package net.sf.ehcache.distribution;
import net.sf.ehcache.Ehcache;
import net.sf.ehcache.Element;
import java.io.Serializable;
/**
* An Event Message, in respect of a particular cache.
* <p/>
* The message is Serializable, so that it can be sent across the network.
* <p/>
* The value of an Element is referenced with a SoftReference, so that a
* value will fail to be delivered in preference to an OutOfMemory error.
*
* #author Greg Luck
* #version $Id: EventMessage.java 5192 2012-01-18 17:39:08Z cdennis $
*/
public class EventMessage implements Serializable {
private static final long serialVersionUID = -293616939110963630L;
/**
* Enumeration of event types.
*/
public enum EventType {
/**
* A put or update event.
*/
PUT,
/**
* A remove or invalidate event.
*/
REMOVE,
/**
* A removeAll, which removes all elements from a cache
*/
REMOVE_ALL;
}
/**
* The event component.
*/
private final EventType type;
/**
* The element component.
*/
private final Element element;
/**
* The key component.
*/
private final Serializable key;
/**
* The associated cache.
*/
private final transient Ehcache cache;
/**
* Full constructor.
*
* #param cache
* #param type
* #param key
* #param element
*/
public EventMessage(Ehcache cache, EventType type, Serializable key, Element element) {
this.cache = cache;
this.type = type;
this.key = key;
this.element = element;
}
/**
* Gets the associated {#code Ehcache}.
*
* #return the associated cache
*/
public final Ehcache getEhcache() {
return cache;
}
/**
* Gets the event.
*
* #return either {#link #PUT} or {#link #REMOVE}
*/
public final EventType getType() {
return type;
}
/**
* #return the element component of the message. null if a {#link #REMOVE} event
*/
public final Element getElement() {
return element;
}
/**
* #return the key component of the message. null if a {#link #PUT} event
*/
public final Serializable getSerializableKey() {
return key;
}
}