I have a RabbitTemplate configured with a RetryTemplate in the following way:
#Bean
public RabbitTemplate myRabbitTemplate()
{
RabbitTemplate template = new RabbitTemplate(myConnectionFactory());
template.setChannelTransacted(true);
template.setMessageConverter(myMessageConverter());
template.setReplyAddress(MY_RESPONSE_PRIVATE_QUEUE);
template.setQueue(MY_RESPONSE_PRIVATE_QUEUE);
template.setMandatory(true);
template.setEncoding("UTF-8");
template.setExchange(MY_REQUEST_EXCHANGE);
template.setBeforePublishPostProcessors(new MyMessagePostProcessor());
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(500);
backOffPolicy.setMultiplier(10.0);
backOffPolicy.setMaxInterval(10000);
retryTemplate.setBackOffPolicy(backOffPolicy);
template.setRetryTemplate(retryTemplate);
template.setAfterReceivePostProcessors(new DelegatingDecompressingPostProcessor());
return template;
}
Doing connection tests using TCPView and WireShark, I think that in one of the connection cuts one of the sent messages has been duplicated (I guess by the configuration of the RetryTemplate).
Throughout the project there have been other connection failures and errors (ShutdownSignalException, TimeOut ...) and we have never detected duplicate messages.
Under what circumstances is the sending of the message re-attempted?
Thank you
EDIT 11/03/2019
Now I have a little more information about the repeated messages. During the sending of the message (and before receiving the ACK), a connection error occurred:
06-03-2019 10:09:52.919|WARN |com.rabbitmq.client.impl.ForgivingExceptionHandler|120||AMQP Connection XXX.XX.XX.XX:XXXXX|An unexpected connection driver error occured (Exception message: Connection reset)
com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:868)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:858)
at com.rabbitmq.client.impl.AMQConnection.handleFailure(AMQConnection.java:681)
at com.rabbitmq.client.impl.AMQConnection.access$400(AMQConnection.java:47)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:582)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
The server receives two messages with the same spring_listener_return_correlation but different spring_request_return_correlation:
spring_listener_return_correlation=0475d6f3-4859-418d-93fe-ed96fb28e733, spring_request_return_correlation=2926
spring_listener_return_correlation=0475d6f3-4859-418d-93fe-ed96fb28e733, spring_request_return_correlation=2927
Can the configuration of the "RetryTemplate" be the cause of this duplicate shipment?
Thanks
Related
we have a microservice which consumes a message using #RabbitListener and persist data into database, generate a response on successful processing of message and send it using #sendTO to different queue for auditing.
When running Rabbit in HA failover, while sending response if connection is lost the message currently being processed is correctly returned to the queue but database transaction (jpa transaction in our case) is not rolled back , response is never sent.
I read from this issue(https://github.com/spring-projects/spring-amqp/issues/696) that this is "best effort 1PC" transaction synchronization; RabbitMQ does not support XA transactions. The Rabbit tx is committed after the DB tx and there is a possibility the DB tx might commit and the rabbit rolled back; you have to deal with the small possibility of duplicate messages.
But in our case when we retry request, we are treating it as duplicate message and response is never created for this request. is there a way where we can only retry sending response message in case of connection lost exceptions rather than reprocessing request again? I looked at ConditionalRejectingErrorHandler.DefaultExceptionStrategy, it has access only to original request,no way to access response lost during connection failure. Please suggest what's the best way to handle this?
our code looks like:
SpringBootApplication
#EnableJpaRepositories("com.***")
#EnableJpaAuditing
#EnableTransactionManagement
#EnableEncryptableProperties
public class PcaClinicalValidationApplication {
#RabbitListener(queues = "myqueue"
#SendTo("exchange/routingKey")
#Timed) description = "Time taken to process a request")
public Message receivemessage(HashMap<String, Object> myMap, Message requestMessage)
throws Exception {
//business logic goes here
Message message = MessageBuilder.fromMessage(requestMessage)
//add some headers
return message;
}
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setRetryTemplate(new RetryTemplate());
factory.setReplyRecoveryCallback(ctx -> {
Message failed = SendRetryContextAccessor.getMessage(ctx);
Address replyTo = SendRetryContextAccessor.getAddress(ctx);
Throwable t = ctx.getLastThrowable();
//wrote to a file
serializer.serialize(failed);
return null;
});
return factory;
}
The listener container factory uses a RabbitTemplate in its replyTemplate property - this is used to send the reply.
You can configure a RetryTemplate into that RabbitTemplate to retry sending the reply.
When retries are exhausted, you can add a RecoveryCallback which will get the failed reply and you can save it off someplace and use it when the redelivery occurs.
Why direct message is not sent?
spring-social-twitter:1.1.2.RELEASE does not work with spring-web:5.0.6.RELEASE
Example of main class:
package com.test;
import org.springframework.social.twitter.api.ResourceFamily;
import org.springframework.social.twitter.api.impl.TwitterTemplate;
public class Main {
public static void main(String[] args) {
final TwitterTemplate template = new TwitterTemplate("TEST", "TEST","TEST", "TEST");
template.userOperations().getRateLimitStatus(ResourceFamily.DIRECT_MESSAGES);
System.out.println("Successfuly obtained rate limits");
System.out.println("Sending DM");
template.directMessageOperations().sendDirectMessage("user_id", "text");
}
}
dependencies from gradle build file:
dependencies {
compile "org.springframework:spring-web:5.0.6.RELEASE"
compile "org.springframework.social:spring-social-twitter:1.1.2.RELEASE"
compile "com.fasterxml.jackson.core:jackson-databind:2.9.5"
}
Result of execution:
Successfuly obtained rate limits
Sending DM
Exception in thread "main" org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://api.twitter.com/1.1/direct_messages/new.json": cannot retry due to server authentication, in streaming mode; nested exception is java.net.HttpRetryException: cannot retry due to server authentication, in streaming mode
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:732)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:698)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:457)
at org.springframework.social.twitter.api.impl.DirectMessageTemplate.sendDirectMessage(DirectMessageTemplate.java:77)
at com.test.Main.main(Main.java:13)
Caused by: java.net.HttpRetryException: cannot retry due to server authentication, in streaming mode
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1692)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
at org.springframework.http.client.SimpleClientHttpResponse.getRawStatusCode(SimpleClientHttpResponse.java:55)
at org.springframework.web.client.DefaultResponseErrorHandler.hasError(DefaultResponseErrorHandler.java:51)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:754)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:724)
... 4 more
You didn't provide a callback_url.
Go to your twitter app Settings page:
https://apps.twitter.com/
and fill in Callback Url input field. This will fix your I/O problem.
When subscribing to ObjectMaterialize
((IObjectContextAdapter)context).ObjectContext.ObjectMaterialized += (sender, e) => { /* Note, fails even when the handler is empty */ }
this works fine in production code. However, as soon as I add this, I get in tests exceptions:
Message: System.Data.SqlClient.SqlException : A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
Removing that single line (ObjectMaterialized += .. ) resolves it again. I have been debugging, but cannot find the cause. Does anyone here happen to have encountered this before?
I'm a doofus... This was in a test assembly that didn't have a ConnectionString, and apparently adding this handler causes a connection to be established when instantiating the DbContext.
So, if you have this problem: Add connection strings to your App.config of the test project.
I'm publishing messages into RabbitMQ and I would like to track the errors when RabbitMQ is down, for this I added one RetryTemplate with the recovery callback, but the recovery callback only provides this method getLastThrowable() and I'm not sure how to provide the details of the messages that failed when RabbitMQ is down. (as per documentation "The RecoveryCallback is somewhat limited in that the retry context only contains the
lastThrowable field. For more sophisticated use cases, you should use an external
RetryTemplate so that you can convey additional information to the RecoveryCallback via
the context’s attributes") but I don't know how to do that, if anyone could help me with one example that will be awesome.
Rabbit Template
public RabbitTemplate rabbitMqTemplate(RecoveryCallback publisherRecoveryCallback) {
RabbitTemplate r = new RabbitTemplate(rabbitConnectionFactory);
r.setExchange(exchangeName);
r.setRoutingKey(routingKey);
r.setConnectionFactory(rabbitConnectionFactory);
r.setMessageConverter(jsonMessageConverter());
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialBackOffPolicy backOffPolicy = new ExponentialBackOffPolicy();
backOffPolicy.setInitialInterval(500);
backOffPolicy.setMultiplier(10.0);
backOffPolicy.setMaxInterval(10000);
retryTemplate.setBackOffPolicy(backOffPolicy);
r.setRetryTemplate(retryTemplate);
r.setRecoveryCallback(publisherRecoveryCallback);
return r;
}
Recovery Callback
#Component
public class PublisherRecoveryCallback implements RecoveryCallback<AssortmentEvent> {
#Override
public AssortmentEvent recover(RetryContext context) throws Exception {
log.error("Error publising event",context.getLastThrowable());
//how to get message details here??
return null;
}
}
AMQP Outbound Adapter
return IntegrationFlows.from("eventsChannel")
.split()
.handle(Amqp.outboundAdapter(rabbitMqTemplate)
.exchangeName(exchangeName)
.confirmCorrelationExpression("payload")
.confirmAckChannel(ackChannel)
.confirmNackChannel(nackChannel)
)
.get();
The isn't possible because the function RabbitTemplate.execute() is already not aware about message you send, because it may be performed from any other method, where we might not have messages to deal:
return this.retryTemplate.execute(
(RetryCallback<T, Exception>) context -> RabbitTemplate.this.doExecute(action, connectionFactory),
(RecoveryCallback<T>) this.recoveryCallback);
What I suggest you to do is like storing message to the ThreadLocal before send and get it from there from your custom RecoveryCallback.
We have an evirironment as follows:
CPE: 2 Servers
ICN: 2 servers
Application Server: WAS 8.5.5 Base
Both content Engine and Navigator are configured for high availability using Load Balancer. However, in case ICN 1 is connected to CPE1 and CPE1 is dwn, then Navigator is unable to connect to CPE2 even though load balancer of CPE is pointing to CPE2.
The logs are as follows:
javax.naming.NamingException: NMSV0610I: A NamingException is being thrown from a javax.naming.Context implementation. Details follow:
Context implementation: com.ibm.ws.naming.jndicos.CNContextImpl
Context method: lookupExt
Context name: HDOSYS0202Node01Cell/nodes/HDOSYS0202Node01/servers/server1
Target name: FileNet/Engine,10.39.128.66:2809/FileNet/Engine
Other data:
Exception stack trace: javax.naming.NamingException: Error during resolve [Root exception is org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible vmcid: IBM minor code: E07 completed: No]
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1867)
at com.ibm.ws.naming.jndicos.CNContextImpl.doLookup(CNContextImpl.java:1776)
at com.ibm.ws.naming.jndicos.CNContextImpl.lookupExt(CNContextImpl.java:1433)
at com.ibm.ws.naming.jndicos.CNContextImpl.lookup(CNContextImpl.java:615)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:165)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
at org.apache.aries.jndi.DelegateContext.lookup(DelegateContext.java:161)
at javax.naming.InitialContext.lookup(InitialContext.java:436)
com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1864)
Caused by: org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible vmcid: IBM minor code: E07 completed: No
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:412)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:271)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:258)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:376)
at java.net.Socket.connect(Socket.java:546)
at com.ibm.ws.orbimpl.transport.WSTCPTransportConnection.createSocket(WSTCPTransportConnection.java:370)
at com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.java:366)
at com.ibm.ws.orbimpl.transport.WSTransport.getConnection(WSTransport.java:437)
at com.ibm.CORBA.transport.TransportBase.getConnection(TransportBase.java:188)
at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:100)
at com.ibm.rmi.iiop.GIOPImpl.getConnection(GIOPImpl.java:134)
at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:178)
at com.ibm.rmi.corba.ClientDelegate._createRequest(ClientDelegate.java:2010)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1186)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1272)
Content Platform Engine does not support session replication which would be required to failover. Once the connection is established, the client will bind to the specific endpoint so neither corbaloc nor a load balancing alias will help. If the nodes are not in a Cluster the peer will not be in the JNDI tree so do not know about each other. What you have is called a "stovepipe" configuration. You can load balance the front end, but each front end will talk to a specific backend, so not highly available. You could put the CE's behind a hardware load balancer (SNAT) but it would still lack fail over. CPE will run on JBOSS but ICN does not, so to be highly available you'll need to deploy to WebSphere ND or Weblogic.
Could you share the URI used to establish CPE connection?
When Content Platform Engine is made highly available through an application server cluster configuration the Content Platform Engine URI should have the following form (with no carriage returns):
corbaloc::node1_hostname:BOOTSTRAP_ADDRESS,:node2_hostname:BOOTSTRAP_ADDRESS/cell/clusters/your_websphere_cluster_name/FileNet/Engine
Example:
corbaloc::testnode1:9810,:testnode2:9810/cell/clusters/testwascluster/FileNet/Engine
This configuration requires the WebSphere cluster name in addition to the node names as part of the URI. The bootstrap port for a cluster configuration (by default, port 9810) is usually different from the bootstrap port on a non-cluster (standalone) configuration (by default, port 2809).
Only one URI is used regardless of SSL use. WebSphere EJB over SSL is automatically established if EJB security is enabled.
I found a link containing code to solve the issue in my case. The only problem is how to implement this code for Content Navigator
"This may help. I have recently written an EJB print app which is used by other apps at my company to generate printable documents. I am also using an access bean on the client to remotely call my EJB. The client is a 4 server cluster, and my EJB is a 2 server cluster. I have also experienced problems with the "connection refused" exception if I stop the application server(s) running my EJB when calling without restarting the client. Here is what I've done so far to resolve the issue.
Looking at the access bean, after you create an instance, when you call your remote method (whatever that may be and in my case is renderDocuments() which i will use in my example below) the access bean does the following:"
public DocumentRenderOutputContext renderDocuments
DocumentRequestList documentRequestList)
{
try
{
instantiateEJB();
return ejbRef().renderDocuments
documentRequestList);
}
catch (NamingException ne)
{
throw new DocumentRenderException(ne);
}
catch (CreateException ce)
{
throw new DocumentRenderException(ce);
}
catch (RemoteException re)
{
THE EXCEPTION THROWN WHEN THE APP SERVER IS
BROUGHT DOWN WITHOUT RESTARTING THE CLIENT
WILL BE CAUGHT HERE
}
}
If you bring down your EJB app server(s) without re-starting the client, the remote exception above will catch the "connect refused" exception.
So what i do inside the remote exception catch is the following:
try
{
//see below for methods
reset();
return retryRenderDocuments(documentRequestList);
}
catch (NamingException ne)
{
throw new DocumentRenderException(ne);
}
catch (CreateException ce)
{
throw new DocumentRenderException(ce);
}
catch (RemoteException remote)
{
throw new DocumentRenderException(re);
}
private void reset() throws NamingException
{
resetHomeCache();
resetEJBRef();
}
private DocumentRenderOutputContext retryRenderDocuments
DocumentRequestList documentRequestList)
throws
RemoteException,
NamingException,
CreateException,
DocumentRenderException
{
DocumentRenderOutputContext outputContext = null;
Properties properties = new Properties();
properties.put(
javax.naming.Context.PROVIDER_URL,
getInit_NameServiceURLName()); //im assuming youve
properties.put(
PROPS.JNDI_CACHE_OBJECT,
PROPS.JNDI_CACHE_OBJECT_CLEARED);
InitialContext initialContext = new InitialContext(properties);
Object object = initialContext.lookup(getInit_JNDIName());
ECommercePrintHome homeRef = (ECommercePrintHome) object;
ECommercePrint printEngine = homeRef.create();
outputContext = printEngine.renderDocuments(documentRequestList);
return outputContext;
}
Ref:- http://www.theserverside.com/discussions/thread.tss?thread_id=31495