Our application is able to connect to RabbitMQ server, messages get received in the message listeners, everything works fine. But when RabbitMQ is restarted then, we get below exception in the logs and messages are not received in the listeners. Once we restart our application container as well, then everything starts working. We cannot restart our application in Prodcution environment and we want the application to be able to recover connection to RabbitMQ once RabbitMQ server is up. Can someone please help.
2018-05-10 09:10:01,561[SimpleAsyncTaskExecutor-34]|DEBUG|org.springframework.beans.factory.support.DefaultListableBeanFactory|13-org.springframework.beans-3.1.4.RELEASE|Returning cached instance of singleton bean 'org.springframework.amqp.core.Queue#3'
2018-05-10 09:10:01,560[SimpleAsyncTaskExecutor-27]|DEBUG|org.springframework.amqp.rabbit.listener.BlockingQueueConsumer|435-wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar-0.0.0|Closing Rabbit Channel: null
2018-05-10 09:10:01,563[SimpleAsyncTaskExecutor-34]|DEBUG|org.springframework.amqp.rabbit.listener.BlockingQueueConsumer|435-wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar-0.0.0|Starting consumer Consumer: tags=[[]], channel=null, acknowledgeMode=MANUAL local queue size=0
2018-05-10 09:10:01,560[SimpleAsyncTaskExecutor-32]|DEBUG|org.springframework.beans.factory.support.DefaultListableBeanFactory|13-org.springframework.beans-3.1.4.RELEASE|Returning cached instance of singleton bean 'org.springframework.amqp.core.Queue#1'
2018-05-10 09:10:01,559[SimpleAsyncTaskExecutor-29]|WARN|org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer|435-wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar-0.0.0|Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:54)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:371)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:80)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:130)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:365)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1009)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_80]
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)[:1.7.0_80]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)[:1.7.0_80]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)[:1.7.0_80]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)[:1.7.0_80]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)[:1.7.0_80]
at java.net.Socket.connect(Socket.java:579)[:1.7.0_80]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625)[:1.7.0_80]
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:50)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:61)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:99)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:925)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:884)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:842)[432:com.rabbitmq.client:4.3.0]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1026)[432:com.rabbitmq.client:4.3.0]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)[435:wrap_file_._jetstream_thirdparty_spring-rabbit-1.3.6.RELEASE.jar:0]
... 7 more
This exception seems wierd to me because, if RabbitMQ is refusing connections then, how is the connection established when we do an application restart.
This issue is giving me a very hard time.
The 1.3.6 is too old and out of support version. Would be great if you consider to upgrade to the latest one: https://projects.spring.io/spring-amqp/, at least 1.7.7 . Also read this: https://docs.spring.io/spring-amqp/docs/2.0.3.RELEASE/reference/html/_reference.html#auto-recovery.
Related
I have created a .net core 3.1 console application for reading azure service bus queue messages and deployed the EXE in a client on-premise VM. It was working initially but now it is not working from VM (now also working from local machine). I am getting a time-out (socket exception) while executing the exe in on-premise VM. I am using shared access policy connection strung to connect the service bus.
Exception :
Azure.Messaging.ServiceBus.ServiceBusException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ErrorCode: TimedOut (ServiceCommunicationProblem)
---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
at Microsoft.Azure.Amqp.Transport.TransportStream.EndRead(IAsyncResult asyncResult)
at Microsoft.Azure.Amqp.Transport.TransportStream.<>c__DisplayClass22_0.b__1(IAsyncResult a)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
The most common connection issue in an enterprise environment is that the ports needed for AMQP over TCP (5671/5672) are not open. Changing the transport to AMQP over WebSockets often helps, as it will use port 443 and may be routed through a proxy, if needed.
Both the transport and the proxy (if needed) can be specified using the ServiceBusClientOptions when creating your client:
var options = new ServiceBusClientOptions
(
TransportType = ServiceBusTransportType.AmqpWebSockets,
WebProxy = new WebProxy("https://proxyserver:80", true)
};
var client = new ServiceBusClient("<< CONNECTION STRING >>", options);
For more information, you may want to look at the Service Bus troubleshooting guide.
I have a mule application which exposes a rest webservice.
<inbound-endpoint exchange-pattern="request-response" address="${webservice.url}" doc:name="Generic"></inbound-endpoint>
I am using jersey resources for rest component.
The code works fine in QA but in UAT it throws the following error.
ERROR org.mule.exception.DefaultSystemExceptionStrategy - Caught exception in Exception Strategy: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[?:1.8.0_111]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_111]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_111]
at java.io.BufferedInputStream.read(BufferedInputStream.java:265) ~[?:1.8.0_111]
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) ~[commons-httpclient-3.1.jar:?]
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) ~[commons-httpclient-3.1.jar:?]
at org.mule.transport.http.HttpServerConnection.readLine(HttpServerConnection.java:245) ~[mule-transport-http-3.8.1.jar:3.8.1]
at org.mule.transport.http.HttpServerConnection.getRequestLine(HttpServerConnection.java:557) ~[mule-transport-http-3.8.1.jar:3.8.1]
at org.mule.transport.http.HttpRequestDispatcherWork.run(HttpRequestDispatcherWork.java:67) ~[mule-transport-http-3.8.1.jar:3.8.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Telnet to the port is working. cron jobs are working fine in the server except the rest methods.
Connection request means that the other side closed the connection. Given that the application works in one environment I assume that the problem is the environment. You have to find out what is different between the environments to cause that problem. Is there anything between the client and the Mule application? A load balancer, proxy, etc? Difference in timeouts? Performing a network traffic capture from both sides could be helpful to isolate the problem.
Background
In my application, it needs to connect to zookeeper to get some useful information. So I start a zookeeper container in my integration test with port binding like -p 2859:2181.
zookeeper = new GenericContainer<>("zookeeper:3.6.1")
.withEnv("ZOOKEEPER_CLIENT_PORT", "2181")
.withExposedPorts(2181);
zookeeper.setPortBindings(Arrays.asList("2859:2181"));
zookeeper.start();
Then I use the following command to run the test. It goes smoothly connecting to localhost:2859 without any problems.
mvn -Dtest=xxxIntegrationTest.java test
Problem
However, when I use jenkins to run the same test with same command, it cannot connect to localhost:2859, complaining that
2020-10-28 12:16:04.282 INFO [main-SendThread(localhost:2859)] Opening socket connection to server localhost/127.0.0.1:2859. Will not attempt to authenticate using SASL (unknown error)
2020-10-28 12:16:04.283 WARN [main-SendThread(localhost:2859)] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
I don't know why it behaves differently in these two senarios. Could anyone give me some suggestions?
apparently with no specific reason, and with nothing on neo4j logs, our application is getting this:
2019-01-30 14:15:08,715 WARN com.calenco.core.content3.ContentHandler:177 - Unable to acquire connection from the pool within configured maximum time of 60000ms
org.neo4j.driver.v1.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.processAcquisitionError(ConnectionPoolImpl.java:192)
at org.neo4j.driver.internal.async.pool.ConnectionPoolImpl.lambda$acquire$0(ConnectionPoolImpl.java:89)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at org.neo4j.driver.internal.util.Futures.lambda$asCompletionStage$0(Futures.java:78)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:745)
The neo4j server is still running, and answering requests to either its web browser console, or the cypher-shell CLI. Also, restarting our application re-acquires the connection to neo4j with no issue.
Our application is connecting to neo4j once when it's started and then keeps that connection open for as lunch as it's running, opening and closing sessions against that connection as needed to fulfill the received requests.
It's the 2nd time in less than a month that we see the above exception thrown.
Any ideas?
Thanks in advance
Actually i am using rabbit mq server and mosquitto mqtt client for the connection
but after sometime server is disconnecting from the client
and the error in not traceable.
You should check the rabbitmq log (and log configuration) and also test with different keep alive values.