With reactor-netty-http:1.0.20, on setting HttpServer.idleTimeout(idleTimeoutInMs), no GOWAY packet is seen in tcpdump - timeout

With reactor-netty-http:1.0.20 (with spring-webflux), on setting HttpServer.idleTimeout(idleTimeoutInMs), no GOWAY packet is seen in tcpdump.with protocol H2C, HTTP11, but directly a FIN packet at TCP level.
HttpServer.idleTimeout(idleTimeoutInMs) definition says,
.idleTimeout => Specifies an idle timeout on the connection when it is waiting for an HTTP request (resolution: ms). Once the timeout is reached the connection will be closed.
This definition doesn't say anything about the level at which it sets the idletimeout, is it directly at TCP level or HTTP level.
As per the observation from tcpdump, the idle connection termination is not graceful. The connection gets terminated on reaching the idle timeout, as seen by FIN packet at TCP level. But no GOAWAY packet is observed.
Need to configure idle timeout at HTTP level so that it is accompanied with a GOAWAY packet to terminate connection gracefully.
#Configuration
public class NettyServerFactoryConfig implements WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
#Value("${server.http.port}")
private Integer httpPort;
#Value("${server.idleTimeout}")
private Integer idleTimeout;
#Value("${server.settingsMaxConcurrentStreams}")
private Integer settingsMaxConcurrentStreams;
#Override
public void customize(NettyReactiveWebServerFactory factory) {
LoopResources resource = LoopResources.create("my-resource");
ReactorResourceFactory reactorResourceFactory = new ReactorResourceFactory();
reactorResourceFactory.setLoopResources(resource);
reactorResourceFactory.setUseGlobalResources(false);
factory.setResourceFactory(reactorResourceFactory);
Http2 h2 = new Http2();
h2.setEnabled(true);
factory.setPort(httpPort);
factory.setHttp2(h2);
Consumer<Builder> h2Settings = s -> s.maxConcurrentStreams(settingsMaxConcurrentStreams);
factory.addServerCustomizers(httpServer -> httpServer.http2Settings(h2Settings));
factory.addServerCustomizers(httpServer -> httpServer.idleTimeout(Duration.ofMillis(idleTimeout)));
factory.addServerCustomizers(builder -> builder.protocol(HttpProtocol.H2C, HttpProtocol.HTTP11));
factory.setShutdown(Shutdown.GRACEFUL);
}
}

Related

Spring AMQP how to access response message when connection is lost while sending using #SendTo

we have a microservice which consumes a message using #RabbitListener and persist data into database, generate a response on successful processing of message and send it using #sendTO to different queue for auditing.
When running Rabbit in HA failover, while sending response if connection is lost the message currently being processed is correctly returned to the queue but database transaction (jpa transaction in our case) is not rolled back , response is never sent.
I read from this issue(https://github.com/spring-projects/spring-amqp/issues/696) that this is "best effort 1PC" transaction synchronization; RabbitMQ does not support XA transactions. The Rabbit tx is committed after the DB tx and there is a possibility the DB tx might commit and the rabbit rolled back; you have to deal with the small possibility of duplicate messages.
But in our case when we retry request, we are treating it as duplicate message and response is never created for this request. is there a way where we can only retry sending response message in case of connection lost exceptions rather than reprocessing request again? I looked at ConditionalRejectingErrorHandler.DefaultExceptionStrategy, it has access only to original request,no way to access response lost during connection failure. Please suggest what's the best way to handle this?
our code looks like:
SpringBootApplication
#EnableJpaRepositories("com.***")
#EnableJpaAuditing
#EnableTransactionManagement
#EnableEncryptableProperties
public class PcaClinicalValidationApplication {
#RabbitListener(queues = "myqueue"
#SendTo("exchange/routingKey")
#Timed) description = "Time taken to process a request")
public Message receivemessage(HashMap<String, Object> myMap, Message requestMessage)
throws Exception {
//business logic goes here
Message message = MessageBuilder.fromMessage(requestMessage)
//add some headers
return message;
}
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory,
SimpleRabbitListenerContainerFactoryConfigurer configurer) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setRetryTemplate(new RetryTemplate());
factory.setReplyRecoveryCallback(ctx -> {
Message failed = SendRetryContextAccessor.getMessage(ctx);
Address replyTo = SendRetryContextAccessor.getAddress(ctx);
Throwable t = ctx.getLastThrowable();
//wrote to a file
serializer.serialize(failed);
return null;
});
return factory;
}
The listener container factory uses a RabbitTemplate in its replyTemplate property - this is used to send the reply.
You can configure a RetryTemplate into that RabbitTemplate to retry sending the reply.
When retries are exhausted, you can add a RecoveryCallback which will get the failed reply and you can save it off someplace and use it when the redelivery occurs.

Artemis broker Intercept mqtt client connection

I added a mqtt interceptor into my artemis broker in order to intercept mqtt client connection:
public class SimpleMQTTInterceptor implements MQTTInterceptor
{
#Override
public boolean intercept(final MqttMessage mqttMessage, RemotingConnection connection) throws ActiveMQException
{
System.out.println("MQTT Interceptor gets called ");
if (mqttMessage instanceof MqttConnectMessage)
{
System.out.println("MQTT connection intercepted ");
}
return true;
}
My client apache paho connect to the broker via this port "ws://0.0.0.0:61614".
My problem is that only message published to topics are intercepted.
Why this doesn't intercept CONNECT message ?
The current version of ActiveMQ Artemis, 2.2.0, at the time I write this response, only supports intercepting MQTT Publish control packets. I opened a pull request adding that feature, therefore, it should be present on future versions.

Prevent keeping unused DB connection

Problem description:
Lets have a service method which is called from controller:
class PaymentService {
static transactional = false
public void pay(long id) {
Member member = Member.get(id)
//long running task executing HTTP request
requestPayment(member)
}
}
The problem is if 8 users hit the same service in the same time and the time to execute the requestPayment(member) method is 30 seconds, the whole application gets stucked for 30 seconds.
The problem is even bigger than it seems, because if the HTTP request is performing well, nobody realizes any trouble. The serious problem is that availability of our web service depends on the availability of our external partner/component (in our use-case payment gateway). So when your partner starts to have performance issues, you will have them as well and even worse it will affect all parts of your app.
Evaluation:
The cause of problem is that Member.get(id) reserves a DB connection from pool and it keeps it for further use, despite requestPayment(member) method never needs to access DB. When next (9-th) request hits any other part of the application which requires DB connection (transactional service, DB select, ...) it keeps waiting (or timeouts if maxWait is set to lower duration) until the pool has an available connection, which can last even 30 seconds in our use case.
The stacktrace for the waiting thread is:
at java.lang.Object.wait(Object.java:-1)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1115)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
Or for timeout:
JDBC begin failed
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1167)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 7 more
Obviously the same issue happens with transactional service, however it makes much more sense since the connection is reserved for the transaction.
As a temporary solution its possible to increase the pool size with maxActive property on datasource, however it doesn't solve the real problem of holding an unused connection.
As a permanent solution its possible to enclose all DB operations to transactional behavior (withTransaction{..}, #Transactional), which returns the connection back to pool after commit (or to my surprise also withNewSession{..} works). But we need to be sure that the whole call chain from controller up to the requestPayment(member) method doesn't leak the connection.
I'd like to be able to throw an exception in the requestPayment(member) method if the connection is "leaked" (similar to Propagation.NEVER transactional behavior), so I can reveal the issue early during test phase.
After digging in the source code I've found the solution:
class PaymentService {
static transactional = false
def sessionFactory
public void pay(long id) {
Member member = Member.get(id)
sessionFactory.currentSession.disconnect()
//long running task executing HTTP request
requestPayment(member)
}
}
The above statement releases the connection back to pool.
If executed from transactional context, an exception is thrown (org.hibernate.HibernateException connnection proxy not usable after transaction completion), since we can't release such a connection (which is exactly what I needed).
Javadoc:
Disconnect the Session from the current JDBC connection. If the
connection was obtained by Hibernate close it and return it to the
connection pool; otherwise, return it to the application.
This is used by applications which supply JDBC connections to
Hibernate and which require long-sessions (or long-conversations)

c3p0 connection pool unable to see physical connections in oracle monitor screen

import com.mchange.v2.c3p0.*;
import java.sql.*;
public class DBconnectionpool{
public static void main(String ar[]) throws Exception{
ComboPooledDataSource cpds = new ComboPooledDataSource();
cpds.setDriverClass( "oracle.jdbc.driver.OracleDriver" );
cpds.setJdbcUrl( "jdbc:oracle:thin:#localhost:1521:xe" );
cpds.setUser("bms");
cpds.setPassword("abc");
cpds.setInitialPoolSize(5);
Connection con = cpds.getConnection();
System.out.println("got the connection"+con);
}
}
after Executing the above code ,In the Oracle monitor page i'm unable to see the 5 physical connections (i.e as i have set cpds.setInitialPoolSize(5)) instead i see only one connection.
your program exits immediately after one Connection is acquired. try sleeping for 30000 ms after you've acquired your Connection.

What is the timeout for SignalR persistence connection?

I created Chat application using SignalR and Asp.net MVC. if user is idle for sometime, he didn't receive any messages from the server.
Is there any timeout for SignalR persistence connection ?
If yes, how do I modify or reactivate my connection ?
This has now been upated on the Wiki https://github.com/SignalR/SignalR/wiki/Configuring-SignalR
Code
using System;
using SignalR;
namespace WebApplication1
{
public class Global : System.Web.HttpApplication
{
void Application_Start(object sender, EventArgs e)
{
// Make connections wait 50s maximum for any response. After
// 50s are up, trigger a timeout command and make the client reconnect.
GlobalHost.Configuration.ConnectionTimeout = TimeSpan.FromSeconds(50);
}
}
}
SignalR's default timeout is 110 seconds for any connection. It will reconnected automatically and you shouldn't miss any messages. That sounds like you have some other problem. If you want to tweak the timeout in 0.4 like this (assuming asp.net):
// Make idle connections reconnect every 60 seconds
AspNetHost.DependencyResolver.Resolve<IConfigurtionManager>().ReconnectTimeout = TimeSpan.FromSeconds(60);
Make sure You add using SignalR.Infrastructure to get the Resolve extension method (we're fixing this in the next version).

Resources