jetty replay request on timeout - timeout

We are facing an issue in Jetty where on timeout it replays the original request again if we don't complete the request from async context. Here is the behavior, for every request we set a async listener with timeout, so we have 2 threads in play, one (Jetty Thread1) is listening on timeout and other (Thread2) is serving thread. Now let us say write data to client takes longer than timeout, since the request is not completed timeout thread gets triggered, it checks that someone is writing data so it returns silently. Jetty doesn't like returning silently, it replays the request back so another serving and timeout thread gets created and it goes on until data is written and async context is completed.
The code in question is here - In HttpChannelState in expired() method
if (aListeners!=null)
{
for (AsyncListener listener : aListeners)
{
try
{
listener.onTimeout(event);
}
catch(Exception e)
{
LOG.debug(e);
event.setThrowable(e);
_channel.getRequest().setAttribute(RequestDispatcher.ERROR_EXCEPTION,e);
break;
}
}
}
boolean dispatch=false;
synchronized (this)
{
if (_async==Async.EXPIRING)
{
_async=Async.EXPIRED;
if (_state==State.ASYNC_WAIT)
{
_state=State.ASYNC_WOKEN;
dispatch=true;
}
}
}
if (dispatch)
scheduleDispatch(); // <------------ dispatch again why
}

This is normal behaviour. You have put the request into async state and then not handled the timeout, so the request is redispatch with a DispatcherType of ASYNC.
If you add your own timeout listener and within that timeout you either complete or dispatch the asyncContext, then jetty will not redispatch it (unless your listener called dispatch).
You can also protect your async servlet code with a test for the DispatcherType, although that can be confused if you have multiple concerns that might be handled async.
asyncContext.addListener(new AsyncListener()
{
#Override
public void onTimeout(AsyncEvent event) throws IOException
{
event.getAsyncContext().complete();
}
#Override
public void onStartAsync(AsyncEvent event) throws IOException
{
}
#Override
public void onError(AsyncEvent event) throws IOException
{
}
#Override
public void onComplete(AsyncEvent event) throws IOException
{
}
});

Related

Is Orleans reminder execution interleaved?

If there are two different reminders on the same grain activation to be fired at the same point, given that grain execution context is single-threaded, will both reminders be executed and interleaved at the same time?
Also, is the reminder execution limited by the default 30s timeout ?
Reminders are invoked using regular grain method calls: the IRemindable interface is a regular grain interface. IRemindable.ReceiveReminder(...) is not marked as [AlwaysInterleave], so it will only be interleaved if your grain class is marked as [Reentrant].
In short: no, reminder calls are not interleaved by default.
Reminders do not override the SiloMessagingOptions.ResponseTimeout value, so the default execution time will be 30s.
If you have a reminder that might need a very long time to execute, you can follow a pattern of starting the long-running work in a background task and ensuring that it is still running (not completed or faulted) whenever the relevant reminder fires.
Here is an example of using that pattern:
public class MyGrain : Grain, IMyGrain
{
private readonly CancellationTokenSource _deactivating = new CancellationTokenSource();
private Task _processQueueTask;
private IGrainReminder _reminder = null;
public Task ReceiveReminder(string reminderName, TickStatus status)
{
// Ensure that the reminder task is running.
if (_processQueueTask is null || _processQueueTask.IsCompleted)
{
if (_processQueueTask?.Exception is Exception exception)
{
// Log that an error occurred.
}
_processQueueTask = DoLongRunningWork();
_processQueueTask.Ignore();
}
return Task.CompletedTask;
}
public override async Task OnActivateAsync()
{
if (_reminder != null)
{
return;
}
_reminder = await RegisterOrUpdateReminder(
"long-running-work",
TimeSpan.FromMinutes(1),
TimeSpan.FromMinutes(1)
);
}
public override async Task OnDeactivateAsync()
{
_deactivating.Cancel(throwOnFirstException: false);
Task processQueueTask = _processQueueTask;
if (processQueueTask != null)
{
// Optionally add some max deactivation timeout here to stop waiting after (eg) 45 seconds
await processQueueTask;
}
}
public async Task StopAsync()
{
if (_reminder == null)
{
return;
}
await UnregisterReminder(_reminder);
_reminder = null;
}
private async Task DoLongRunningWork()
{
// Log that we are starting the long-running work
while (!_deactivating.IsCancellationRequested)
{
try
{
// Do long-running work
}
catch (Exception exception)
{
// Log exception. Potentially wait before retrying loop, since it seems like GetMessageAsync may have failed for us to end up here.
}
}
}
}

SimpleRabbitListenerContainerFactory and defaultRequeueRejected

As per doc, defaultRequeueRejected's default value is true, but looking at code it seems its false. I am not sure if I am missing anything or we have to change that in SimpleRabbitListenerContainerFactory.java
EDIT
Sample code, after putting message in test queue, I expect it to stay in queue since its failing but it is throwing it out. I want message to be retried so I configured that in container factory if it fails after retry I want it to be back in queue. I am sure I am missing understanding here.
#SpringBootApplication
public class MsgRequeExampleApplication {
public static void main(String[] args) {
SpringApplication.run(MsgRequeExampleApplication.class, args);
}
#Bean(name = "myContainerFactory")
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
factory.setMissingQueuesFatal(false);
FixedBackOffPolicy backOffPolicy = new FixedBackOffPolicy();
backOffPolicy.setBackOffPeriod(500);
factory.setAdviceChain(new Advice[] { org.springframework.amqp.rabbit.config.RetryInterceptorBuilder.stateless()
.maxAttempts(2).backOffPolicy(backOffPolicy).build() });
return factory;
}
#RabbitListener(queues = "test", containerFactory = "myContainerFactory")
public void processAdvisory(Message message) throws MyBusinessException {
try{
//Simulating exception while processing message
String nullString=null;
nullString.length();
}catch(Exception ex){
throw new MyBusinessException(ex.getMessage());
}
}
public class MyBusinessException extends Exception {
public MyBusinessException(String msg) {
super(msg);
}
}
}
There is a good description in the SimpleMessageListenerContainer JavaDocs:
/**
* Set the default behavior when a message is rejected, for example because the listener
* threw an exception. When true, messages will be requeued, when false, they will not. For
* versions of Rabbit that support dead-lettering, the message must not be requeued in order
* to be sent to the dead letter exchange. Setting to false causes all rejections to not
* be requeued. When true, the default can be overridden by the listener throwing an
* {#link AmqpRejectAndDontRequeueException}. Default true.
* #param defaultRequeueRejected true to reject by default.
*/
public void setDefaultRequeueRejected(boolean defaultRequeueRejected) {
this.defaultRequeueRejected = defaultRequeueRejected;
}
Does it make sense to you?
UPDATE
To requeue after retry exhausting you need to configure some custom MessageRecoverer on the RetryInterceptorBuilder with the code like:
.recoverer((message, cause) -> {
ReflectionUtils.rethrowRuntimeException(cause);
})
This way the exception will be thrown to the listener container and according its defaultRequeueRejected the message will be requeued or not.

Not able to receive onNext and onComplete call on subscribed mono

I was trying reactor library and I'm not able to figure out why below mono never return back with onNext or onComplete call. I think I missing very trivial thing. Here's a sample code.
MyServiceService service = new MyServiceService();
service.save("id")
.map(myUserMono -> new MyUser(myUserMono.getName().toUpperCase(), myUserMono.getId().toUpperCase()))
.subscribe(new Subscriber<MyUser>() {
#Override
public void onSubscribe(Subscription s) {
System.out.println("Subscribed!" + Thread.currentThread().getName());
}
#Override
public void onNext(MyUser myUser) {
System.out.println("OnNext on thread " + Thread.currentThread().getName());
}
#Override
public void onError(Throwable t) {
System.out.println("onError!" + Thread.currentThread().getName());
}
#Override
public void onComplete() {
System.out.println("onCompleted!" + Thread.currentThread().getName());
}
});
}
private static class MyServiceService {
private Repository myRepo = new Repository();
public Mono<MyUser> save(String userId) {
return myRepo.save(userId);
}
}
private static class Repository {
public Mono<MyUser> save(String userId) {
return Mono.create(myUserMonoSink -> {
Future<MyUser> submit = exe.submit(() -> this.blockingMethod(userId));
ListenableFuture<MyUser> myUserListenableFuture = JdkFutureAdapters.listenInPoolThread(submit);
Futures.addCallback(myUserListenableFuture, new FutureCallback<MyUser>() {
#Override
public void onSuccess(MyUser result) {
myUserMonoSink.success(result);
}
#Override
public void onFailure(Throwable t) {
myUserMonoSink.error(t);
}
});
});
}
private MyUser blockingMethod(String userId) throws InterruptedException {
Thread.sleep(5000);
return new MyUser("blocking", userId);
}
}
Above code only prints Subcribed!main. What I'm not able to figure out is why that future callback is not pushing values through myUserMonoSink.success
The important thing to keep in mind is that a Flux or Mono is asynchronous, most of the time.
Once you subscribe, the asynchronous processing of saving the user starts in the executor, but execution continues in your main code after .subscribe(...).
So the main thread exits, terminating your test before anything was pushed to the Mono.
[sidebar]: when is it ever synchronous?
When the source of data is a Flux/Mono synchronous factory method. BUT with the added pre-requisite that the rest of the chain of operators doesn't switch execution context. That could happen either explicitly (you use a publishOn or subscribeOn operator) or implicitly (some operators like time-related ones, eg. delayElements, run on a separate Scheduler).
Simply put, your source is ran in the ExecutorService thread of exe, so the Mono is indeed asynchronous. Your snippet on the other hand is ran on main.
How to fix the issue
To observe the correct behavior of Mono in an experiment (as opposed to fully async code in production), several possibilities are available:
keep subscribe with system.out.printlns, but add a new CountDownLatch(1) that is .countDown() inside onComplete and onError. await on the countdown latch after the subscribe.
use .log().block() instead of .subscribe(...). You lose the customization of what to do on each event, but log() will print those out for you (provided you have a logging framework configured). block() will revert to blocking mode and do pretty much what I suggested with the CountDownLatch above. It returns the value once available or throws an Exception in case of error.
instead of log() you can customize logging or other side effects using .doOnXXX(...) methods (there's one for pretty much every type of event + combinations of events, eg. doOnSubscribe, doOnNext...)
If you're doing a unit test, use StepVerifier from the reactor-tests project. It will subscribe to the flux/mono and wait for events when you call .verify(). See the reference guide chapter on testing (and the rest of the reference guide in general).
Issue is that in created anonymous class onSubscribe method does nothing.
If you look at implementation of LambdaSubscriber, it requests some number of events.
Also it's easier to extend BaseSubscriber as it has some predefined logic.
So your subscriber implementation would be:
MyServiceService service = new MyServiceService();
service.save("id")
.map(myUserMono -> new MyUser(myUserMono.getName().toUpperCase(), myUserMono.getId().toUpperCase()))
.subscribe(new BaseSubscriber<MyUser>() {
#Override
protected void hookOnSubscribe(Subscription subscription) {
System.out.println("Subscribed!" + Thread.currentThread().getName());
request(1); // or requestUnbounded();
}
#Override
protected void hookOnNext(MyUser myUser) {
System.out.println("OnNext on thread " + Thread.currentThread().getName());
// request(1); // if wasn't called requestUnbounded() 2
}
#Override
protected void hookOnComplete() {
System.out.println("onCompleted!" + Thread.currentThread().getName());
}
#Override
protected void hookOnError(Throwable throwable) {
System.out.println("onError!" + Thread.currentThread().getName());
}
});
Maybe it's not the best implementation, I'm new to reactor too.
Simon's answer has pretty good explanation about testing asynchronous code.

spring websocket : SubProtocolWebSocketHandler

I have stability problems with Spring web-sockets. Sometimes I have a similar exception to that describe in https://jira.spring.io/browse/SPR-12812.
The patch is not available, then I have implemented my own code with a custom SubProtocolWebSocketHandler.
public class WBSSubProtocolWebSocketHandler extends SubProtocolWebSocketHandler {
private static final Logger LOG = LoggerFactory.getLogger(WBSSubProtocolWebSocketHandler.class);
public WBSSubProtocolWebSocketHandler(MessageChannel clientInboundChannel, SubscribableChannel clientOutboundChannel) {
super(clientInboundChannel, clientOutboundChannel);
}
#Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
super.afterConnectionClosed(session, status);
LOG.debug("WebSocket Connection closed for client with session ID {}", session.getId());
}
#Override
public void afterConnectionEstablished(WebSocketSession session) throws Exception {
// WebSocketHandlerDecorator could close the session
// https://jira.spring.io/browse/SPR-12812
if (!session.isOpen()) {
LOG.warn("WebSocket Connection established for client with session ID {} was closed.", session.getId());
return;
}
super.afterConnectionEstablished(session);
LOG.debug("WebSocket Connection established for client with session ID {}", session.getId());
}
#Override
public void handleTransportError(WebSocketSession session, Throwable exception) throws Exception {
super.handleTransportError(session, exception);
LOG.warn("WebSocket transport error for client with session ID {}", session.getId());
}
#Override
public void handleMessage(WebSocketSession session, WebSocketMessage message) throws Exception {
super.handleMessage(session, message);
LOG.debug("Websocket incoming message ({}) from client with session ID {}", message.getPayload().toString(), session.getId());
}
#Override
public void handleMessage(Message message) throws MessagingException {
super.handleMessage(message);
LOG.debug("Websocket incoming message : {}, header {}", message);
}
}
Now my problem is to reproduce the exception to see if the problem is solved. I tried various ways, but without success. I can not reproduce the closing of the connection. Does anyone have an idea?
We also have a second problem. The client application (angularjs application) sometimes reports that the socket-web connection is lost. But I do not understand why because on the server I have no error / warning in the logs.
How can I identify the problem and reproduce it?

JavaFX - waiting for task to finish

I have a JavaFX application which instantiates several Task objects.
Currently, my implementation (see below) calls the behavior runFactory() which performs computation under a Task object. Parallel to this, nextFunction() is invoked. Is there a way to have nextFunction() "wait" until the prior Task is complete?
I understand thread.join() waits until the running thread is complete, but with GUIs, there are additional layers of complexity due to the event dispatch thread.
As a matter of fact, adding thread.join() to the end of the code-segment below only ceases UI interaction.
If there are any suggestions how to make nextFunction wait until its prior function, runFactory is complete, I'd be very appreciative.
Thanks,
// High-level class to run the Knuth-Morris-Pratt algorithm.
public class AlignmentFactory {
public void perform() {
KnuthMorrisPrattFactory factory = new KnuthMorrisPrattFactory();
factory.runFactory(); // nextFunction invoked w/out runFactory finishing.
// Code to run once runFactory() is complete.
nextFunction() // also invokes a Task.
...
}
}
// Implementation of Knuth-Morris-Pratt given a list of words and a sub-string.
public class KnuthMorrisPratt {
public void runFactory() throws InterruptedException {
Thread thread = null;
Task<Void> task = new Task<Void>() {
#Override public Void call() throws InterruptedException {
for (InputSequence seq: getSequences) {
KnuthMorrisPratt kmp = new KnuthMorrisPratt(seq, substring);
kmp.align();
}
return null;
}
};
thread = new Thread(task);
thread.setDaemon(true);
thread.start();
}
When using Tasks you need to use setOnSucceeded and possibly setOnFailed to create a logic flow in your program, I propose that you also make runFactory() return the task rather than running it:
// Implementation of Knuth-Morris-Pratt given a list of words and a sub-string.
public class KnuthMorrisPratt {
public Task<Void> runFactory() throws InterruptedException {
return new Task<Void>() {
#Override public Void call() throws InterruptedException {
for (InputSequence seq: getSequences) {
KnuthMorrisPratt kmp = new KnuthMorrisPratt(seq, substring);
kmp.align();
}
return null;
}
};
}
// High-level class to run the Knuth-Morris-Pratt algorithm.
public class AlignmentFactory {
public void perform() {
KnuthMorrisPrattFactory factory = new KnuthMorrisPrattFactory();
Task<Void> runFactoryTask = factory.runFactory();
runFactoryTask.setOnSucceeded(new EventHandler<WorkerStateEvent>() {
#Override
public void handle(WorkerStateEvent t)
{
// Code to run once runFactory() is completed **successfully**
nextFunction() // also invokes a Task.
}
});
runFactoryTask.setOnFailed(new EventHandler<WorkerStateEvent>() {
#Override
public void handle(WorkerStateEvent t)
{
// Code to run once runFactory() **fails**
}
});
new Thread(runFactoryTask).start();
}
}

Resources