Hello good reactor folks - I trying to write some reactive code (surprising eh?) and have hit a slight snag. I think it might be a reactor bug, but thought I'd ask here first before posting a bug.
For context: I have a cache Map<Key, Mono<Value>>. A client will request data - we check the cache and use what is essentially computeIfAbsent to place a Mono with .cache() into the cache if nothing has yet been cached for that key. The client then takes the Mono and does magic (not relevant here). Now, the catch is that the population of the cache may encounter transient errors, so we don't want to cache errors - the current request will error but the "next" client, when it subscribes, should trigger the entire pipeline to rerun.
Having read around, for example this closed issue, I settled on Mono#cache(ttlForValue, ttlForError, ttlForEmpty).
This is where things get interesting.
As I don't want to cache error (or empty, but ignore that) signals I found the following documentation promising
If the relevant TTL generator throws any Exception, that exception will be propagated to the Subscriber that encountered the cache miss, but the cache will be immediately cleared, so further Subscribers might re-populate the cache in case the error was transient. In case the source was emitting an error, that error is dropped and added as a suppressed exception. In case the source was emitting a value, that value is dropped.
emphasis mine
So I tried the following (shamelessly cribbing the example in the linked GitHub issue)
public class TestBench {
public static void main(String[] args) throws Exception {
var sampleService = new SampleService();
var producer = Mono.fromSupplier(sampleService::call).cache(
__ -> Duration.ofHours(24),
//don't cache errors
e -> {throw Exceptions.propagate(e);},
//meh
() -> {throw new RuntimeException();});
try {
producer.block();
} catch (RuntimeException e) {
System.out.println("Caught exception : " + e);
}
sampleService.serverAvailable = true;
var result = producer.block();
System.out.println(result);
}
static final class SampleService {
volatile boolean serverAvailable = false;
String call() {
System.out.println("Calling service with availability: " + serverAvailable);
if (!serverAvailable) throw new RuntimeException("Error");
return "Success";
}
}
}
Output
09:12:23.991 [main] DEBUG reactor.util.Loggers$LoggerFactory - Using Slf4j logging framework
Calling service with availability: false
09:12:24.034 [main] ERROR reactor.core.publisher.Operators - Operator called default onErrorDropped
java.lang.RuntimeException: Error
at uk.co.borismorris.testbench.TestBench$SampleService.call(TestBench.java:40)
at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:56)
at reactor.core.publisher.MonoCacheTime.subscribe(MonoCacheTime.java:123)
at reactor.core.publisher.Mono.block(Mono.java:1474)
at uk.co.borismorris..boris.testbench.TestBench.main(TestBench.java:26)
Caught exception : reactor.core.Exceptions$BubblingException: java.lang.RuntimeException: Error
Exception in thread "main" java.lang.RuntimeException: Error
at uk.co.borismorris.testbench.TestBench$SampleService.call(TestBench.java:40)
at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:56)
at reactor.core.publisher.MonoCacheTime.subscribe(MonoCacheTime.java:123)
at reactor.core.publisher.Mono.block(Mono.java:1474)
at uk.co.borismorris.testbench.TestBench.main(TestBench.java:26)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:93)
at reactor.core.publisher.Mono.block(Mono.java:1475)
at uk.co.borismorris.testbench.TestBench.main(TestBench.java:31)
Well, that didn't work - the error is cached and the second subscriber just sees the same error.
Looking at the code the cause is obvious
Duration ttl = null;
try {
ttl = main.ttlGenerator.apply(signal);
}
catch (Throwable generatorError) {
signalToPropagate = Signal.error(generatorError);
STATE.set(main, signalToPropagate); //HERE
if (signal.isOnError()) {
//noinspection ThrowableNotThrown
Exceptions.addSuppressed(generatorError, signal.getThrowable());
}
}
The STATE is set to the error signal, not cleared at all. But this isn't the whole story,
the reason for the code not clearing the cache is below this block
if (ttl != null) {
main.clock.schedule(main, ttl.toMillis(), TimeUnit.MILLISECONDS);
}
else {
//error during TTL generation, signal != updatedSignal, aka dropped
if (signal.isOnNext()) {
Operators.onNextDropped(signal.get(), currentContext());
}
else if (signal.isOnError()) {
Operators.onErrorDropped(signal.getThrowable(), currentContext());
}
//immediate cache clear
main.run();
}
In this case ttl == null because the generation of the ttl threw an Exception. The signal is an error so that branch is entered and Operators.onErrorDropped is called
public static void onErrorDropped(Throwable e, Context context) {
Consumer<? super Throwable> hook = context.getOrDefault(Hooks.KEY_ON_ERROR_DROPPED,null);
if (hook == null) {
hook = Hooks.onErrorDroppedHook;
}
if (hook == null) {
log.error("Operator called default onErrorDropped", e);
throw Exceptions.bubble(e);
}
hook.accept(e);
}
So here we can see that if there is no onError hook in the context and no default set then throw Exceptions.bubble(e) is called and the code in MonoCacheTime returns early, failing to call main.run(). Hence the error stays cached indefinitely as there is no TTL!
The following code fixes that problem
public class TestBench {
private static final Logger logger = LoggerFactory.getLogger(TestBench.class);
private static final Consumer<Throwable> onErrorDropped = e -> logger.error("Dropped", e);
static {
//add default hook
Hooks.onErrorDropped(onErrorDropped);
}
public static void main(String[] args) throws Exception {
var sampleService = new SampleService();
var producer = Mono.fromSupplier(sampleService::call).cache(
__ -> Duration.ofHours(24),
//don't cache errors
e -> {throw Exceptions.propagate(e);},
//meh
() -> {throw new RuntimeException();});
try {
producer.block();
} catch (RuntimeException e) {
System.out.println("Caught exception : " + e);
}
sampleService.serverAvailable = true;
var result = producer.block();
System.out.println(result);
}
static final class SampleService {
volatile boolean serverAvailable = false;
String call() {
System.out.println("Calling service with availability: " + serverAvailable);
if (!serverAvailable) throw new RuntimeException("Error");
return "Success";
}
}
}
But this adds a global Hook, which isn't ideal. The code hints at the ability to add per-pipeline hooks, but I cannot figure out how to do that. The following works, but is obviously a hack
.subscriberContext(ctx -> ctx.put("reactor.onErrorDropped.local", onErrorDropped))
Questions
Is the above a bug, should the absence of a onErrorDropped hook cause errors to be cached indefinitely?
Is there a way to set the onErrorDropped hook in the subscriberContext rather than globally?
Follow up
From the code; it seems that returning null from a TTL generator function is supported and has the same behaviour when the signal is immediately cleared. In the case where it isn't, the subscriber sees the original error rather than the error from the TTL generator and a suppressed error - which seems perhaps neater
public static void main(String[] args) throws Exception {
var sampleService = new SampleService();
var producer = Mono.fromSupplier(sampleService::call).cache(
__ -> Duration.ofHours(24),
//don't cache errors
e -> null,
//meh
() -> null);
try {
producer.block();
} catch (RuntimeException e) {
System.out.println("Caught exception : " + e);
}
sampleService.serverAvailable = true;
var result = producer.block();
System.out.println(result);
}
Is this behaviour supported? Should it be documented?
You've indeed found a bug! And I think the documentation can also be improved for this variant of cache:
The focus on how it deals with exceptions inside TTL Function is probably misleading
There should be a documented straightforward way of "ignoring" a category of signals in the source (which is you case: you want subsequent subscribers to "retry" when the source is erroring).
The behavior is bugged due to the use of onErrorDropped (which defaults to throwing the dropped exception, thus preventing the main.run() state reset).
Unfortunately, the tests use StepVerifier#verifyThenAssertThat(), which set an onErrorDropped hook, so that last bug was never identified.
Returning null in the TTL function is not working better because the same bug happens, but this time with the original source exception being dropped/bubbled.
But there is an ideal semantic for propagating an error to the first subscriber and let the second subscriber retry: to return Duration.ZERO in the ttl Function. This is undocumented, but works right now:
IllegalStateException exception = new IllegalStateException("boom");
AtomicInteger count = new AtomicInteger();
Mono<Integer> source = Mono.fromCallable(() -> {
int c = count.incrementAndGet();
if (c == 1) throw exception;
return c;
});
Mono<Integer> cache = source.cache(v -> Duration.ofSeconds(10),
e -> Duration.ZERO,
() -> Duration.ofSeconds(10));
assertThat(cache.retry().block()).isEqualTo(2);
I'll open an issue to fix the state reset bug and focus the javadoc on the above solution, while moving the bit dealing with throwing TTL Functions in a separate shorter paragraph at the end.
edit: https://github.com/reactor/reactor-core/issues/1783
Related
When multiple onErrorContinue added to the pipeline to handle specific type of exception thrown from flatMap, the exception handling is not working as expected.
The below code, I expect, the elements 1 to 6 should be dropped and element 7 to 10 should be consumed by the subscriber.
public class FlatMapOnErrorContinueExample {
public static void main(String[] args) {
Flux.just(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
.flatMap(number -> {
if (number <= 3) {
return Mono.error(new NumberLesserThanThree("Number is lesser than 3"));
} else if (number > 3 && number <= 6) {
return Mono.error(new NumberLesserThanSixButGretherThan3("Number is grether than 6"));
} else {
return Mono.just(number);
}
})
.onErrorContinue(NumberLesserThanThree.class,
(throwable, object) -> System.err.println("Exception: Dropping the element because it is lesser than 3"))
.onErrorContinue(NumberLesserThanSixButGretherThan3.class,
(throwable, object) -> System.err.println("Exception: Dropping the element because it is lesser than 6 but grether than 3"))
.onErrorContinue((throwable, object) ->
System.err.println("Exception: " + throwable.getMessage()))
.subscribe(number -> System.out.println("number is " + number),
error -> System.err.println("Exception in Subscription " + error.getMessage()));
}
public static class NumberLesserThanThree extends RuntimeException {
public NumberLesserThanThree(final String msg) {
super(msg);
}
}
public static class NumberLesserThanSixButGretherThan3 extends RuntimeException {
public NumberLesserThanSixButGretherThan3(final String msg) {
super(msg);
}
}
}
Here is the output what I am getting:
Exception: Dropping the element because it is lesser than 3
Exception: Dropping the element because it is lesser than 3
Exception: Dropping the element because it is lesser than 3
Exception in Subscription Number is grether than 6
Question: Why the 2nd onErrorContinue is not called but the exception send to subscriber?
Additional Note:
if i remove 1st and 2nd onErrorContinue, then all exception are handled by 3rd onErrorContinue. I could use this approach to receive all exception and check for the type of exception and proceed with handling. However, I would like to make it cleaner exception handling rather than adding if..else block.
How this question is different from Why does Thread.sleep() trigger the subscription to Flux.interval()?
1) This question about exception handling and the order of exception handling; The other question is about processing elements in parallel and making the main thread waiting for the all the element processing complete
3) This question dont have any concern about threading, even if add Thread.sleep(10000) after . subscribe, there is no change in behaviour.
This again comes down to the unusual behaviour of onErrorContinue. It breaks the rule in that it doesn't "catch" errors and then change the behaviour downstream as a result, it actually allows supporting operators to "look ahead" at it and behave accordingly, thus changing the result upstream.
This is weird, and leads to some behaviour that's not immediately obvious, such as is the case here. As far as I'm aware, all supporting operators only look ahead to the next onErrorContinue operator, rather than recursively searching ahead for all such operators. Instead, they will evaluate the predicate of the next onErrorContinue (in this case whether it's of a certain type), and then behave accordingly - either invoking the handler if the predicate returns true, or throwing the error downstream if not. (There's no case where it will then move onto the next onErrorContinue operator, then the next, until a predicate is matched.)
Clearly this is a contrived example - but because of these idiosyncrasies, I'd almost always recommend avoiding onErrorContinue. There's two "normal" ways that can happen where flatMap() is involved:
If flatMap() has an "inner reactive chain" in it, that is it calls another method or series of methods that return a publisher - then just use onErrorResume() at the end of the flatMap() call to handle those errors. You can chain onErrorResume() since it works with downstream, not upstream operators. This is by far the most common case.
If flatMap() is an imperative collection of if / else that's returning different publishers such as it is here and you want to / have to keep the imperative style, throw exceptions instead of using Mono.error(), and catch as appropriate, returning Mono.empty() in case of an error:
.flatMap(number -> {
try {
if (number <= 3) {
throw new NumberLessThanThree();
} else if (number <= 6) {
throw new NumberLessThanSixButGreaterThan3();
} else {
return Mono.just(number);
}
}
catch(NumberLessThanThree ex) {
//Handle it
return Mono.empty();
}
catch(NumberLessThanSixButGreaterThan3 ex) {
//As above
}
})
In general, using one of these two approaches will make it much easier to reason about what's going on.
(For the sake of completeness after reading the comments - this isn't anything to do with the reactive chain being unable to complete before the main thread exits.)
Type mismatch: cannot convert from Mono to Mono. Wants to add error messge to B then i wants to return Mono of B.
public Mono<B> case1(){
A a = new A();
a.setName("srihari");
B b = new B();
a.setName("G SRIHARI");
Mono<A> mono = Mono.error(new RuntimeException("HARISRI"));
return mono.onErrorResume(d->{
b.setErrorMessage(d.getMessage());
return Mono.just(b);
}).subscribeOn(Schedulers.elastic());
As you can see in the signature of onErrorResume(), the fallback function can only result in a Mono of the same type as the source Mono. This is for good reason.
What if you instead had Mono<A> mono = Mono.just(new A());. Now, no error will be thrown, so the fallback function will never be invoked. Therefore, you still have a Mono<A>, not a Mono<B>.
This isn't really anything special with Reactor. Consider the same kind of flow with a try catch:
public B foo() {
try {
// throw new RuntimeException();
return new A();
} catch (Exception e) {
return new B();
}
}
Obviously, this code does not compile because if an error is not thrown, A will be returned, but the method must return B. This is essentially what you are trying to write but with Reactor.
I am trying to create a Dart function that essentially wraps other functions with some boilerplate error handling code, and otherwise returns the value returned by the original function. A key requirement is that it should accept functions with multiple different return types, while avoiding duplicating the common error handling logic across multiple different functions. I found one approach that seems to work by using the dynamic type, except that the compiler is not able to detect type mismatches, so they are only caught at runtime.
Is there a better way to accomplish what I'm aiming for here, and particularly in a way that catches type mismatches at compile time?
Below is a simplified example of my code, where the functions compile fine, but at runtime getAString will raise an error Dart Error: Unhandled exception: type 'List<String>' is not a subtype of type 'String'
/// Signature of API function calls
typedef APIFunctionCall = dynamic Function();
dynamic doWithErrorHandling(APIFunctionCall fn, {retries: 2}) async {
for (int attempts = 0; attempts < retries + 1; attempts++) {
try {
return await fn();
}
on Exception catch (e) {
print(
"This is just an example; actual function does a bunch of more specific error handling.");
}
}
}
Future<String> getAString() async {
// Want a function that can support multiple return types but detect type errors
String doesReturnAString = await doWithErrorHandling(() async => 'hello world'); // This runs fine
String doesntReturnAString = await doWithErrorHandling(() async => <String>['hello', 'world']); // This throws an Error
return doesntReturnAString;
}
You can abstract over the return type using a type parameter:
Future<T> doWithErrorHandling<T>(Future<T> fn(), {int retries = 2}) async {
do {
try {
return await fn();
} catch (e) {
// record error.
}
retries--;
} while (retries >= 0);
return null; // or whatever.
}
With that, you can call with any function. In most cases, the type argument can be inferred from the static type of the argument function, or from the type expected by the surrounding context, but if not, you can write it yourself.
Future<String> getAString() async {
String doesReturnAString = await doWithErrorHandling(() async => 'hello world');
// The next line has a compile-time type error!
String doesntReturnAString = await doWithErrorHandling(() async => <String>['hello', 'world']);
return doesntReturnAString;
}
(As an unrelated hint, you should never catch Exception. Dart errors do not implement Exception, they implement Error. Exception is a meaningless marker interface used by some thrown objects that the user is intended to catch and handle, but in that case, you should be catching the particular exception, like on FormatException, not the plain Exception. So, general rule: Never write on Exception).
In my Production code, I am getting errors in my logs when a Mono times out.
I have managed to recreate these errors with the following code:
#Test
public void testScheduler() {
Mono<String> callableMethod1 = callableMethod();
callableMethod1.block();
Mono<String> callableMethod2 = callableMethod();
callableMethod2.block();
}
private Mono<String> callableMethod() {
return Mono.fromCallable(() -> {
Thread.sleep(60);
return "Success";
})
.subscribeOn(Schedulers.elastic())
.timeout(Duration.ofMillis(50))
.onErrorResume(throwable -> Mono.just("Timeout"));
}
In the Mono.fromCallable I am making a blocking call using a third-party library. When this call times out, I get errors similar to
reactor.core.publisher.Operators - Operator called default onErrorDropped
reactor.core.publisher.Operators - Scheduler worker in group main failed with an uncaught exception
These errors also seem to be intermittent, sometimes when I run the code provided I get no errors at all. However when I repeat the call in a loop of say 10, I consistently get them.
Question: Why does this error happen?
Answer:
When the duration given to the timeout() operator has passed, it throws a TimeoutException. That results in the following outcomes:
An onError signal is sent to the main reactive chain. As a result, the main execution is resumed and the process moves on (i.e., onErrorResume() is executed).
Shortly after outcome #1, the async task defined within fromCallable() is interrupted, which triggers a 2nd exception (InterruptedException). The main reactive chain can no longer handle this InterruptedException because the TimeoutException happened first and already caused the main reactive chain to resume (Note: this behavior of not generating a 2nd onError signal conforms with the Reactive Stream Specification -> Publisher #7).
Since the 2nd exception (InterruptedException) can't be handled gracefully by the main chain, Reactor logs it at error level to let us know an unexpected exception occurred.
Question: How do I get rid of them?
Short Answer: Use Hooks.onErrorDropped() to change the log level:
Logger logger = Logger.getLogger(this.getClass().getName());
#Test
public void test() {
Hooks.onErrorDropped(error -> {
logger.log(Level.WARNING, "Exception happened:", error);
});
Mono.fromCallable(() -> {
Thread.sleep(60);
return "Success";
})
.subscribeOn(Schedulers.elastic())
.timeout(Duration.ofMillis(50))
.onErrorResume(throwable -> Mono.just("Timeout"))
.doOnSuccess(result -> logger.info("Result: " + result))
.block();
}
Long Answer: If your use-case allows, you could handle the exception happening within fromCallable() so that the only exception affecting the main chain is the TimeoutException. In that case, the onErrorDropped() wouldn't happen in the first place.
#Test
public void test() {
Mono.fromCallable(() -> {
try {
Thread.sleep(60);
} catch (InterruptedException ex) {
//release resources, rollback actions, etc
logger.log(Level.WARNING, "Something went wrong...", ex);
}
return "Success";
})
.subscribeOn(Schedulers.elastic())
.timeout(Duration.ofMillis(50))
.onErrorResume(throwable -> Mono.just("Timeout"))
.doOnSuccess(result -> logger.info("Result: " + result))
.block();
}
Extra References:
https://tacogrammer.com/onerrordropped-explained/
https://medium.com/#kalpads/configuring-timeouts-in-spring-reactive-webclient-4bc5faf56411
I needed to display some validation messages received from the service each in its proper place, and I solved it putting the messages in an exception:
class InvalidInputException extends RuntimeException {
def errors
InvalidInputException(String s) {
super(s)
}
InvalidInputException(String s, errors) {
super(s)
this.errors = errors
}
}
That way, I could throw the exception sending the errors:
if (errors) {
throw new InvalidInputException("There were some errors", errors)
}
.. and then I deal with the errors later in the controller, after catching the exception:
...
catch (InvalidInputException e) {
if (e.errors) {
// set them up to display appropriately
}
// render the view
}
Now, I've read somewhere that Groovy's exceptions can cost too much, so... is this too bad?
What problems may I encounter putting additional data in an Exception?
It's much easier than fiddling with returned error messages, and the code is much shorter.
If you are concerned about the performance of exceptions in Java, I suggest you to look at this other question.
If you not create a exception, the other possibility is to make your service return an object that represents the result of this flow. Something like:
class MyServiceResult {
List<String> errorCodes = [] //codes for i18n
void addErrorCode(String errorCode) {
errorCodes << errorCode //add error to list
}
boolean isValid() {
return (!(errorCodes.size() > 0)) //if we have errors then isn't valid.
}
List<String> getErrorCodes() {
return errorCodes.asImmutable()
}
}
And just use it in your service method
class MyService {
MyServiceResult someMethod() {
MyServiceResult result = new MyServiceResult()
...
result.addErrorCode('some.key.here')
...
return result
}
}
class MyController {
def myService
def action() {
MyServiceResult result = myService.someMethod()
if(!result.isValid()) {
//handle errors
}
}
}
But it's important to say that it can be 2x slower than creating an exception. You can check all details in this post.