I currently have exception handling being done in an abstract class that all my routes inherit. Something like this:
onException(SocketException,HttpOperationFailedException)
.handled(true)
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.log('retry failed, sending to the route failed coordinator')
.to(routeFailedCoordinator)
Now, I want to do some different things based on different response codes. For all codes other than 200, HttpOperationFailedException get's thrown. For 4XX codes, I want to send the message on to a failed queue and send an email, if enabled for that particular route. For all other errors, I want to go through the retry cycle. Here's what works for the 4XX errors:
onException(HttpOperationFailedException)
.handled(true)
.process { Exchange x ->
HttpOperationFailedException ex = x.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught a HttpOperationFailedException: statusCode=${ex?.statusCode}")
ProducerTemplate producer = x.getContext().createProducerTemplate()
if (ex?.statusCode >= 400 && ex?.statusCode < 500) {
log.debug("Skipping retries ...")
producer.send(routeFailedEndpoint, x)
x.in.body = "Request:\n${x.in.body}\n\nResponse: ${ex.statusCode}\n${ex.responseBody}".toString()
if (sendFailedEmailEnabled)
producer.send('direct:routeFailedEmailHandler', x)
} else {
producer.send(routeFailedRetryEndpoint, x)
}
}.stop()
How do I add code for retrying like in the first code snippet? I tried using nested choice()...when()...otherwise() clauses and kept getting compile errors.
Anyone had to do something similar?
Here is my code with nested choice()..when()..otherwise() clauses:
onException(HttpOperationFailedException)
.handled(true)
.choice()
.when { Exchange x ->
HttpOperationFailedException ex = x.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught a HttpOperationFailedException: statusCode=${ex?.statusCode}")
if (ex?.statusCode >= 400 && ex?.statusCode < 500) {
log.debug("Skipping retries ...")
x.in.body = "Request:\n${x.in.body}\n\nResponse: ${ex.statusCode}\n${ex.responseBody}".toString()
return true // don't retry
}
log.debug("Performing retries ...")
return false // do attempt retries
}.choice()
.when { !sendFailedEmailEnabled }.to(routeFailedEndpoint)
.otherwise()
.multicast().to(routeFailedEndpoint, 'direct:routeFailedEmailHandler').endChoice()
.otherwise()
.getParent().getParent().getParent()
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.to(routeFailedCoordinator)
You would have to have 2 onException blocks:
One onException with the redelivery settings for redelivery attempts
Another onException that handles the exception and send that email and what you want to do.
Use an onWhen on both onException blocks, to return true or false in either situation based on that http status code. The onWhen is executed by Camel to know which of the onException blocks to use (you can have more, but first to return true is used).
You can find more details on the Camel website, or in the Camel in Action book that has a full chapter devoted to error handling.
Thanks, Claus, you pointed me in the right direction.
Basically, as Claus said, use multiple onException blocks, each using an onWhen clause ...
onException(HttpOperationFailedException)
.onWhen(new Predicate() {
public boolean matches(Exchange exchange) {
HttpOperationFailedException ex = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught an HttpOperationFailedException: statusCode=${ex?.statusCode}, processing 4XX error")
return (ex?.statusCode >= 400 && ex?.statusCode < 500)
}
}).handled(true)
.to(routeFailedEndpoint)
.choice()
.when { sendFailedEmailEnabled }.process(prepareFailureEmail).to('direct:routeFailedEmailHandler')
onException(HttpOperationFailedException)
.onWhen(new Predicate() {
public boolean matches(Exchange exchange) {
HttpOperationFailedException ex = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught an HttpOperationFailedException: statusCode=${ex?.statusCode}, processing >=500 error")
return (ex?.statusCode >= 500)
}
}).handled(true)
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.to(routeFailedCoordinator)
Related
EDIT: here https://github.com/wujek-srujek/reactor-retry-test is a repository with all the code.
I have the following Spring WebClient code to POST to a remote server (Kotlin code without imports for brevity):
private val logger = KotlinLogging.logger {}
#Component
class Client(private val webClient: WebClient) {
companion object {
const val maxRetries = 2L
val firstBackOff = Duration.ofSeconds(5L)
val maxBackOff = Duration.ofSeconds(20L)
}
fun send(uri: URI, data: Data): Mono<Void> {
return webClient
.post()
.uri(uri)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(data)
.retrieve()
.toBodilessEntity()
.doOnSubscribe {
logger.info { "Calling backend, uri: $uri" }
}
.retryExponentialBackoff(maxRetries, firstBackOff, maxBackOff, jitter = false) {
logger.debug { "Call to $uri failed, will retry (#${it.iteration()} of max $maxRetries)" }
}
.doOnError {
logger.error { "Call to $uri with $maxRetries retries failed with $it" }
}
.doOnSuccess {
logger.info { "Call to $uri succeeded" }
}
.then()
}
}
(It returns an empty Mono as we don't expect an answer, nor do we care about it.)
I would like to test 2 cases, and one of them is giving me headaches, namely the one in which I want to test that all the retries have been fired. We are using MockWebServer (https://github.com/square/okhttp/tree/master/mockwebserver) and the StepVerifier from reactor-test. (The test for success is easy and doesn't need any virtual time scheduler magic, and works just fine.) Here is the code for the failing one:
#JsonTest
#ContextConfiguration(classes = [Client::class, ClientConfiguration::class])
class ClientITest #Autowired constructor(
private val client: Client
) {
lateinit var server: MockWebServer
#BeforeEach
fun `init mock server`() {
server = MockWebServer()
server.start()
}
#AfterEach
fun `shutdown server`() {
server.shutdown()
}
#Test
fun `server call is retried and eventually fails`() {
val data = Data()
val uri = server.url("/server").uri()
val responseStatus = HttpStatus.INTERNAL_SERVER_ERROR
repeat((0..Client.maxRetries).count()) {
server.enqueue(MockResponse().setResponseCode(responseStatus.value()))
}
StepVerifier.withVirtualTime { client.send(uri, data) }
.expectSubscription()
.thenAwait(Duration.ofSeconds(10)) // wait for the first retry
.expectNextCount(0)
.thenAwait(Duration.ofSeconds(20)) // wait for the second retry
.expectNextCount(0)
.expectErrorMatches {
val cause = it.cause
it is RetryExhaustedException &&
cause is WebClientResponseException &&
cause.statusCode == responseStatus
}
.verify()
// assertions
}
}
I am using withVirtualTime because I don't want the test to take nearly seconds.
The problem is that the test blocks indefinitely. Here is the (simplified) log output:
okhttp3.mockwebserver.MockWebServer : MockWebServer[51058] starting to accept connections
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#1 of max 2)
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#2 of max 2)
As you can see, the first retry works, but the second one blocks. I don't know how to write the test so that it doesn't happen. To make matters worse, the client will actually use jitter, which will make the timing hard to anticipate.
The following test using StepVerifier but without WebClient works fine, even with more retries:
#Test
fun test() {
StepVerifier.withVirtualTime {
Mono
.error<RuntimeException>(RuntimeException())
.retryExponentialBackoff(5,
Duration.ofSeconds(5),
Duration.ofMinutes(2),
jitter = true) {
println("Retrying")
}
.then()
}
.expectSubscription()
.thenAwait(Duration.ofDays(1)) // doesn't matter
.expectNextCount(0)
.expectError()
.verify()
}
Could anybody help me fix the test, and ideally, explain what is wrong?
This is a limitation of virtual time and the way the clock is manipulated in StepVerifier. The thenAwait methods are not synchronized with the underlying scheduling (that happens for example as part of the retryBackoff operation). This means that the operator submits retry tasks at a point where the clock has already been advanced by one day. So the second retry is scheduled for + 1 day and 10 seconds, since the clock is at +1 day. After that, the clock is never advanced so the additional request is never made to MockWebServer.
Your case is made even more complicated in the sense that there is an additional component involved, the MockWebServer, that still works "in real time".
Though advancing the virtual clock is a very quick operation, the response from the MockWebServer still goes through a socket and thus has some amount of latency to the retry scheduling, which makes things more complicated from the test writing perspective.
One possible solution to explore would be to externalize the creation of the VirtualTimeScheduler and tie advanceTimeBy calls to the mockServer.takeRequest(), in a parallel thread.
I am very new to Groovy and this is an old application where the author is no longer with our organization. None of the previous questions that look similar offered any help. The application needs to send a simple message to the user to warn they are missing an entry before they con continue on.
I have made no fewer than 20 changes from flash.message to confirm. Flash causes the application to jump all the way to the user login function. This confirm is giving a crash message: Error 500: Executing action [submitrequest] of controller [SdrmController] caused exception: Runtime error executing action
def submitrequest = {
def testChecker
testChecker = [params.fullExpName].flatten().findAll { it != null }
log.info('testChecker.size = ' + testChecker.size)
if (testChecker.size > 0) {
if (!confirm('Submitting can not be undone, are you sure?')) return
} else {
if (!confirm('You have to pick an expedition. Please return to your Request and pick at least one expedition.')) return
} else {
return
}
}
// rest of long time working code here
}
Expected Result is a simple message to screen tell the user to pick an "Expedition" from a list and then the code returns to the same point so the user can make the change then hit the submit again.
Then full message:
No signature of method: SdrmController.confirm() is applicable for argument types: (java.lang.String) values: [You have to pick an expedition. Please return to your Request and pick at least one expedition.] Possible solutions: notify(), render(java.lang.String)
-- flash.message worked for our situation.
`legChecker = [params.programLeg].flatten().findAll{it!=null}
if(requestInstance.futurePast == "future" && expChecker.size<1) {
flash.message = " you must select a future expedition "
render(view: 'stepstart', model: [....])
return
}`
We have a Jenkins pipeline script that requests approval from the user after all the preparatory steps are complete, before it actually applies the changes.
We want to add a timeout to this step, so that if there is no input from the user then the build is aborted, and are currently working on using this kind of method:
try {
timeout(time: 30, unit: 'SECONDS') {
userInput = input("Apply changes?")
}
} catch(err) {
def user = err.getCauses()[0].getUser()
if (user.toString == 'SYSTEM') { // if it's system it's a timeout
didTimeout = true
echo "Build timed out at approval step"
} else if (userInput == false) { // if not and input is false it's the user
echo "Build aborted by: [${user}]"
}
}
This code is based on examples found here: https://support.cloudbees.com/hc/en-us/articles/226554067-Pipeline-How-to-add-an-input-step-with-timeout-that-continues-if-timeout-is-reached-using-a-default-value and other places online, but I really dislike catching all errors then working out what's caused the exception using err.getCauses()[0].getUser(). I'd rather explicitly catch(TimeoutException) or something like that.
So my question is, what are the actual exceptions that would be thrown by either the approval step timing out or the userInput being false? I haven't been able to find anything in the docs or Jenkins codebase so far about this.
The exception class they are referring to is org.jenkinsci.plugins.workflow.steps.FlowInterruptedException.
Cannot believe that this is an example provided by CloudBeeds.
Most (or probably all?) other exceptions won't even have the getCauses() method which of course would throw another exception then from within the catch block.
Furthermore as you already mentioned it is not a good idea to just catch all exceptions.
Edit:
By the way: Scrolling further down that post - in the comments - there you'll find an example catching a FlowInterruptedException.
Rather old topic, but it helped me, and I've done some more research on it.
As I figured out, FlowInterruptedException's getCauses()[0] has .getUser() only when class of getCauses()[0] is org.jenkinsci.plugins.workflow.support.steps.input.Rejection. It is so only when timeout occured while input was active. But, if timeout occured not in input, getCause()[0] will contain object of another class: org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ExceededTimeout (directly mentioning timeout).
So, I end up with this:
def is_interrupted_by_timeout(org.jenkinsci.plugins.workflow.steps.FlowInterruptedException e, Boolean throw_again=true) {
// if cause is not determined, re-throw exception
try {
def cause = e.getCauses()[0]
def cause_class = cause.getClass()
//echo("cause ${cause} class: ${cause_class}")
if( cause_class == org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ExceededTimeout ) {
// strong detection
return true
} else if( cause_class == org.jenkinsci.plugins.workflow.support.steps.input.Rejection ) {
// indirect detection
def user = cause.getUser()
if( user.toString().equals('SYSTEM') ) {
return true
} else {
return false
}
}
} catch(org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException e_access) {
// here, we may deal with situation when restricted methods are not approved:
// show message and Jengins' admin will copy/paste and execute them only once per Jenkins installation.
error('''
To run this job, Jenkins admin needs to approve some Java methods.
There are two possible ways to do this:
1. (better) run this code in Jenkins Console (URL: /script):
import org.jenkinsci.plugins.scriptsecurity.scripts.ScriptApproval;
def scriptApproval = ScriptApproval.get()
scriptApproval.approveSignature('method org.jenkinsci.plugins.workflow.steps.FlowInterruptedException getCauses')
scriptApproval.approveSignature('method org.jenkinsci.plugins.workflow.support.steps.input.Rejection getUser')
scriptApproval.save()
'''.stripIndent())
return null
}
if( throw_again ) {
throw e
} else {
return null
}
}
And now, you may catch it with something like this:
try {
...
} catch (org.jenkinsci.plugins.workflow.steps.FlowInterruptedException err) {
if( is_interrupted_by_timeout(err) ) {
echo('It is timeout!')
}
}
P.S. I agree, this is bad Jenkins design.
I am in grails 2.3.1 - trying to use the async features.
This is bulk data processing. I am trying to synchronise 2 databases, which involves comparing both and returning a list of 'deltas'. I am trying to speed up the process
The documentation says I can just add a set of closures to a PromiseList and then call onComplete() to check that all the closures have completed. These are my attempts - directly building on "You can also construct a PromiseList manually" in the documentation:
def tasksMemberDeltas = new PromiseList()
pages.each {Integer page ->
tasksMemberDeltas << {findCreateMemberDeltas(page, (page + pageSize) - 1)}
if (page % 30 == 0) {
tasksMemberDeltas.onComplete {
tasksMemberDeltas = new PromiseList()
}
}
Returns:
Error groovy.lang.MissingMethodException:
No signature of method: java.util.ArrayList.onComplete()
In the end I called .get() which calls waitAll. Going into .get() and finding that it did waitAll was my revelation.
So if I have a single task I call:
waitAll finalDeltas
If I have a list I call:
taskFinalDeltas.get()
onComplete() logically relates to a single delta. Not the list. So this works OK:
Promise memberDeleteDeltas = task {
findDeleteAndTagDeltas()
}
memberDeleteDeltas.onError { Throwable err ->
println "An error occured ${err.message}"
}
memberDeleteDeltas.onComplete { result ->
println "Completed create deltas"
}
waitAll(memberDeleteDeltas)
I am trying to figure out how to find out exact reason of (async) HttpRequest (from 'dart:html') failure, and, to be honest, I am a bit lost here.
The onError callback receives only HttpRequestProgressError object, which doesn't have anything useful, and the HttpRequest object itself has "status" set to "0" in case of failure, even console shows "Failed to load resource" with no details.
What I want is to know the exact reason - like "connection refused" or "host name not resolved".
Is this possible at all?
Thank you!
Unfortunately, there is no property to report the error as detailed as you'd like. The reason is that JavaScript doesn't support this.
There are the properties status and statusText on the HttpRequest object (which you could get from your HttpRequestProgressEvent with evt.target, but those represent HTTP status codes. Every other error has the status code 0 - request failed. This could be anything, and the only place to look at is the browser's console, because this is an Exception thrown by the browser.
If your request was synchronous, you could surround the send() with a try-catch. If your request is async, this won't work.
See here
#library('Request');
#import('dart:html');
#import("dart:json");
typedef void RequestHandler(String responseText);
typedef void ErrorHandler(String error);
class ResourceRequest {
XMLHttpRequest request;
RequestHandler _callbackOnSuccess;
ErrorHandler _callbackOnFailure;
ResourceRequest.openGet(String url, RequestHandler callbackOnSuccess, [ErrorHandler callbackOnFailure])
: request = new XMLHttpRequest(),
_callbackOnSuccess = callbackOnSuccess,
_callbackOnFailure = callbackOnFailure {
request.open("GET", url, async : true);
request.on.loadEnd.add((XMLHttpRequestProgressEvent e) => onLoadEnd(e));
}
void send() {
request.send();
}
void onLoadEnd(XMLHttpRequestProgressEvent event) {
if (request.readyState == 4 && request.status == 200) {
_callbackOnSuccess(request.responseText);
} else if (_callbackOnFailure != null) {
_callbackOnFailure(request.statusText);
}
}
}