I am in grails 2.3.1 - trying to use the async features.
This is bulk data processing. I am trying to synchronise 2 databases, which involves comparing both and returning a list of 'deltas'. I am trying to speed up the process
The documentation says I can just add a set of closures to a PromiseList and then call onComplete() to check that all the closures have completed. These are my attempts - directly building on "You can also construct a PromiseList manually" in the documentation:
def tasksMemberDeltas = new PromiseList()
pages.each {Integer page ->
tasksMemberDeltas << {findCreateMemberDeltas(page, (page + pageSize) - 1)}
if (page % 30 == 0) {
tasksMemberDeltas.onComplete {
tasksMemberDeltas = new PromiseList()
}
}
Returns:
Error groovy.lang.MissingMethodException:
No signature of method: java.util.ArrayList.onComplete()
In the end I called .get() which calls waitAll. Going into .get() and finding that it did waitAll was my revelation.
So if I have a single task I call:
waitAll finalDeltas
If I have a list I call:
taskFinalDeltas.get()
onComplete() logically relates to a single delta. Not the list. So this works OK:
Promise memberDeleteDeltas = task {
findDeleteAndTagDeltas()
}
memberDeleteDeltas.onError { Throwable err ->
println "An error occured ${err.message}"
}
memberDeleteDeltas.onComplete { result ->
println "Completed create deltas"
}
waitAll(memberDeleteDeltas)
Related
I am very new to Groovy and this is an old application where the author is no longer with our organization. None of the previous questions that look similar offered any help. The application needs to send a simple message to the user to warn they are missing an entry before they con continue on.
I have made no fewer than 20 changes from flash.message to confirm. Flash causes the application to jump all the way to the user login function. This confirm is giving a crash message: Error 500: Executing action [submitrequest] of controller [SdrmController] caused exception: Runtime error executing action
def submitrequest = {
def testChecker
testChecker = [params.fullExpName].flatten().findAll { it != null }
log.info('testChecker.size = ' + testChecker.size)
if (testChecker.size > 0) {
if (!confirm('Submitting can not be undone, are you sure?')) return
} else {
if (!confirm('You have to pick an expedition. Please return to your Request and pick at least one expedition.')) return
} else {
return
}
}
// rest of long time working code here
}
Expected Result is a simple message to screen tell the user to pick an "Expedition" from a list and then the code returns to the same point so the user can make the change then hit the submit again.
Then full message:
No signature of method: SdrmController.confirm() is applicable for argument types: (java.lang.String) values: [You have to pick an expedition. Please return to your Request and pick at least one expedition.] Possible solutions: notify(), render(java.lang.String)
-- flash.message worked for our situation.
`legChecker = [params.programLeg].flatten().findAll{it!=null}
if(requestInstance.futurePast == "future" && expChecker.size<1) {
flash.message = " you must select a future expedition "
render(view: 'stepstart', model: [....])
return
}`
I am trying to learn Reactor but I am having a lot of trouble with it. I wanted to do a very simple proof of concept where I simulate calling a slow down stream service 1 or more times. If you use reactor and stream the response the caller doesn't have to wait for all the results.
So I created a very simple controller but it is not behaving like I expect. When the delay is "inside" my flatMap (inside the method I call) the response is not returned until everything is complete. But when I add a delay after the flatMap the data is streamed.
Why does this code result in a stream of JSON
#GetMapping(value = "/test", produces = { MediaType.APPLICATION_STREAM_JSON_VALUE })
Flux<HashMap<String, Object>> customerCards(#PathVariable String customerId) {
Integer count = service.getCount(customerId);
return Flux.range(1, count).
flatMap(k -> service.doRestCall(k)).delayElements(Duration.ofMillis(5000));
}
But this does not
#GetMapping(value = "/test2", produces = { MediaType.APPLICATION_STREAM_JSON_VALUE })
Flux<HashMap<String, Object>> customerCards(#PathVariable String customerId) {
Integer count = service.getCount(customerId);
return Flux.range(1, count).
flatMap(k -> service.doRestCallWithDelay(k));
}
It think I am missing something very basic of the reactor API. On that note. can anyone point to a good book or tutorial on reactor? I can't seem to find anything good to learn this.
Thanks
The code inside the flatMap runs on the main thread (that is the thread the controller runs). As a result the whole process is blocked and the method doesnt return immediately. Have in mind that Reactor doesnt impose a particular threading model.
On the contrary, according to the documentation, in the delayElements method signals are delayed and continue on the parallel default Scheduler. That means that the main thread is not blocked and returns immediately.
Here are two corresponding examples:
Blokcing code:
Flux.range(1, 500)
.map(i -> {
//blocking code
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " - Item : " + i);
return i;
})
.subscribe();
System.out.println("main completed");
Result:
main - Item : 1
main - Item : 2
main - Item : 3
...
main - Item : 500
main completed
Non-blocking code:
Flux.range(1, 500)
.delayElements(Duration.ofSeconds(1))
.subscribe(i -> {
System.out.println(Thread.currentThread().getName() + " - Item : " + i);
});
System.out.println("main Completed");
//sleep main thread in order to be able to print the println of the flux
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
e.printStackTrace();
}
Result:
main Completed
parallel-1 - Item : 1
parallel-2 - Item : 2
parallel-3 - Item : 3
parallel-4 - Item : 4
...
Here is the project reactor reference guide
"delayElements" method only delay flux element by a given duration, see javadoc for more details
I think you should post details about methods "service.doRestCallWithDelay(k)" and "service.doRestCall(k)" if you need more help.
I currently have exception handling being done in an abstract class that all my routes inherit. Something like this:
onException(SocketException,HttpOperationFailedException)
.handled(true)
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.log('retry failed, sending to the route failed coordinator')
.to(routeFailedCoordinator)
Now, I want to do some different things based on different response codes. For all codes other than 200, HttpOperationFailedException get's thrown. For 4XX codes, I want to send the message on to a failed queue and send an email, if enabled for that particular route. For all other errors, I want to go through the retry cycle. Here's what works for the 4XX errors:
onException(HttpOperationFailedException)
.handled(true)
.process { Exchange x ->
HttpOperationFailedException ex = x.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught a HttpOperationFailedException: statusCode=${ex?.statusCode}")
ProducerTemplate producer = x.getContext().createProducerTemplate()
if (ex?.statusCode >= 400 && ex?.statusCode < 500) {
log.debug("Skipping retries ...")
producer.send(routeFailedEndpoint, x)
x.in.body = "Request:\n${x.in.body}\n\nResponse: ${ex.statusCode}\n${ex.responseBody}".toString()
if (sendFailedEmailEnabled)
producer.send('direct:routeFailedEmailHandler', x)
} else {
producer.send(routeFailedRetryEndpoint, x)
}
}.stop()
How do I add code for retrying like in the first code snippet? I tried using nested choice()...when()...otherwise() clauses and kept getting compile errors.
Anyone had to do something similar?
Here is my code with nested choice()..when()..otherwise() clauses:
onException(HttpOperationFailedException)
.handled(true)
.choice()
.when { Exchange x ->
HttpOperationFailedException ex = x.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught a HttpOperationFailedException: statusCode=${ex?.statusCode}")
if (ex?.statusCode >= 400 && ex?.statusCode < 500) {
log.debug("Skipping retries ...")
x.in.body = "Request:\n${x.in.body}\n\nResponse: ${ex.statusCode}\n${ex.responseBody}".toString()
return true // don't retry
}
log.debug("Performing retries ...")
return false // do attempt retries
}.choice()
.when { !sendFailedEmailEnabled }.to(routeFailedEndpoint)
.otherwise()
.multicast().to(routeFailedEndpoint, 'direct:routeFailedEmailHandler').endChoice()
.otherwise()
.getParent().getParent().getParent()
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.to(routeFailedCoordinator)
You would have to have 2 onException blocks:
One onException with the redelivery settings for redelivery attempts
Another onException that handles the exception and send that email and what you want to do.
Use an onWhen on both onException blocks, to return true or false in either situation based on that http status code. The onWhen is executed by Camel to know which of the onException blocks to use (you can have more, but first to return true is used).
You can find more details on the Camel website, or in the Camel in Action book that has a full chapter devoted to error handling.
Thanks, Claus, you pointed me in the right direction.
Basically, as Claus said, use multiple onException blocks, each using an onWhen clause ...
onException(HttpOperationFailedException)
.onWhen(new Predicate() {
public boolean matches(Exchange exchange) {
HttpOperationFailedException ex = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught an HttpOperationFailedException: statusCode=${ex?.statusCode}, processing 4XX error")
return (ex?.statusCode >= 400 && ex?.statusCode < 500)
}
}).handled(true)
.to(routeFailedEndpoint)
.choice()
.when { sendFailedEmailEnabled }.process(prepareFailureEmail).to('direct:routeFailedEmailHandler')
onException(HttpOperationFailedException)
.onWhen(new Predicate() {
public boolean matches(Exchange exchange) {
HttpOperationFailedException ex = exchange.getProperty(Exchange.EXCEPTION_CAUGHT, HttpOperationFailedException.class)
log.debug("Caught an HttpOperationFailedException: statusCode=${ex?.statusCode}, processing >=500 error")
return (ex?.statusCode >= 500)
}
}).handled(true)
.maximumRedeliveries(settings.maximumRedeliveries)
.redeliverDelay(settings.redeliverDelay)
.useCollisionAvoidance()
.collisionAvoidanceFactor(settings.collisionAvoidanceFactor)
.onRedelivery(redeliveryProcessor)
.to(routeFailedCoordinator)
In My Grails service, there is a part of a method I wish to run asynchronously.
Following, the doc for 2.3.x http://grails.org/doc/2.3.0.M1/guide/async.html
I do
public class MyService {
public void myMethod() {
Promise p = task {
// Long running task
}
p.onError { Throwable err ->
println "An error occured ${err.message}"
}
p.onComplete { result ->
println "Promise returned $result"
}
// block until result is called
def result = p.get()
}
}
However, I want to execute mine without any blocking. The p.get() method blocks. How do I execute the promise without any sort of blocking. I don't care if myMethod() returns, it is a kinda of fire and forget method.
So, according to the documentation if you don't call .get() or .waitAll() but rather just make use of onComplete you can run your task without blocking the current thread.
Here is a very silly example that I worked up in the console to as a proof of concept.
import static grails.async.Promises.*
def p = task {
// Long running task
println 'Off to do something now ...'
Thread.sleep(5000)
println '... that took 5 seconds'
return 'the result'
}
p.onError { Throwable err ->
println "An error occured ${err.message}"
}
p.onComplete { result ->
println "Promise returned $result"
}
println 'Just to show some output, and prove the task is running in the background.'
Running the above example gives you the following output:
Off to do something now ...
Just to show some output, and prove the task is running in the background.
... that took 5 seconds
Promise returned the result
I am trying to clear out a collection and update it at the same time. It has children and finding the current items in the collection and deleting them asynchronously would save me a lot of time.
Step 1. Find all the items in the collection.
Step 2. Once I know what the items are, fork a process to delete them.
def memberRedbackCriteria = MemberRedback.createCriteria()
// #1 Find all the items in the collection.
def oldList = memberRedbackCriteria.list { fetchMode("memberCategories", FetchMode.EAGER) }
// #2 Delete them.
Promise deleteOld = task {
oldList.each { MemberRedback rbMember ->
rbMember.memberCategories.clear()
rbMember.delete()
}
}
The error message is: Illegal attempt to associate a collection with two open sessions
I am guessing that because I find the items, then fork, this creates a new session so that the collection is built before forking and a new session is used to delete the items.
I need to collect the items in the current thread, otherwise I am not sure what the state would be.
Note that using one async task for all the deletions is effectively running all the delete operations in series in a single thread. Assuming your database can handle multiple connections and concurrent modification of a table, you could parallelize the deletions by using a PromiseList, as in the following (note untested code follows).
def deletePromises = new PromiseList()
redbackIds.each { Long rbId ->
deletePromises << MemberRedback.async.task {
withTransaction {
def memberRedbackCriteria = createCriteria()
MemberRedback memberRedback = memberRedbackCriteria.get {
idEq(rbId)
fetchMode("memberCategories", FetchMode.EAGER) }
memberRedback.memberCategories.clear()
memberRedback.delete()
}
}
}
deletePromises.onComplete { List results ->
// do something with the results, if you want
}
deletePromises.onError { Throwable err ->
// do something with the error
}
Found a solution. Put the ids into a list and collect them as part of the async closure.
Note also that you cannot reuse the criteria as per http://jira.grails.org/browse/GRAILS-1967
// #1 find the ids
def redbackIds = MemberRedback.executeQuery(
'select mr.id from MemberRedback mr',[])
// #2 Delete them.
Promise deleteOld = task {
redbackIds.each { Long rbId ->
def memberRedbackCriteria = MemberRedback.createCriteria()
MemberRedback memberRedback = memberRedbackCriteria.get {
idEq(rbId)
fetchMode("memberCategories", FetchMode.EAGER) }
memberRedback.memberCategories.clear()
memberRedback.delete()
}
}
deleteOld.onError { Throwable err ->
println "deleteAllRedbackMembers An error occured ${err.message}"
}