run something async in Grails 2.3 - grails

In My Grails service, there is a part of a method I wish to run asynchronously.
Following, the doc for 2.3.x http://grails.org/doc/2.3.0.M1/guide/async.html
I do
public class MyService {
public void myMethod() {
Promise p = task {
// Long running task
}
p.onError { Throwable err ->
println "An error occured ${err.message}"
}
p.onComplete { result ->
println "Promise returned $result"
}
// block until result is called
def result = p.get()
}
}
However, I want to execute mine without any blocking. The p.get() method blocks. How do I execute the promise without any sort of blocking. I don't care if myMethod() returns, it is a kinda of fire and forget method.

So, according to the documentation if you don't call .get() or .waitAll() but rather just make use of onComplete you can run your task without blocking the current thread.
Here is a very silly example that I worked up in the console to as a proof of concept.
import static grails.async.Promises.*
def p = task {
// Long running task
println 'Off to do something now ...'
Thread.sleep(5000)
println '... that took 5 seconds'
return 'the result'
}
p.onError { Throwable err ->
println "An error occured ${err.message}"
}
p.onComplete { result ->
println "Promise returned $result"
}
println 'Just to show some output, and prove the task is running in the background.'
Running the above example gives you the following output:
Off to do something now ...
Just to show some output, and prove the task is running in the background.
... that took 5 seconds
Promise returned the result

Related

Test using StepVerifier blocks when using Spring WebClient with retry

EDIT: here https://github.com/wujek-srujek/reactor-retry-test is a repository with all the code.
I have the following Spring WebClient code to POST to a remote server (Kotlin code without imports for brevity):
private val logger = KotlinLogging.logger {}
#Component
class Client(private val webClient: WebClient) {
companion object {
const val maxRetries = 2L
val firstBackOff = Duration.ofSeconds(5L)
val maxBackOff = Duration.ofSeconds(20L)
}
fun send(uri: URI, data: Data): Mono<Void> {
return webClient
.post()
.uri(uri)
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(data)
.retrieve()
.toBodilessEntity()
.doOnSubscribe {
logger.info { "Calling backend, uri: $uri" }
}
.retryExponentialBackoff(maxRetries, firstBackOff, maxBackOff, jitter = false) {
logger.debug { "Call to $uri failed, will retry (#${it.iteration()} of max $maxRetries)" }
}
.doOnError {
logger.error { "Call to $uri with $maxRetries retries failed with $it" }
}
.doOnSuccess {
logger.info { "Call to $uri succeeded" }
}
.then()
}
}
(It returns an empty Mono as we don't expect an answer, nor do we care about it.)
I would like to test 2 cases, and one of them is giving me headaches, namely the one in which I want to test that all the retries have been fired. We are using MockWebServer (https://github.com/square/okhttp/tree/master/mockwebserver) and the StepVerifier from reactor-test. (The test for success is easy and doesn't need any virtual time scheduler magic, and works just fine.) Here is the code for the failing one:
#JsonTest
#ContextConfiguration(classes = [Client::class, ClientConfiguration::class])
class ClientITest #Autowired constructor(
private val client: Client
) {
lateinit var server: MockWebServer
#BeforeEach
fun `init mock server`() {
server = MockWebServer()
server.start()
}
#AfterEach
fun `shutdown server`() {
server.shutdown()
}
#Test
fun `server call is retried and eventually fails`() {
val data = Data()
val uri = server.url("/server").uri()
val responseStatus = HttpStatus.INTERNAL_SERVER_ERROR
repeat((0..Client.maxRetries).count()) {
server.enqueue(MockResponse().setResponseCode(responseStatus.value()))
}
StepVerifier.withVirtualTime { client.send(uri, data) }
.expectSubscription()
.thenAwait(Duration.ofSeconds(10)) // wait for the first retry
.expectNextCount(0)
.thenAwait(Duration.ofSeconds(20)) // wait for the second retry
.expectNextCount(0)
.expectErrorMatches {
val cause = it.cause
it is RetryExhaustedException &&
cause is WebClientResponseException &&
cause.statusCode == responseStatus
}
.verify()
// assertions
}
}
I am using withVirtualTime because I don't want the test to take nearly seconds.
The problem is that the test blocks indefinitely. Here is the (simplified) log output:
okhttp3.mockwebserver.MockWebServer : MockWebServer[51058] starting to accept connections
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#1 of max 2)
Calling backend, uri: http://localhost:51058/server
MockWebServer[51058] received request: POST /server HTTP/1.1 and responded: HTTP/1.1 500 Server Error
Call to http://localhost:51058/server failed, will retry (#2 of max 2)
As you can see, the first retry works, but the second one blocks. I don't know how to write the test so that it doesn't happen. To make matters worse, the client will actually use jitter, which will make the timing hard to anticipate.
The following test using StepVerifier but without WebClient works fine, even with more retries:
#Test
fun test() {
StepVerifier.withVirtualTime {
Mono
.error<RuntimeException>(RuntimeException())
.retryExponentialBackoff(5,
Duration.ofSeconds(5),
Duration.ofMinutes(2),
jitter = true) {
println("Retrying")
}
.then()
}
.expectSubscription()
.thenAwait(Duration.ofDays(1)) // doesn't matter
.expectNextCount(0)
.expectError()
.verify()
}
Could anybody help me fix the test, and ideally, explain what is wrong?
This is a limitation of virtual time and the way the clock is manipulated in StepVerifier. The thenAwait methods are not synchronized with the underlying scheduling (that happens for example as part of the retryBackoff operation). This means that the operator submits retry tasks at a point where the clock has already been advanced by one day. So the second retry is scheduled for + 1 day and 10 seconds, since the clock is at +1 day. After that, the clock is never advanced so the additional request is never made to MockWebServer.
Your case is made even more complicated in the sense that there is an additional component involved, the MockWebServer, that still works "in real time".
Though advancing the virtual clock is a very quick operation, the response from the MockWebServer still goes through a socket and thus has some amount of latency to the retry scheduling, which makes things more complicated from the test writing perspective.
One possible solution to explore would be to externalize the creation of the VirtualTimeScheduler and tie advanceTimeBy calls to the mockServer.takeRequest(), in a parallel thread.

function lines after httpRequest are not executed in groovy jenkins pipeline

None of the lines after making httpRequest are getting executed. Everything else works fine in this function. What could be going wrong here?
However, network request is going fine and I am able to see the response in the console. httpRequest is being made via plugin
I've even tried CURL - but lines after curl are not executed.
#NonCPS
def doPRCommentBasedTesting() {
def causes = currentBuild.rawBuild.getCauses()
def commentURL
for(cause in causes) {
if (cause.class.toString().contains("GitHubPullRequestCommentCause")) {
commentURL = cause.getCommentUrl()
commentURL = commentURL.substring(commentURL.lastIndexOf("-") + 1)
println "This job was caused by job " + commentURL
def url1 = "https://<git_url>/api/v3/repos/<owner>/<repo>/issues/comments/" + commentURL
def commentText = httpRequest authentication: '<auth_cred>', url: url1, consoleLogResponseBody: true
println commentText
println commentText.getClass()
println "hello world, how are you doing today?"
}
else {
println "Root cause : " + cause.toString()
}
}
println "==============================="
return 0
}
A non cps function does not have the ability to pause in between because it runs in a go. You need to put network call into a different function that is not marked as nonCPS and then it will work. In general the nonCPS block should be very small and limited to code that cannot be serialised

Waiting for running Reactor Mono instances to complete

I wrote this code to spin off a large number of WebClients (limited by reactor.ipc.netty.workerCount), start the Mono immediately, and wait for the all Monos to complete:
List<Mono<List<MetricDataModel>>> monos = new ArrayList<>(metricConfigs.size());
for (MetricConfig metricConfig : metricConfigs) {
try {
monos.add(extractMetrics.queryMetricData(metricConfig)
.doOnSuccess(result -> {
metricDataList.addAll(result);
})
.cache());
} catch (Exception e) {
}
}
Mono.when(monos)
.doFinally(onFinally -> {
Map<String, Date> latestMap;
try {
latestMap = extractInsights.queryInsights();
Transform transform = new Transform(copierConfig.getEventType());
ArrayList<Event> eventList = transform.toEvents(latestMap, metricDataList);
} catch (Exception e) {
log.error("copy: mono: when: {}", e.getMessage(), e);
}
})
.block();
It 'works', that is the results are as expected.
Two questions:
Is this correct? Does cache() result in the when waiting for all Monos to complete?
Is it efficient? Is there a way to make this faster?
Thanks.
You should try as much as possible to:
use Reactor operators and compose a single reactive chain
avoid using doOn* operators for something other than side-effects (like logging)
avoid shared state
Your code could look a bit more like
List<MetricConfig> metricConfigs = //...
Mono<List<MetricDataModel>> data = Flux.fromIterable(metricConfigs)
.flatMap(config -> extractMetrics.queryMetricData(config))
.collectList();
Also, the cache() operator does not wait the completion of the stream (that's actually then()'s job).

Grails Asynchronous Request with HttpSession behaves strangely

I have a simple controller with an action for a long running task and an action for checking a status of the long task:
class AsyncController {
def index() { }
def longTerm() {
session.longTerm = 0
session.longTermDone = false
task {
for (int i; i < 10; i++ ) {
try {
sleep(3000) //NOT WORKING
println " TASK: sessionID ${session.id} value ${session.longTerm++}"
//sleep(3000) //WORKING
} catch (e) {
println(e.class.name)
}
}
session.longTermDone = true
}
render(text: [] as JSON, status: 200)
}
def longTermStatus() {
println "STATUS: sessionID ${session.id} value ${session.longTerm}"
render(text: [successCount: session.longTerm, done: session.longTermDone] as JSON, status: 200)
}
}
In the longTerm action there is a problem with HttpSession.
If the first line of code in the task closure is doing something with HttpSession the rest of the code is working fine. But if the first line is doing something else I get NullPointerException when I try to access session.id
Working code example is at https://github.com/machacekjan/StackOverflowAsyncRequest
Does anyone know why Grails behaves this way?
The issue here appears to be that you're performing the render() outside of the tasks block. If you move render() inside the tasks block, the NullPointerException disappears.
I think this is because the render() finishes the request and you bypass the Servlet 3 Async support. You need to return a promise from the action, which task() does .
Unfortunately, render() doesn't seem to work with the async stuff in Grails 2.3.7 or 2.3.10. But that's a separate issue.

Grails async - creating promises

I am in grails 2.3.1 - trying to use the async features.
This is bulk data processing. I am trying to synchronise 2 databases, which involves comparing both and returning a list of 'deltas'. I am trying to speed up the process
The documentation says I can just add a set of closures to a PromiseList and then call onComplete() to check that all the closures have completed. These are my attempts - directly building on "You can also construct a PromiseList manually" in the documentation:
def tasksMemberDeltas = new PromiseList()
pages.each {Integer page ->
tasksMemberDeltas << {findCreateMemberDeltas(page, (page + pageSize) - 1)}
if (page % 30 == 0) {
tasksMemberDeltas.onComplete {
tasksMemberDeltas = new PromiseList()
}
}
Returns:
Error groovy.lang.MissingMethodException:
No signature of method: java.util.ArrayList.onComplete()
In the end I called .get() which calls waitAll. Going into .get() and finding that it did waitAll was my revelation.
So if I have a single task I call:
waitAll finalDeltas
If I have a list I call:
taskFinalDeltas.get()
onComplete() logically relates to a single delta. Not the list. So this works OK:
Promise memberDeleteDeltas = task {
findDeleteAndTagDeltas()
}
memberDeleteDeltas.onError { Throwable err ->
println "An error occured ${err.message}"
}
memberDeleteDeltas.onComplete { result ->
println "Completed create deltas"
}
waitAll(memberDeleteDeltas)

Resources