I have a problem in a Grails 3.3.8 application after the upgrade from Grails 2.5.6.
I have a service which using Row.findAll() to get records from an H2 database. Then it creates a list of closures for future execution. Then the list is running by a ThreadExecutor via invokeAll(). In each closure I get data via Row.findById().
It is working when I run program, but it does not work in integration tests. I checked that Row.findAll().size() returns 0 inside the closure but just before the invokeAll() it returns a positive number.
Update:
I prepare small test for that:
#Integration
#Rollback
class TestSpec extends Specification {
void "test something"() {
when:
f()
then:
g()
}
private void f() {
Raw raw = new Raw()
raw.text = "text"
raw.save(flush: true)
}
private void g() {
Closure closure = {
try {
def x = rawService.getRawSize()
if (x != 1) throw new Exception("A: x = " + x)
} catch (Exception e) {
e.printStackTrace()
throw e
}
}
def x = rawService.getRawSize()
executorService.invokeAll([closure])
}
}
The code above is not working. It throws an Exception.
Wrap the body of your f() method in a Raw.withNewSession block. This will save your new instance in a separate session. After the closure is finished the session will close, persisting your instance and allowing it to be accessed in another session or thread.
private void f() {
Raw.withNewSession {
Raw raw = new Raw()
raw.text = "text"
raw.save(flush: true)
}
}
Related
In my app I use spock in version 2.0-M1-groovy-2.5. I have a problem that even though I mokced a class I get npe from the internals of mocked method.
I have simple class called ReactiveRestHighLevelClient which looks like this
open class ReactiveRestHighLevelClient(val restHighLevelClient: RestHighLevelClient, private val objectMapper: ObjectMapper) {
...
fun index(indexRequest: IndexRequest): Mono<IndexResponse> {
return Mono.create<IndexResponse> { sink ->
restHighLevelClient.indexAsync( // < -- HERE I GET THE NPE EVEN THOUGH THE METHOD IS MOCKED
indexRequest,
object : ActionListener<IndexResponse> {
override fun onFailure(e: Exception) {
e.printStackTrace()
sink.error(e)
}
override fun onResponse(response: IndexResponse) {
sink.success(response)
}
}
)
}
}
...
}
I have class ThreadModelElasticsearchService which uses the ReactiveRestHighLevelClient and looks like this:
#Component
class ThreadModelElasticsearchService(
private val objectMapper: ObjectMapper,
private val reactiveElasticsearchClient: ReactiveRestHighLevelClient,
private val extraFactsService: ExtraFactsService,
private val customerDataService: CustomerDataService,
rateLimiterRegistry: RateLimiterRegistry
) {
...
fun save(operationId: String, threadModel: ThreadModel): Mono<ThreadModel> {
val id = threadModel.threadId
?: ThreadModel.createKey(threadModel.partnerId, threadModel.customerId)
return reactiveElasticsearchClient
.index(
IndexRequest(ThreadModel.INDEX, ThreadModel.TYPE, id)
.source(objectMapper.writeValueAsString(threadModel), XContentType.JSON)
)
.thenReturn(threadModel)
.doOnNext { logger.info("[operation_id=$operationId] -- Saved : $it") }
.onErrorMap { e ->
logger.error("[operation_id=$operationId] -- Error calling elasticSearch", e)
ElasticRepositoryError(e)
}
.rateLimit(elasticRateLimiter)
}
...
}
Finally I have my test class called ThreadModelElasticsearchServiceTest which looks like this:
class ThreadModelElasticsearchServiceTest extends Specification {
ReactiveRestHighLevelClient reactiveElasticsearchClient = Mock()
... other mocks
ThreadModelElasticsearchService threadModelReactiveElasticsearchClient = new
ThreadModelElasticsearchService(objectMapper, reactiveElasticsearchClient, extraFactsService, customerDataService, RateLimiterRegistry.of(RateLimiterConfig.ofDefaults()))
def "should work"() {
given:
ThreadModel threadModel = new ThreadModel(...)
reactiveElasticsearchClient.index(_ as IndexRequest) >> Mono.just(indexResponse)
expect:
threadModelReactiveElasticsearchClient.save("should-work", threadModel).block()
}
When I run the tests I get execption like
java.lang.NullPointerException: null
at com.cb.elastic.search.api.elasticsearch.ReactiveRestHighLevelClient$index$1.accept(ReactiveRestHighLevelClient.kt:76)
which points to the body of the index method of the ReactiveRestHighLevelClient mock which is strange.
How to solve this ?
To prevent the XY problem, I'll start from the beginning:
I have a non-blocking SOAP client which I wrapped it to make the return type Mono<T> (By default it accepts callback. I can elaborate on this if needed).
Now I want to do (given ID):
1. Get the code by ID
2. Do something with the code
3. After that, get Foo and Bar and create FooBar
What I wrote was:
public class MyService {
private final MySoapClient soapClient;
public Mono<FooBarDto> doSomething(String id) {
return Mono.just(id)
.flatMap(soapClient::getCode) // returns Mono<String>
.flatMap(code ->
soapClient.doSomething(code) // returns Mono<Void>
.then(getFooBar(id, code))); // See this
}
private Mono<FooBarDto> getFooBar(String id, String code) {
return Mono.zip(
soapClient.getFoo(code), // returns Mono<Foo>
soapClient.getBar(code) // returns Mono<Bar>
).map(tuple2 -> toFooBarDto(id, tuple2));
}
private FooBarDto toFooBarDto(String id, Tuple2<Foo, Bar> tuple2) {
return FooBarDto.builder()/* set properties */.build();
}
}
Now the problem is, because methods of the SOAP client are not lazy (the moment you call them they start the process), the semantic of then won't work here. Meaning I want to get Foo and Bar when doSomething is done. They all start together.
I tried to change it fix it by changing then to flatMap, but made it even worse. The getFooBar never got called. (1. Can someone please explain why?).
So what I ended up doing was to wrap SOAP calls again to make them lazy:
public class MySoapClient {
private final AutoGeneratedSoapClient client;
Mono<Foo> getFoo(GetFooRequest request) {
return Mono.just(request).flatMap(this::doGetMsisdnByIccid);
}
private Mono<Foo> doGetFoo(GetFooRequest request) {
val handler = new AsyncHandler<GetFooRequest>();
client.getFoo(request, handler);
return Mono.fromFuture(handler.future);
}
private static class AsyncHandler<T> implements javax.xml.ws.AsyncHandler<T> {
private final CompletableFuture<T> future = new CompletableFuture<>();
#Override
public void handleResponse(Response<T> res) {
try {
future.complete(res.get());
} catch (Exception e) {
future.completeExceptionally(e);
}
}
}
}
Is there any better way to do it? Specifically:
2. Using CompeletableFuture and the callback.
3. Making methods lazy in the SOAP client.
I tried to change it fix it by changing then to flatMap, but made it
even worse. The getFooBar never got called. (1. Can someone please
explain why?)
I think a Mono<Void> always completes empty (or error), so subsequent flatMap is never called.
Using CompeletableFuture and the callback.
Making methods lazy in the SOAP client.
To make the call lazy you can do one of the followings:
1, You can use Mono.fromFuture which accepts a supplier:
private Mono<Foo> doGetFoo(GetFooRequest request) {
return Mono.fromFuture(() -> {
val handler = new AsyncHandler<GetFooRequest>();
client.getFoo(request, handler);
return handler.future;
});
}
2, You can use Mono.defer:
private Mono<Foo> doGetFoo(GetFooRequest request) {
return Mono.defer(() -> {
val handler = new AsyncHandler<GetFooRequest>();
client.getFoo(request, handler);
return Mono.fromFuture(handler.future);
});
}
3, You can get rid of CompletableFuture and use Mono.create instead, something like this:
private Mono<Foo> doGetFoo(GetFooRequest request) {
return Mono.create(sink -> {
AsyncHandler<Foo> handler = response ->
{
try
{
sink.success(response.get());
} catch (Exception e)
{
sink.error(e);
}
};
client.getFoo(request, handler);
});
}
If you do any of these it will be safe to use then method and it will work as expected.
I know this breaks a lot of Rx rules, but I really like RxJava-JDBC and so do my teammates. Relational databases are very core to what we do and so is Rx.
However there are some occasions where we do not want to emit as an Observable<ResultSet> but would rather just have a pull-based Java 8 Stream<ResultSet> or Kotlin Sequence<ResultSet>. But we are very accustomed to the RxJava-JDBC library which only returns an Observable<ResultSet>.
Therefore, I am wondering if there is a way I can turn an Observable<ResultSet> into a Sequence<ResultSet> using an extension function, and not do any intermediary collection or toBlocking() calls. Below is all I have so far but my head is spinning now trying to connect push and pull based systems, and I cannot buffer either as the ResultSet is stateful with each onNext() call. Is this an impossible task?
import rx.Observable
import rx.Subscriber
import java.sql.ResultSet
fun Observable<ResultSet>.asSequence() = object: Iterator<ResultSet>, Subscriber<ResultSet>() {
private var isComplete = false
override fun onCompleted() {
isComplete = true
}
override fun onError(e: Throwable?) {
throw UnsupportedOperationException()
}
override fun onNext(rs: ResultSet?) {
throw UnsupportedOperationException()
}
override fun hasNext(): Boolean {
throw UnsupportedOperationException()
}
override fun next(): ResultSet {
throw UnsupportedOperationException()
}
}.asSequence()
I'm not sure that's the easiest way to achieve what you want but you can try this code. It converts an Observable to an Iterator by creating a blocking queue and publishing all events from the Observable to this queue. The Iterable pulls events from the queue and blocks if there're none. Then it modify its own state depending on received current event.
class ObservableIterator<T>(
observable: Observable<T>,
scheduler: Scheduler
) : Iterator<T>, Closeable {
private val queue = LinkedBlockingQueue<Notification<T>>()
private var cached: Notification<T>? = null
private var completed: Boolean = false
private val subscription =
observable
.materialize()
.subscribeOn(scheduler)
.subscribe({ queue.put(it) })
override fun hasNext(): Boolean {
cacheNext()
return !completed
}
override fun next(): T {
cacheNext()
val notification = cached ?: throw NoSuchElementException()
check(notification.isOnNext)
cached = null
return notification.value
}
private fun cacheNext() {
if (completed) {
return
}
if (cached == null) {
queue.take().let { notification ->
if (notification.isOnError) {
completed = true
throw RuntimeException(notification.throwable)
} else if (notification.isOnCompleted) {
completed = true
} else {
cached = notification
}
}
}
}
override fun close() {
subscription.unsubscribe()
completed = true
cached = null
}
}
You can use the following helper function:
fun <T> Observable<T>.asSequence() = Sequence { toBlocking().getIterator() }
The observable will be subscribed to when the sequence returned is called for iterator.
If an observable emits elements on the same thread it was subscribed to (like Observable.just for example), it will populate the buffer of the iterator before it gets a chance to be returned.
In this case you might need to direct subscription to the different thread with a call to subscribeOn:
observable.subscribeOn(scheduler).asSequence()
However, while toBlocking().getIterator() doesn't buffer all results it could buffer some of them if they aren't consumed timely by the iterator. That might be a problem if a ResultSet gets somehow expired when the next ResultSet arrives.
I'm new to reactive programming using rxjava and after going through the simpler examples I'm now trying to figure out how to work with continuous streams. The problem I have with the example below is that the program doesn't terminate after I've taken the 3 elements. My assumption is that I somehow need to unsubscribe to my observable but I don't fully grasp how to terminate the while loop and make the program exit.
I've come across the following post RxJava -- Terminating Infinite Streams but I still can't figure out what I'm missing.
class MyTwitterDataProvider {
/*
This example is written in Groovy
Instance variables and constructor omitted
*/
public Observable<String> getTweets() {
BufferedReader reader = new BufferedReader(new InputStreamReader(getTwitterStream()))
Observable.create({ observer ->
executor.execute(new Runnable() {
def void run() {
String newLine
while ((newLine = reader.readLine()) != null) {
System.out.println("printing tweet: $newLine")
observer.onNext(newLine)
}
observer.onCompleted()
}
})
})
}
def InputStream getTwitterStream() {
// code omitted
}
public static void main (String [] args) {
MyTwitterDataProvider provider = new MyTwitterDataProvider()
Observable<String> myTweetsObservable = provider.getTweets().take(3)
Subscription myTweetSubscription = myTweetsObservable.subscribe({tweet-> println("client prints: $tweet")})
// myTweetSubscription.unsubscribe()
}
}
You must add a check in your loop to see if the observer is still subscribed:
while ((newLine = reader.readLine()) != null && !observer.isUnsubsribed()) {
System.out.println("printing tweet: $newLine")
observer.onNext(newLine)
}
I am developing in a Grails application. What I want to do is to lock the request/response, create a promise, and let someone else resolve it, that is somewhere else in the code, and then flush the response.
What I find really strange is that the Promise promise = task {} interface has no method that resembles resolve or similar.
I need to lock the response until someone resolves the promise, which is a global/static property set in development mode.
Promise interface:
http://grails.org/doc/latest/api/grails/async/Promise.html
I have looked at the GPars doc and can't find anything there that resembles a resolve method.
How can I create a promise, that locks the response or request, and then flushes the response when someone resolves it?
You can call get() on the promise which will block until whatever the task is doing completes, but I imagine what that is not what you want. What you want seems to be equivalent to a GPars DataflowVariable:
http://gpars.org/1.0.0/javadoc/groovyx/gpars/dataflow/DataflowVariable.html
Which allows using the left shift operator to resolve the value from another thread. Currently there is no way to use the left shift operator via Grails directly, but since Grails' promise API is just a layer over GPars this can probably be accomplished by using the GPars API directly with something like:
import org.grails.async.factory.gpars.*
import groovyx.gpars.dataflow.*
import static grails.async.Promise.*
def myAction() {
def dataflowVar = new DataflowVariable()
task {
// do some calculation and resolve data flow variable
def expensiveData = ...
dataflowVar << expensiveData
}
return new GParsPromise(dataflowVar)
}
It took me quite some time to get around this and have a working answer.
I must say that it appears as if Grails is quite a long way of making this work properly.
task { }
will always execute immediatly, so the call is not put on hold until dispatch() or whatever is invoked which is a problem.
Try this to see:
public def test() {
def dataflowVar = new groovyx.gpars.dataflow.DataflowVariable()
task {
// do some calculation and resolve data flow variable
println '1111111111111111111111111111111111111111111111111111'
//dataflowVar << expensiveData
}
return new org.grails.async.factory.gpars.GparsPromise(dataflowVar);
}
If you are wondering what this is for, it is to make the lesscss refresh automatically in grails, which is a problem when you are using import statements in less. When the file is touched, the lesscss compiler will trigger a recompilation, and only when it is done should it respond to the client.
On the client side I have some javascript that keeps replacing the last using the refresh action here:
In my controller:
/**
* Refreshes link resources. refresh?uri=/resource/in/web-app/such/as/empty.less
*/
public def refresh() {
return LessRefresh.stackRequest(request, params.uri);
}
A class written for this:
import grails.util.Environment
import grails.util.Holders
import javax.servlet.AsyncContext
import javax.servlet.AsyncEvent
import javax.servlet.AsyncListener
import javax.servlet.http.HttpServletRequest
/**
* #Author SecretService
*/
class LessRefresh {
static final Map<String, LessRefresh> FILES = new LinkedHashMap<String, LessRefresh>();
String file;
Boolean touched
List<AsyncContext> asyncContexts = new ArrayList<AsyncContext>();
String text;
public LessRefresh(String file) {
this.file = file;
}
/** Each request will be put on hold in a stack until dispatchAll below is called when the recompilation of the less file finished **/
public static AsyncContext stackRequest(HttpServletRequest request, String file) {
if ( !LessRefresh.FILES[file] ) {
LessRefresh.FILES[file] = new LessRefresh(file);
}
return LessRefresh.FILES[file].handleRequest(request);
}
public AsyncContext handleRequest(HttpServletRequest request) {
if ( Environment.current == Environment.DEVELOPMENT ) {
// We only touch it once since we are still waiting for the less compiler to finish from previous edits and recompilation
if ( !touched ) {
touched = true
touchFile(file);
}
AsyncContext asyncContext = request.startAsync();
asyncContext.setTimeout(10000)
asyncContexts.add (asyncContext);
asyncContext.addListener(new AsyncListener() {
#Override
void onComplete(AsyncEvent event) throws IOException {
event.getSuppliedResponse().writer << text;
}
#Override
void onTimeout(AsyncEvent event) throws IOException {
}
#Override
void onError(AsyncEvent event) throws IOException {
}
#Override
void onStartAsync(AsyncEvent event) throws IOException {
}
});
return asyncContext;
}
return null;
}
/** When recompilation is done, dispatchAll is called from LesscssResourceMapper.groovy **/
public void dispatchAll(String text) {
this.text = text;
if ( asyncContexts ) {
// Process all
while ( asyncContexts.size() ) {
AsyncContext asyncContext = asyncContexts.remove(0);
asyncContext.dispatch();
}
}
touched = false;
}
/** A touch of the lessfile will trigger a recompilation **/
int count = 0;
void touchFile(String uri) {
if ( Environment.current == Environment.DEVELOPMENT ) {
File file = getWebappFile(uri);
if (file && file.exists() ) {
++count;
if ( count < 5000 ) {
file << ' ';
}
else {
count = 0
file.write( file.getText().trim() )
}
}
}
}
static File getWebappFile(String uri) {
new File( Holders.getServletContext().getRealPath( uri ) )
}
}
In LesscssResourceMapper.groovy of the lesscsss-recources plugin:
...
try {
lessCompiler.compile input, target
// Update mapping entry
// We need to reference the new css file from now on
resource.processedFile = target
// Not sure if i really need these
resource.sourceUrlExtension = 'css'
resource.contentType = 'text/css'
resource.tagAttributes?.rel = 'stylesheet'
resource.updateActualUrlFromProcessedFile()
// ==========================================
// Call made here!
// ==========================================
LessRefresh.FILES[resource.sourceUrl.toString()]?.dispatchAll( target.getText() );
} catch (LessException e) {
log.error("error compiling less file: ${originalFile}", e)
}
...
In the index.gsp file:
<g:set var="uri" value="${"${App.files.root}App/styles/empty.less"}"/>
<link media="screen, projection" rel="stylesheet" type="text/css" href="${r.resource(uri:uri)}" refresh="${g.createLink(controller:'home', action:'refresh', params:[uri:uri])}" resource="true">
JavaScript method refreshResources to replace the previous link href=...
/**
* Should only be used in development mode
*/
function refreshResources(o) {
o || (o = {});
var timeoutBegin = o.timeoutBegin || 1000;
var intervalRefresh = o.intervalRefresh || 1000;
var timeoutBlinkAvoid = o.timeoutBlinkAvoid || 400 ;
var maxErrors = o.maxErrors || 200 ;
var xpath = 'link[resource][type="text/css"]';
// Find all link[resource]
$(xpath).each(function(i, element) {
refresh( $(element) );
});
function refresh(element) {
var parent = element.parent();
var next = element.next();
var outer = element.clone().attr('href', '').wrap('<p>').parent().html();
var uri = element.attr('refresh');
var errorCount = 0;
function replaceLink() {
var link = $(outer);
link.load(function () {
// The link has been successfully added! Now remove the other ones, then do again
errorCount = 0;
// setTimeout needed to avoid blinking, we allow duplicates for a few milliseconds
setTimeout(function() {
var links = parent.find(xpath + '[refresh="'+uri+'"]');
var i = 0;
// Remove all but this one
while ( i < links.length - 1 ) {
links[i++].remove();
}
replaceLinkTimeout();
}, timeoutBlinkAvoid );
});
link.error(function(event, handler) {
console.log('Error refreshing: ' + outer );
++errorCount;
if ( errorCount < maxErrors ) {
// Load error, it happens. Remove this & redo!
link.remove();
replaceLink();
}
else {
console.log('Refresh: Aborting!')
}
});
link.attr('href', urlRandom(uri)).get(0);
link.insertBefore(next); // Insert just after
}
function urlRandom(uri) {
return uri + "&rand=" + Math.random();
}
function replaceLinkTimeout() {
setTimeout(function() {
replaceLink();
}, intervalRefresh ) ;
}
// Waith 1s before triggering the interval
setTimeout(function() {
replaceLinkTimeout();
}, timeoutBegin);
}
};
Comments
I am unsure why Javascript style promises have not been added to the Grails stack.
You can not render or stuff like that in the onComplete. render, redirect and what not are not available.
Something tells me that Grails and Promises/Futures are not there yet. The design of the GPars libraries seems not take into account of the core features which is to resolve later. At least it is not simple to do so.
It would be great if the dispatch() method actually could be invoked with some paramaters to pass from the resolving context. I am able to go around this using static properties.
I might continue to write my own solution and possibly contribute with a more fitting solutions around the AsyncContext class, but for now, this is enough for me.
I just wanted to refresh my less resources automatically.
Phew...
EDIT:
I made it to support several number of files. It is complete now!