I wrote this code to spin off a large number of WebClients (limited by reactor.ipc.netty.workerCount), start the Mono immediately, and wait for the all Monos to complete:
List<Mono<List<MetricDataModel>>> monos = new ArrayList<>(metricConfigs.size());
for (MetricConfig metricConfig : metricConfigs) {
try {
monos.add(extractMetrics.queryMetricData(metricConfig)
.doOnSuccess(result -> {
metricDataList.addAll(result);
})
.cache());
} catch (Exception e) {
}
}
Mono.when(monos)
.doFinally(onFinally -> {
Map<String, Date> latestMap;
try {
latestMap = extractInsights.queryInsights();
Transform transform = new Transform(copierConfig.getEventType());
ArrayList<Event> eventList = transform.toEvents(latestMap, metricDataList);
} catch (Exception e) {
log.error("copy: mono: when: {}", e.getMessage(), e);
}
})
.block();
It 'works', that is the results are as expected.
Two questions:
Is this correct? Does cache() result in the when waiting for all Monos to complete?
Is it efficient? Is there a way to make this faster?
Thanks.
You should try as much as possible to:
use Reactor operators and compose a single reactive chain
avoid using doOn* operators for something other than side-effects (like logging)
avoid shared state
Your code could look a bit more like
List<MetricConfig> metricConfigs = //...
Mono<List<MetricDataModel>> data = Flux.fromIterable(metricConfigs)
.flatMap(config -> extractMetrics.queryMetricData(config))
.collectList();
Also, the cache() operator does not wait the completion of the stream (that's actually then()'s job).
Related
I have implemented retries around some code in Jenkins pipeline.
retry(2) {
// code
}
Is there a way to identify when a retry has happened? Besides just manually checking console logs.
Would like to identify builds which have retried some flakey code and send notification.
I don't know of a way to know that inside the retry{ } block.
You could try a fairly generic solution (even maybe as a global var in a shared pipeline library), such as using a try catch block (since the retry will wait for an exception):
int retryAttempts = 0
retry(2) {
try {
if (retryAttempts>0) {
// a retry is occurring
// Do pre-retry logic, if needed
...
}
// Do stuff
.....
} catch (e) {
retryAttempts++ // a retry WILL occur
throw e // rethrow to trigger retry
}
}
if (retryAttempts>0) {
// a retry has occurred
// Do post-retry logic, if needed
...
}
I am trying to learn Reactor but I am having a lot of trouble with it. I wanted to do a very simple proof of concept where I simulate calling a slow down stream service 1 or more times. If you use reactor and stream the response the caller doesn't have to wait for all the results.
So I created a very simple controller but it is not behaving like I expect. When the delay is "inside" my flatMap (inside the method I call) the response is not returned until everything is complete. But when I add a delay after the flatMap the data is streamed.
Why does this code result in a stream of JSON
#GetMapping(value = "/test", produces = { MediaType.APPLICATION_STREAM_JSON_VALUE })
Flux<HashMap<String, Object>> customerCards(#PathVariable String customerId) {
Integer count = service.getCount(customerId);
return Flux.range(1, count).
flatMap(k -> service.doRestCall(k)).delayElements(Duration.ofMillis(5000));
}
But this does not
#GetMapping(value = "/test2", produces = { MediaType.APPLICATION_STREAM_JSON_VALUE })
Flux<HashMap<String, Object>> customerCards(#PathVariable String customerId) {
Integer count = service.getCount(customerId);
return Flux.range(1, count).
flatMap(k -> service.doRestCallWithDelay(k));
}
It think I am missing something very basic of the reactor API. On that note. can anyone point to a good book or tutorial on reactor? I can't seem to find anything good to learn this.
Thanks
The code inside the flatMap runs on the main thread (that is the thread the controller runs). As a result the whole process is blocked and the method doesnt return immediately. Have in mind that Reactor doesnt impose a particular threading model.
On the contrary, according to the documentation, in the delayElements method signals are delayed and continue on the parallel default Scheduler. That means that the main thread is not blocked and returns immediately.
Here are two corresponding examples:
Blokcing code:
Flux.range(1, 500)
.map(i -> {
//blocking code
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + " - Item : " + i);
return i;
})
.subscribe();
System.out.println("main completed");
Result:
main - Item : 1
main - Item : 2
main - Item : 3
...
main - Item : 500
main completed
Non-blocking code:
Flux.range(1, 500)
.delayElements(Duration.ofSeconds(1))
.subscribe(i -> {
System.out.println(Thread.currentThread().getName() + " - Item : " + i);
});
System.out.println("main Completed");
//sleep main thread in order to be able to print the println of the flux
try {
Thread.sleep(30000);
} catch (InterruptedException e) {
e.printStackTrace();
}
Result:
main Completed
parallel-1 - Item : 1
parallel-2 - Item : 2
parallel-3 - Item : 3
parallel-4 - Item : 4
...
Here is the project reactor reference guide
"delayElements" method only delay flux element by a given duration, see javadoc for more details
I think you should post details about methods "service.doRestCallWithDelay(k)" and "service.doRestCall(k)" if you need more help.
I am working on a windows service that works on my development machine, but when I deploy to a test server I find that the Quartz Scheduler I have set up does not work.
There are no exceptions thrown or any clues as to why this is.
Here is my code;
protected override void OnStart(string[] args)
{
try
{
this.SetDailySchedule();
}
catch (SchedulerException ex)
{
this.eventLog.WriteEntry("WMS FM Loader: Schedule Error - " + ex.Message);
throw ex;
}
catch (Exception ex)
{
this.eventLog.WriteEntry("WMS FM Loader: Unexpected Error - " + ex.Message);
throw ex;
}
}
private void SetDailySchedule()
{
this.eventLog.WriteEntry("On Start".Log());
var schedule = this.GetSchedule();
try
{
this.schedFact = new StdSchedulerFactory();
this.sched = this.schedFact.GetScheduler();
}
catch (Exception ex)
{
this.eventLog.WriteEntry(ex.Message.Log());
throw;
}
this.eventLog.WriteEntry(string.Format("Got a scheduler: {0}", schedule).Log());
try
{
ScheduleTheLoad(schedule);
}
catch (Exception ex)
{
this.eventLog.WriteEntry(ex.Message.Log());
throw;
}
this.eventLog.WriteEntry("WMS FM Loader: Scheduler ready");
}
private void ScheduleTheLoad(string schedule)
{
var job = JobBuilder.Create<LoadWorksOrders>().WithIdentity("LoadWorksOrdersJob").Build();
// Trigger the job to run now, and then every day
var trigger =
TriggerBuilder.Create()
.ForJob(job)
.WithIdentity("LoadWorksOrdersTrigger")
.StartNow()
.WithCronSchedule(schedule)
.Build();
this.sched.ScheduleJob(job, trigger);
this.sched.Start();
}
I manually start the service and the event log shows that the scheduler is ready, but LoadWorksOrdersJob doe snot get run.
How should I fix this?
Are you running the two methods in the windows service OnStart? Are you using a service account? Any exceptions in the event viewer/ have you started /stopped the service from services.msc?
I also had this problem. It took me a day to work it out.
For me it was that my assembly had dots/periods in its name. e.g.
Project.UpdateService
When I changed it to...
ProjectUpdateService
... it worked fine. It always worked on the development machine. It just would not work on the remote machine.
UPDATE: It may have been the length of the service that has caused this issue. By removing the dots I shortened the service name. It looks like the maximum length is 25 characters.
I use my postgres database query to determine my next action. And I need to wait for the results before I can execute the next line of code. Now my conn.query returns a Future but I can't manage to get it async when I place my code in another function.
main() {
// get the database connection string from the settings.ini in the project root folder
db = getdb();
geturl().then((String url) => print(url));
}
Future geturl() {
connect(db).then((conn) {
conn.query("select trim(url) from crawler.crawls where content IS NULL").toList()
.then((result) { return result[0].toString(); })
.catchError((err) => print('Query error: $err'))
.whenComplete(() {
conn.close();
});
});
}
I just want geturl() to wait for the returned value but whatever I do; it fires immediately. Can anyone point me a of a piece of the docs that explains what I am missing here?
You're not actually returning a Future in geturl currently. You have to actually return the Futures that you use:
Future geturl() {
return connect(db).then((conn) {
return conn.query("select trim(url) from crawler.crawls where content IS NULL").toList()
.then((result) { return result[0].toString(); })
.catchError((err) => print('Query error: $err'))
.whenComplete(() {
conn.close();
});
});
}
To elaborate on John's comment, here's how you'd implement this using async/await. (The async/await feature was added in Dart 1.9)
main() async {
try {
var url = await getUrl();
print(url);
} on Exception catch (ex) {
print('Query error: $ex');
}
}
Future getUrl() async {
// get the database connection string from the settings.ini in the project root folder
db = getdb();
var conn = await connect(db);
try {
var sql = "select trim(url) from crawler.crawls where content IS NULL";
var result = await conn.query(sql).toList();
return result[0].toString();
} finally {
conn.close();
}
}
I prefer, in scenarios with multiple-chained futures (hopefully soon a thing of the past once await comes out), to use a Completer. It works like this:
Future geturl() {
final c = new Completer(); // declare a completer.
connect(db).then((conn) {
conn.query("select trim(url) from crawler.crawls where content IS NULL").toList()
.then((result) {
c.complete(result[0].toString()); // use the completer to return the result instead
})
.catchError((err) => print('Query error: $err'))
.whenComplete(() {
conn.close();
});
});
return c.future; // return the future to the completer instead
}
To answer your 'where are the docs' question: https://www.dartlang.org/docs/tutorials/futures/
You said that you were trying to get your geturl() function to 'wait for the returned value'. A function that returns a Future (as in the example in the previous answer) will execute and return immediately, it will not wait. In fact that is precisely what Futures are for, to avoid code doing nothing or 'blocking' while waiting for data to arrive or an external process to finish.
The key thing to understand is that when the interpreter gets to a call to then() or 'catchError()' on a Future, it does not execute the code inside, it puts it aside to be executed later when the future 'completes', and then just keeps right on executing any following code.
In other words, when using Futures in Dart you are setting up chunks of code that will be executed non-linearly.
I want to store position coords (latitude, longitude) in a table in my MySQL DB querying a url in a way similar to this one: http://locationstore.com/postlocation.php?latitude=var1&longitude=var2 every ten seconds. PHP script works like a charm. Getting the coords in the device ain't no problem either. But making the request to the server is being a hard one. My code goes like this:
public class LocationHTTPSender extends Thread {
for (;;) {
try {
//fetch latest coordinates
coords = this.coords();
//reset url
this.url="http://locationstore.com/postlocation.php";
// create uri
uri = URI.create(this.url);
FireAndForgetDestination ffd = null;
ffd = (FireAndForgetDestination) DestinationFactory.getSenderDestination
("MyContext", uri);
if(ffd == null)
{
ffd = DestinationFactory.createFireAndForgetDestination
(new Context("MyContext"), uri);
}
ByteMessage myMsg = ffd.createByteMessage();
myMsg.setStringPayload("doesnt matter");
((HttpMessage) myMsg).setMethod(HttpMessage.POST);
((HttpMessage) myMsg).setQueryParam("latitude", coords[0]);
((HttpMessage) myMsg).setQueryParam("longitude", coords[1]);
((HttpMessage) myMsg).setQueryParam("user", "1");
int i = ffd.sendNoResponse(myMsg);
ffd.destroy();
System.out.println("Lets sleep for a while..");
Thread.sleep(10000);
System.out.println("woke up");
} catch (Exception e) {
// TODO Auto-generated catch block
System.out.println("Exception message: " + e.toString());
e.printStackTrace();
}
}
I haven't run this code to test it, but I would be suspicious of this call:
ffd.destroy();
According to the API docs:
Closes the destination. This method cancels all outstanding messages,
discards all responses to those messages (if any), suspends delivery
of all incoming messages, and blocks any future receipt of messages
for this Destination. This method also destroys any persistable
outbound and inbound queues. If Destination uses the Push API, this
method will unregister associated push subscriptions. This method
should be called only during the removal of an application.
So, if you're seeing the first request succeed (at least sometimes), and subsequent requests fail, I would try removing that call to destroy().
See the BlackBerry docs example for this here
Ok so I finally got it running cheerfully. The problem was with the transport selection; even though this example delivered WAP2 (among others) as an available transport in my device, running the network diagnostics tool showed only BIS as available. It also gave me the connection parameters that I needed to append at the end of the URL (;deviceside=false;ConnectionUID=GPMDSEU01;ConnectionType=mds-public). The code ended up like this:
for (;;) {
try {
coords.refreshCoordinates();
this.defaultUrl();
this.setUrl(stringFuncs.replaceAll(this.getUrl(), "%latitude%", coords.getLatitude() + ""));
this.setUrl(stringFuncs.replaceAll(this.getUrl(), "%longitude%", coords.getLongitude() + ""));
cd = cf.getConnection(this.getUrl());
if (cd != null) {
try {
HttpConnection hc = (HttpConnection)cd.getConnection();
final int i = hc.getResponseCode();
hc.close();
} catch (Exception e) {
}
}
//dormir
Thread.sleep(15000);
} catch (Exception e) {
} finally {
//cerrar conexiones
//poner objetos a null
}
Thanks for your help #Nate, it's been very much appreciated.