I have 2 asynchronous Monos, I'd like to start processing immediately whenver a result is available.
I am looking at the Reactor API firstWithSignal that seems promising. but problem is it cancels the later one as soon as one result is returned.
Is there a way I can keep both run through but can continue with whichever result that is available?
Related
I'm writing a data pipeline using Reactor and Reactor Kafka and use spring's Message<> to save
the ReceiverOffset of ReceiverRecord in the headers, to be able to use ReciverOffset.acknowledge() when finish processing. I'm also using the out-of-order commit feature enabled.
When an event process fails I want to be able to log the error, write to another topic that represents all the failure events, and commit to the source topic. I'm currently solving that by returning Either<Message<Error>,Message<myPojo>> from each processing stage, that way the stream will not be stopped by exceptions and I'm able to save the original event headers and eventually commit the failed messages at the button of the pipeline.
The problem is that each step of the pipline gets Either<> as input and needs to filter the previous errors, apply the logic only on the Either.right and that could be cumbersome, especially when working with buffers and the operator get 'List<Either<>>' as input. So I would want to keep my business pipeline clean and get only Message<MyPojo> as input but also not missing errors that need to be handled.
I read that sending those message erros to other channel or stream is a soulution for that.
Spring Integration uses that pattern for error handling and I also read an article (link to article) that solves this problem in Akka Streams using 'divertTo()':
I couldn't find documentation or code examples of how to implement that in Reactor,
is there any way to use Spring Integration error channel with Reactor? or any other ideas to implement that?
Not familiar with reactor per se, but you can keep the stream linear. The trick, since Vavr's Either is right-biased is to use flatMap, which would take a function from Message<MyPojo> to Either<Message<Error>, Message<MyPojo>>. If the Either coming in is a right (i.e. a Message<MyPojo>, the function gets invoked and otherwise it just gets passed through.
// Apologies if the Java is atrocious... haven't written Java since pre-Java 8
incomingEither.flatMap(
myPojoMessage -> ... // compute a new Either
)
Presumably at some point you want to do something (publish to a dead-letter topic, tickle metrics, whatever) with the Message<Error> case, so for that, orElseRun will come in handy.
I'm using Neo4j. For large data imports from external csvs, parquets, etc. there is a very handful command for "fire and forget", the apoc.periodic.submit. There is also the apoc.periodic.list that list the background jobs.
During the execution of the background job it appears in the output of apoc.periodic.list. But after it finishes, either by an error or by a successful execution, it will disappear from this list without any feedback from the completion status.
Is there a general way to check if a background job finish status? Is there a more suitable API for my purposes?
If there is a way to directly check error status on the fire&forget routines, I don't see it in the documentation (they are fire&forget, so it comes with the territory?)
Ideas
don't background the query itself, background a process/task that waits for a blocking Cypher execution to finish and capture the error code...
check for success instead of failure? (if it didn't succeed you know it failed right?), this may be evident based on what the Cypher does, or you could add a graph content update for this purpose. E.g. Update property on a NODE with last_updated. Do that last so that if the cypher fails, the property is not updated
You could enable query log and then check there to see what happened, most likely this query has a unique signature and the last execution could be found easily in the log (with status/error code)
I'm using Alamofire to execute a variable number of requests in a for loop which are all asynchronous. I need to find a way to be notified when all the requests complete (successfully or not) and to collect their results for processing. Is there a feature for this built in or perhaps a third party library for such tasks?
I'm not sure if their is a feature for this, but you could always keep a counter and decrement it on each completion, checking if you've reached zero.
I use ASP.Net MVC 5 and I have a long running action which have to poll webservices, process data and store them in database.
For that I want to use TPL library to start the task async.
But I wonder how to do 3 things :
I want to report progress of this task. For this I think about SignalR
I want to be able to left the page where I start this task from and be able to report the progression across the website (from a panel on the left but this is ok)
And I want to be able to cancel this task globally (from my panel on the left)
I know quite a few about all of technologies involved. But I'm not sure about the best way to achieve this.
Is someone can help me about the best solution ?
The fact that you want to run long running work while the user can navigate away from the page that initiates the work means that you need to run this work "in the background". It cannot be performed as part of a regular HTTP request because the user might cancel his request at any time by navigating away or closing the browser. In fact this seems to be a key scenario for you.
Background work in ASP.NET is dangerous. You can certainly pull it off but it is not easy to get right. Also, worker processes can exit for many reasons (app pool recycle, deployment, machine reboot, machine failure, Stack Overflow or OOM exception on an unrelated thread). So make sure your long-running work tolerates being aborted mid-way. You can reduce the likelyhood that this happens but never exclude the possibility.
You can make your code safe in the face of arbitrary termination by wrapping all work in a transaction. This of course only works if you don't cause non-transacted side-effects like web-service calls that change state. It is not possible to give a general answer here because achieving safety in the presence of arbitrary termination depends highly on the concrete work to be done.
Here's a possible architecture that I have used in the past:
When a job comes in you write all necessary input data to a database table and report success to the client.
You need a way to start a worker to work on that job. You could start a task immediately for that. You also need a periodic check that looks for unstarted work in case the app exits after having added the work item but before starting a task for it. Have the Windows task scheduler call a secret URL in your app once per minute that does this.
When you start working on a job you mark that job as running so that it is not accidentally picked up a second time. Work on that job, write the results and mark it as done. All in a single transaction. When your process happens to exit mid-way the database will reset all data involved.
Write job progress to a separate table row on a separate connection and separate transaction. The browser can poll the server for progress information. You could also use SignalR but I don't have experience with that and I expect it would be hard to get it to resume progress reporting in the presence of arbitrary termination.
Cancellation would be done by setting a cancel flag in the progress information row. The app needs to poll that flag.
Maybe you can make use of message queueing for job processing but I'm always wary to use it. To process a message in a transacted way you need MSDTC which is unsupported with many high-availability solutions for SQL Server.
You might think that this architecture is not very sophisticated. It makes use of polling for lots of things. Polling is a primitive technique but it works quite well. It is reliable and well-understood. It has a simple concurrency model.
If you can assume that your application never exits at inopportune times the architecture would be much simpler. But this cannot be assumed. You cannot assume that there will be no deployments during work hours and that there will be no bugs leading to crashes.
Even if using http worker is a bad thing to run long task I have made a small example of how to manage it with SignalR :
Inside this example you can :
Start a task
See task progression
Cancel task
It's based on :
twitter bootstrap
knockoutjs
signalR
C# 5.0 async/await with CancelToken and IProgress
You can find the source of this example here :
https://github.com/dragouf/SignalR.Progress
Does anyone know of any good resources that fully explain how functions and results will fire in an Adobe AIR app where multiple things are happening at once?
As a simple test, I've created a single service that I just keep changing the url of, then issuing a send(). It seems that no matter how many send() calls I put in, all of these get executed before the 'result' eventListener function gets called for the first time.
Is this how it works? i.e. the current function gets fully executed, with the async returns queueing up to be processed after AIR has finished what it's currently doing.
Likewise, if the user does something while all this is going on, I presume their request goes to the back of the queue as well?
All that makes sense, but I'm just wondering if it's documented anywhere.
While I'm on one, is it recommended practice to reuse the same HTTPService in this way, or is it better to create one for each concurrent transaction? Just because it works, doesn't mean it's the right thing to do...
I'm not aware of any documentation that explains this, but I can confirm that code blocks get executed before async calls are made, or at least before their result is being processed. If it didn't work that way, you would for instance not always be able to attach a responder to a token of a service call, because the result might already have been processed.
var token:AsyncToken = myService.someMethod();
token.addResponder(new Responder(resultHandler, faultHandler));
Developers coming from other platforms find this strange as they would expect the assignment of the responder to be too late.
So while I don't have an official explanation about the technical details inside the Flash Player, I can assure that it works this way.
If the user does something while a call is pending, the new request will indeed just be added as a new asynchronous call. Note that we can't realy speak of a queue, as there is no guarantee that the response of the first call comes in before the response of the second call. This depends on how much time the actual requests take.
You can perfectly reuse an HTTPService instance.
PS: Based on this, we were able to build the Operation API in Spring ActionScript. It is basically an API that allows you to execute asynchronous processes in a uniform way, without having to worry about the details of the actual async process.
The following code executes an async process and attaches a handler to it. This is also something that puzzles many developers at first, for reasons similar to the asyncToken situation.
var operation:IOperation = doSomeOperation();
operation.addCompleteListener(aCompleteHandler);
operation.addErrorListener(anErrorHandler);