How can I use existing Flux operators to make a Flux return incoming values into multiple Lists with a minimum delay between returns?
This can be achieved with a non-trivial set of composed operators.
import java.time.Duration;
import java.util.*;
import reactor.core.publisher.*;
public class DelayedBuffer {
public static void main(String[] args) {
Flux.just(1, 2, 3, 6, 7, 10)
.flatMap(v -> Mono.delayMillis(v * 1000)
.doOnNext(w -> System.out.println("T=" + v))
.map(w -> v)
)
.compose(f -> delayedBufferAfterFirst(f, Duration.ofSeconds(2)))
.doOnNext(System.out::println)
.blockLast();
}
public static <T> Flux<List<T>> delayedBufferAfterFirst(Flux<T> source, Duration d) {
return source
.publish(f -> {
return f.take(1).collectList()
.concatWith(f.buffer(d).take(1))
.repeatWhen(r -> r.takeUntilOther(f.ignoreElements()));
});
}
}
(Note however, that the expected emission pattern may be better matched with a custom operator due to time being involved.)
I thought buffer(Duration) would fit your need, but it doesn't.
edit: leaving this in case someone with your exact same need is tempted to use that operator. This variant of buffer splits the sequence into consecutive time windows (that each produce a buffer). That is, the new delay starts at the end of the previous one, not whenever a new out-of-delay element is emitted.
Related
I would like to have a list of functions and call one of them by using a number n to call the nth function of the list.
I thought that the best way to do this would be by using a map, but I can immediately spot some problems that I don't know how to solve in this example:
import 'dart:math';
void functionOne(x) => print("the randomly generated number is $x");
void functionTwo(x) => print("the inverse of the randomly generated number is ${1/x}");
var example = {0:functionOne(), 1:functionTwo()};
var intValue = Random().nextInt(2);
var biggerIntValue = Random().nextInt(100) + 1;
void main() {
example[0](biggerIntValue);
}
I would like to be able to pass parameters into said function and be able to access the functions by using a number. I know that I could make one giant function and use if statements to do certain actions, but I was wondering if there was any predetermined syntax that would look better than this:
import 'dart:math';
void largeFunction(funcNumber, x) {
if (funcNumber == 0) {functionOne(x);}
if (funcNumber == 1) {functionTwo(x);}
}
void functionOne(x) => print("the randomly generated number is $x");
void functionTwo(x) => print("the inverse of the randomly generated number is ${1/x}");
var intValue = Random().nextInt(2);
var biggerIntValue = Random().nextInt(100) + 1;
void main() {
largeFunction(intValue, biggerIntValue);
}
Thanks for reading my question!
Your problem with the map is that you call the function expressions. Try:
var example = {0:functionOne, 1:functionTwo};
The next problem will be that
example[0](biggerIntValue);
complains that example[0] is nullable.
I'd change that to:
example[0]?.call(biggerIntValue);
so that it only calls something if the integer is a key in the map.
Is there a way to return several values in a function return statement (other than returning an object) like we can do in Go (or some other languages)?
For example, in Go we can do:
func vals() (int, int) {
return 3, 7
}
Can this be done in Dart? Something like this:
int, String foo() {
return 42, "foobar";
}
Dart doesn't support multiple return values.
You can return an array,
List foo() {
return [42, "foobar"];
}
or if you want the values be typed use a Tuple class like the package https://pub.dartlang.org/packages/tuple provides.
See also either for a way to return a value or an error.
I'd like to add that one of the main use-cases for multiple return values in Go is error handling which Dart handle's in its own way with Exceptions and failed promises.
Of course this leaves a few other use-cases, so let's see how code looks when using explicit tuples:
import 'package:tuple/tuple.dart';
Tuple2<int, String> demo() {
return new Tuple2(42, "life is good");
}
void main() {
final result = demo();
if (result.item1 > 20) {
print(result.item2);
}
}
Not quite as concise, but it's clean and expressive code. What I like most about it is that it doesn't need to change much once your quick experimental project really takes off and you start adding features and need to add more structure to keep on top of things.
class FormatResult {
bool changed;
String result;
FormatResult(this.changed, this.result);
}
FormatResult powerFormatter(String text) {
bool changed = false;
String result = text;
// secret implementation magic
// ...
return new FormatResult(changed, result);
}
void main() {
String draftCode = "print('Hello World.');";
final reformatted = powerFormatter(draftCode);
if (reformatted.changed) {
// some expensive operation involving servers in the cloud.
}
}
So, yes, it's not much of an improvement over Java, but it works, it is clear, and reasonably efficient for building UIs. And I really like how I can quickly hack things together (sometimes starting on DartPad in a break at work) and then add structure later when I know that the project will live on and grow.
Create a class:
import 'dart:core';
class Tuple<T1, T2> {
final T1 item1;
final T2 item2;
Tuple({
this.item1,
this.item2,
});
factory Tuple.fromJson(Map<String, dynamic> json) {
return Tuple(
item1: json['item1'],
item2: json['item2'],
);
}
}
Call it however you want!
Tuple<double, double>(i1, i2);
or
Tuple<double, double>.fromJson(jsonData);
You can create a class to return multiple values
Ej:
class NewClass {
final int number;
final String text;
NewClass(this.number, this.text);
}
Function that generates the values:
NewClass buildValues() {
return NewClass(42, 'foobar');
}
Print:
void printValues() {
print('${this.buildValues().number} ${this.buildValues().text}');
// 42 foobar
}
The proper way to return multiple values would be to store those values in a class, whether your own custom class or a Tuple. However, defining a separate class for every function is very inconvenient, and using Tuples can be error-prone since the members won't have meaningful names.
Another (admittedly gross and not very Dart-istic) approach is try to mimic the output-parameter approach typically used by C and C++. For example:
class OutputParameter<T> {
T value;
OutputParameter(this.value);
}
void foo(
OutputParameter<int> intOut,
OutputParameter<String>? optionalStringOut,
) {
intOut.value = 42;
optionalStringOut?.value = 'foobar';
}
void main() {
var theInt = OutputParameter(0);
var theString = OutputParameter('');
foo(theInt, theString);
print(theInt.value); // Prints: 42
print(theString.value); // Prints: foobar
}
It certainly can be a bit inconvenient for callers to have to use variable.value everywhere, but in some cases it might be worth the trade-off.
you can use dartz package for Returning multiple data types
https://www.youtube.com/watch?v=8yMXUC4W1cc&t=110s
Dart is finalizing records, a fancier tuple essentially.
Should be in a stable release a month from the time of writing.
I'll try to update, it's already available with experiments flags.
you can use Set<Object> for returning multiple values,
Set<object> foo() {
return {'my string',0}
}
print(foo().first) //prints 'my string'
print(foo().last) //prints 0
In this type of situation in Dart, an easy solution could return a list then accessing the returned list as per your requirement. You can access the specific value by the index or the whole list by a simple for loop.
List func() {
return [false, 30, "Ashraful"];
}
void main() {
final list = func();
// to access specific list item
var item = list[2];
// to check runtime type
print(item.runtimeType);
// to access the whole list
for(int i=0; i<list.length; i++) {
print(list[i]);
}
}
I am implementing a Heartbeat for a WebFlux SSE endpoint. To avoid a timeout in the client, I want to make sure that an element is emitted at least every, say, 10 seconds.
I came up with the following solution that emits a heartbeat element every 10 seconds regardless of whether a real element has been emitted or not:
originalFlux.mergeWith(Flux.interval(Duration.ofSeconds(10), Duration.ofSeconds(10)).map(ignored -> "heartbeat")
This is probably good enough for my use case but still I wonder if it is possible to emit the heartbeat only if no real element has been emitted in the last 10 seconds. I played around with the timeout operator which implements exactly the timing behavior I am looking for, but that emits an error and cancels the originalFlux instead of just emitting an extra element.
The following code using timeout passes my test but looks too complicated and as far as I understand could lose elements from the originalFlux if they are emitted between cancelling and re-subscribing to it:
ConnectableFlux<String> sharedOriginalFlux = originalFlux.publish();
CompletableFuture<Disposable> eventualSubscription = new CompletableFuture<>();
return addHeartbeat(sharedOriginalFlux)
.doOnSubscribe(ignored -> eventualSubscription.complete(sharedOriginalFlux.connect()))
.doFinally(ignored -> eventualSubscription.thenAccept(Disposable::dispose))
private Flux<String> addHeartbeat(Flux<String> sharedOriginalFlux) {
return sharedOriginalFlux.timeout(
Duration.ofSeconds(10),
Flux.mergeSequential(
Mono.just("heartbeat"),
Flux.defer(() -> addHeartbeat(sharedOriginalFlux))));
}
Is there a simple and safe way to do this?
It's not necessarily simpler, but another option could be to create a separate processor that can wrap the original Flux to provide a heartbeat (which shouldn't miss any elements):
public class HeartbeatProcessor<T> {
private final FluxProcessor<T, T> processor;
private final FluxSink<T> sink;
private final T heartbeatValue;
private final Duration heartbeatPeriod;
private Disposable d;
public HeartbeatProcessor(Flux<T> orig, T heartbeatValue, Duration heartbeatPeriod) {
this.heartbeatValue = heartbeatValue;
this.heartbeatPeriod = heartbeatPeriod;
this.processor = DirectProcessor.<T>create().serialize();
this.sink = processor.sink();
this.d = Mono.just(heartbeatValue).delayElement(heartbeatPeriod).subscribe(this::emit);
orig.subscribe(this::emit);
}
private void emit(T val) {
sink.next(val);
d.dispose();
this.d = Mono.just(heartbeatValue).delayElement(heartbeatPeriod).subscribe(this::emit);
}
public Flux<T> getFlux() {
return processor;
}
}
You could then call it as follows:
new HeartbeatProcessor<>(elements, "heartbeat", Duration.ofSeconds(10))
.getFlux()
.subscribe(System.out::println);
Flux.switchMap is a good candidate for this job: It switches to a new Publisher (and cancels the previous one) whenever the original Flux emits an item. In your case, the new Publisher is your heartbeat Flux.interval, prepended with the original item T:
public static Flux<String> addHeartbeat(Flux<String> originalFlux) {
return originalFlux
.startWith("heartbeat")
.materialize()
.switchMap(signal -> switch (signal.getType()) {
case ON_NEXT -> Flux.interval(Duration.ofSeconds(10))
.map(ignored -> "heartbeat")
.startWith(signal.get());
case ON_COMPLETE -> Mono.empty();
case ON_ERROR -> Mono.error(signal.getThrowable());
default -> Mono.error(new IllegalStateException());
});
}
Flux.switchMap is almost fit for the job, but it differs on two points from your desired solution:
It will only emit elements once the first element is received.
This means you have no heartbeat before the first item. This is solved by adding Flux.startWith("heartbeat"), which will emit "heartbeat" immediately on subscription, which then is processed by the switchMap into a heartbeat every 10 seconds.
The Publisher of the last element is never cancelled.
Since every generated Publisher is a Flux.interval that never completes, the onComplete signal will never reach the user. This is solved by transforming the onComplete signal into an emitted Signal item using Flux.materialize(), then map the onComplete Signal into an empty Publisher just to cancel the previous Publisher. This also creates onNext and onError Signals, which we have to handle each:
a. Signal.ON_NEXT can be processed as usual, retrieving the original with Signal.get()
b. Signal.ON_COMPLETE is mapped to an empty Mono that immediately completes.
c. Signal.ON_ERROR should relay the error downstream using Mono.error(Throwable).
d. The Signal enum contains more values, but they are not produced by Flux.materialize().
Here is the test to test this solution:
#Test
public void shouldAddHeartbeat() {
Flux<String> originalFlux = Flux.just(25, 15, 7, 5)
.concatMap(delay -> Mono.delay(Duration.ofSeconds(delay)).thenReturn(delay + " seconds delay"));
Flux<String> withHeartbeat = addHeartbeat(originalFlux);
StepVerifier.withVirtualTime(() -> withHeartbeat)
.expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(5)).expectNext("25 seconds delay")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(5)).expectNext("15 seconds delay")
.thenAwait(Duration.ofSeconds(7)).expectNext("7 seconds delay")
.thenAwait(Duration.ofSeconds(5)).expectNext("5 seconds delay")
.verifyComplete();
}
I have some initial state in my application and a few of policies that decorates this state with reactively fetched data (each of policy's Mono returns new instance of state with additional data). Eventually I get fully decorated state.
It basically looks like this:
public interface Policy {
Mono<State> apply(State currentState);
}
Usage for fixed number of policies would look like that:
Flux.just(baseState)
.flatMap(firstPolicy::apply)
.flatMap(secondPolicy::apply)
...
.subscribe();
It basically means that entry state for a Mono is result of accumulation of initial state and each of that Mono predecessors.
For my case policies number is not fixed and it comes from another layer of the application as a collection of objects that implements Policy interface.
Is there any way to achieve similar result as in the given code (with 2 flatMap), but for unknown number of policies? I have tried with Flux's reduce method, but it works only if policy returns value, not a Mono.
This seems difficult because you're streaming your baseState, then trying to do an arbitrary number of flatMap() calls on that. There's nothing inherently wrong with using a loop to achieve this, but I like to avoid that unless absolutely necessary, as it breaks the natural reactive flow of the code.
If you instead iterate and reduce the policies into a single policy, then the flatMap() call becomes trivial:
Flux.fromIterable(policies)
.reduce((p1,p2) -> s -> p1.apply(s).flatMap(p2::apply))
.flatMap(p -> p.apply(baseState))
.subscribe();
If you're able to edit your Policy interface, I'd strongly suggest adding a static combine() method to reference in your reduce() call to make that more readable:
interface Policy {
Mono<State> apply(State currentState);
public static Policy combine(Policy p1, Policy p2) {
return s -> p1.apply(s).flatMap(p2::apply);
}
}
The Flux then becomes much more descriptive and less verbose:
Flux.fromIterable(policies)
.reduce(Policy::combine)
.flatMap(p -> p.apply(baseState))
.subscribe();
As a complete demonstration, swapping out your State for a String to keep it shorter:
interface Policy {
Mono<String> apply(String currentState);
public static Policy combine(Policy p1, Policy p2) {
return s -> p1.apply(s).flatMap(p2::apply);
}
}
public static void main(String[] args) {
List<Policy> policies = new ArrayList<>();
policies.add(x -> Mono.just("blah " + x));
policies.add(x -> Mono.just("foo " + x));
String baseState = "bar";
Flux.fromIterable(policies)
.reduce(Policy::combine)
.flatMap(p -> p.apply(baseState))
.subscribe(System.out::println); //Prints "foo blah bar"
}
If I understand the problem correctly, then the most simple solution is to use a regular for loop:
Flux<State> flux = Flux.just(baseState);
for (Policy policy : policies)
{
flux = flux.flatMap(policy::apply);
}
flux.subscribe();
Also, note that if you have just a single baseSate you can use Mono instead of Flux.
UPDATE:
If you are concerned about breaking the flow, you can extract the for loop into a method and apply it via transform operator:
Flux.just(baseState)
.transform(this::applyPolicies)
.subscribe();
private Publisher<State> applyPolicies(Flux<State> originalFlux)
{
Flux<State> newFlux = originalFlux;
for (Policy policy : policies)
{
newFlux = newFlux.flatMap(policy::apply);
}
return newFlux;
}
I have a simple DataFlow java job that reads a few lines from a .csv file. Each line contains a numeric cell, which represents how many steps a certain function has to be performed on that line.
I don't want to perform that using a traditional For loop within the function, in case these numbers become very large. What is the right way to do this using the parallel-friendly DataFlow methodology?
Here's the current Java code:
public class SimpleJob{
static class MyDoFn extends DoFn<String, Integer> {
public void processElement(ProcessContext c) {
String name = c.element().split("\\,")[0];
int val = Integer.valueOf(c.element().split("\\,")[1]);
for (int i = 0; i < val; i++) // <- what's the preferred way to do this in DF?
System.out.println("Processing some function: " + name); // <- do something
c.output(val);
}
}
public static void main() {
DataflowPipelineOptions options = PipelineOptionsFactory
.as(DataflowPipelineOptions.class);
options.setProject(DEF.ID_PROJ);
options.setStagingLocation(DEF.ID_STG_LOC);
options.setRunner(DirectPipelineRunner.class);
Pipeline pipeline = Pipeline.create(options);
pipeline.apply(TextIO.Read.from("Source.csv"))
.apply(ParDo.of(new MyDoFn()));
pipeline.run();
}
}
This is what the "source.csv" looks like (so each number represents how many times I want to run a parallel function on that line):
Joe,3
Mary,4
Peter,2
Curiously enough, this is one of the motivating use cases for Splittable DoFn! That API is currently in heavy development.
However, until that API is available, you can basically mimic most of what it would have done for you:
class ElementAndRepeats { String element; int numRepeats; }
PCollection<String> lines = p.apply(TextIO.Read....)
PCollection<ElementAndRepeats> elementAndNumRepeats = lines.apply(
ParDo.of(...parse number of repetitions from the line...));
PCollection<ElementAndRepeats> elementAndNumSubRepeats = elementAndNumRepeats
.apply(ParDo.of(
...split large numbers of repetitions into smaller numbers...))
.apply(...fusion break...);
elementAndNumSubRepeats.apply(ParDo.of(...execute the repetitions...))
where:
"split large numbers of repetitions" is a DoFn that, e.g., splits an ElementAndRepeats{"foo", 34} into {ElementAndRepeats{"foo", 10}, ElementAndRepeats{"foo", 10}, ElementAndRepeats{"foo", 10}, ElementAndRepeats{"foo", 4}}
fusion break - see here, to prevent the several ParDo's from being fused together, defeating the parallelization