Project reactor backpressure (buffer size?) issue - project-reactor

My goal is to process gui events in parallel, but only when I have processing power plus the last event must always be processed. I have a panel which can be resized. Every resize produces new event. I want to process new width, height of panel on a computation thread pool (Scheduler newParallel = Schedulers.newParallel("Computation", 4);) in ordered fashion. If none of the threads is available I need to drop the oldest gui events and when thread becomes available it should take the latest from the backpressure queue.
I wrote the test app and I have several issues. After gui events stop to being produced there is still considerable time when processing is done which ultimately will manifest as an unwanted animation effect. My guess is that the backpressure queue size=256 kept the old events and is still processing it but it does not match with the result logs. After producing 561 events only 33 events were processed (why not 256?) with ids [0-32, 560]. Is there a way to change the backpressure buffer size (I could not find a way to do it) or maybe there is totally different way I should approach this task?
I attach test code for recreation.
import java.util.Random;
import java.util.concurrent.TimeUnit;
import javafx.application.Application;
import javafx.beans.value.ChangeListener;
import javafx.scene.Scene;
import javafx.scene.layout.StackPane;
import javafx.stage.Stage;
import reactor.core.publisher.Flux;
import reactor.core.publisher.FluxSink.OverflowStrategy;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Scheduler;
import reactor.core.scheduler.Schedulers;
public class BackpressureApp extends Application {
public static void main(String[] args) {
launch(args);
}
private int id = 0;
#Override
public void start(Stage stage) throws Exception {
Scheduler computation = Schedulers.newParallel("Computation", 4);
Flux<Width> flux = Flux.create(sink -> {
stage.widthProperty().addListener((ChangeListener<Number>) (observable, oldValue, newValue) -> {
Width width = new Width(id++, newValue.doubleValue());
System.out.println("[" + Thread.currentThread().getName() + "] PUBLISH width=" + width);
sink.next(width);
});
}, OverflowStrategy.LATEST);
flux.concatMap(width -> Mono.just(width).subscribeOn(computation).map(this::process))
.publishOn(Schedulers.single())
.subscribe(width -> {
System.out.println("[" + Thread.currentThread().getName() + "] RECEIVED width=" + width);
});
stage.setScene(new Scene(new StackPane()));
stage.show();
}
public Width process(Width width) {
Random random = new Random();
int next = random.nextInt(1000);
try {
TimeUnit.MILLISECONDS.sleep(next);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("[" + Thread.currentThread().getName() + "] PROCESS width=" + width + " sleep=" + next);
return width;
}
}
class Width {
private final int id;
private final double width;
public Width(int id, double width) {
super();
this.id = id;
this.width = width;
}
public int getId() {
return id;
}
public double getWidth() {
return width;
}
#Override
public String toString() {
return "Width[id=" + id + ", width=" + width + "]";
}
}

Related

Project Reactor backpressure issue

I am using reactor in a project, and one of the features calls a blocking service, which connects to a device and gets an infinite stream of events.
I am trying to do a load test to see how many calls can I make to the blocking service.
I am generating around 1000 requests to the blocking service
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250)
The problem is that reactor is only processing the first 256 requests, after that it isn't making any more requests.
When I added the .log("preConnect") I can see that it is logging only one request(256) from the downstream subscriber.
I don't understand what I am doing wrong.
I am attaching simplified example which can reproduce the issue.
package test.reactor;
import org.junit.jupiter.api.Test;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;
import java.util.stream.Stream;
public class ReactorTest {
#Test
void testLoad() throws InterruptedException {
AtomicInteger id = new AtomicInteger(0);
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking)
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();
new CountDownLatch(1).await();
}
private Flux<String> blocking(String ip) {
Mono<String> connectMono = Mono.fromCallable(this::connect)
.subscribeOn(Schedulers.boundedElastic())
.map(msg -> "Connected: "+ip + msg);
Flux<String> streamFlux = Mono.fromCallable(this::infiniteNetworkStream)
.subscribeOn(Schedulers.boundedElastic())
.flatMapMany(Flux::fromStream)
.map(msg -> ip + msg);
return connectMono.concatWith(streamFlux);
}
private Stream<String> infiniteNetworkStream() {
return Stream.generate(new Supplier<String>() {
#Override
public String get() {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
return "Hello";
}
});
}
private String connect() throws Exception{
Thread.sleep(100);
return "success";
}
}
Figured out the issue, it has to do with flatmap, the default concurrency for flatmap is 256. It will not request more items from the upstream publisher until the current subscriptions go below 256.
In my case since my flux is infinite, it wasn't processing any after 256.
The solution I found was to increase the concurrency
Flux.just("ip1", "ip2", "ip3", "ip4")
.repeat(250) // will create a total of 1004 messages
.map(str -> str + " id: " + id.incrementAndGet())
.log("preConnect")
.flatMap(this::blocking, 1000) // added 1000 here to increase concurrency
.log()
.subscribeOn(Schedulers.parallel())
.subscribe();

proper video streaming with rxjava

To handle a video stream from a webcam (delivered by opencv) i am considering to use RxJava.
I am hoping to achieve the following:
being able to control the frames per second to be delivered
to be able to handle different inputs - e.g. a life webcam, a video or even a still picture
being able to switch to a picture-by-picture handling under the control of a gui
I have been experimenting a bit with RxJava but i am confused about the debounce, throttleFirst and async operators
Examples like https://stackoverflow.com/a/48723331/1497139 show some code but I am missing more detailed explanation.
Where could I find a decent example for video processing or something similar along the needs mentioned above?
The code below does some non async logic at this time - please let me know if i could build on it:
ImageFetcher
import org.opencv.core.Mat;
import org.opencv.videoio.VideoCapture;
import rx.Observable;
import rx.functions.Action1;
import rx.functions.Func0;
import rx.functions.Func1;
/**
* fetcher for Images
*
*/
public class ImageFetcher {
// OpenCV video capture
private VideoCapture capture = new VideoCapture();
private String source;
protected int frameIndex;
public int getFrameIndex() {
return frameIndex;
}
/**
* fetch from the given source
*
* #param source
* - the source to fetch from
*/
public ImageFetcher(String source) {
this.source = source;
}
/**
* try opening my source
*
* #return true if successful
*/
public boolean open() {
boolean ret = this.capture.open(source);
frameIndex=0;
return ret;
}
/**
* fetch an image Matrix
*
* #return - the image fetched
*/
public Mat fetch() {
if (!this.capture.isOpened()) {
boolean ret = this.open();
if (!ret) {
String msg = String.format(
"Trying to fetch image from unopened VideoCapture and open %s failed",
source);
throw new IllegalStateException(msg);
}
}
final Mat frame = new Mat();
this.capture.read(frame);
frameIndex++;
return !frame.empty() ? frame : null;
}
#Override
protected void finalize() throws Throwable {
super.finalize();
}
/**
* convert me to an observable
* #return a Mat emitting Observable
*/
public Observable<Mat> toObservable() {
// Resource creation.
Func0<VideoCapture> resourceFactory = () -> {
VideoCapture capture = new VideoCapture();
capture.open(source);
return capture;
};
// Convert to observable.
Func1<VideoCapture, Observable<Mat>> observableFactory = capture -> Observable
.<Mat> create(subscriber -> {
boolean hasNext = true;
while (hasNext) {
final Mat frame = this.fetch();
hasNext = frame!=null && frame.rows()>0 && frame.cols()>0;
if (hasNext) {
String msg = String.format("->%6d:%4dx%d", frameIndex, frame.cols(), frame.rows());
System.out.println(msg);
subscriber.onNext(frame);
}
}
subscriber.onCompleted();
});
// Disposal function.
Action1<VideoCapture> dispose = VideoCapture::release;
return Observable.using(resourceFactory, observableFactory, dispose);
}
}
ImageSubscriber
import org.opencv.core.Mat;
import rx.Subscriber;
public class ImageSubscriber extends Subscriber<Mat> {
public Throwable error;
public int cols = 0;
public int rows=0;
public int frameIndex=0;
public boolean completed = false;
public boolean debug = false;
#Override
public void onCompleted() {
completed = true;
}
#Override
public void onError(Throwable e) {
error = e;
}
#Override
public void onNext(Mat mat) {
cols = mat.cols();
rows = mat.rows();
frameIndex++;
if (cols==0 || rows==0)
System.err.println("invalid frame "+frameIndex);
if (debug) {
String msg = String.format("%6d:%4dx%d", frameIndex, cols, rows);
System.out.println(msg);
}
}
};

How to Batch By N Elements in Streaming Pipeline With Small Bundles?

I've implemented batching by N elements as described in this answer:
Can datastore input in google dataflow pipeline be processed in a batch of N entries at a time?
package com.example.dataflow.transform;
import com.example.dataflow.event.ClickEvent;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
import org.joda.time.Instant;
import java.util.ArrayList;
import java.util.List;
public class ClickToClicksPack extends DoFn> {
public static final int BATCH_SIZE = 10;
private List accumulator;
#StartBundle
public void startBundle() {
accumulator = new ArrayList(BATCH_SIZE);
}
#ProcessElement
public void processElement(ProcessContext c) {
ClickEvent clickEvent = c.element();
accumulator.add(clickEvent);
if (accumulator.size() >= BATCH_SIZE) {
c.output(accumulator);
accumulator = new ArrayList(BATCH_SIZE);
}
}
#FinishBundle
public void finishBundle(FinishBundleContext c) {
if (accumulator.size() > 0) {
ClickEvent clickEvent = accumulator.get(0);
long time = clickEvent.getClickTimestamp().getTime();
c.output(accumulator, new Instant(time), GlobalWindow.INSTANCE);
}
}
}
But when I run pipeline in streaming mode there are a lot of batches with just 1 or 2 elements. As I understand it's because of small bundles size. After running for a day average number of elements in batch is roughly 4. I really need it to be closer to 10 for better performance of the next steps.
Is there a way to control bundles size?
Or should I use "GroupIntoBatches" transform for this purpose. In this case it's not clear for me, what should be selected as a key.
UPDATE:
is it a good idea to use java thread id or VM hostname for a key to apply "GroupIntoBatches" transform?
I've ended up doing composite transform with "GroupIntoBatches" inside.
The following answer contains recommendations regarding key selection:
https://stackoverflow.com/a/44956702/4888849
In my current implementation I'm using random keys to achieve parallelism and I'm windowing events in order to emit results regularly even if there are less then BATCH_SIZE events by one key.
package com.example.dataflow.transform;
import com.example.dataflow.event.ClickEvent;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.GroupIntoBatches;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.joda.time.Duration;
import java.util.Random;
/**
* Batch clicks into packs of BATCH_SIZE size
*/
public class ClickToClicksPack extends PTransform, PCollection>> {
public static final int BATCH_SIZE = 10;
// Define window duration.
// After window's end - elements are emitted even if there are less then BATCH_SIZE elements
public static final int WINDOW_DURATION_SECONDS = 1;
private static final int DEFAULT_SHARDS_NUMBER = 20;
// Determine possible parallelism level
private int shardsNumber = DEFAULT_SHARDS_NUMBER;
public ClickToClicksPack() {
super();
}
public ClickToClicksPack(int shardsNumber) {
super();
this.shardsNumber = shardsNumber;
}
#Override
public PCollection> expand(PCollection input) {
return input
// assign keys, as "GroupIntoBatches" works only with key-value pairs
.apply(ParDo.of(new AssignRandomKeys(shardsNumber)))
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(WINDOW_DURATION_SECONDS))))
.apply(GroupIntoBatches.ofSize(BATCH_SIZE))
.apply(ParDo.of(new ExtractValues()));
}
/**
* Assigns to clicks random integer between zero and shardsNumber
*/
private static class AssignRandomKeys extends DoFn> {
private int shardsNumber;
private Random random;
AssignRandomKeys(int shardsNumber) {
super();
this.shardsNumber = shardsNumber;
}
#Setup
public void setup() {
random = new Random();
}
#ProcessElement
public void processElement(ProcessContext c) {
ClickEvent clickEvent = c.element();
KV kv = KV.of(random.nextInt(shardsNumber), clickEvent);
c.output(kv);
}
}
/**
* Extract values from KV
*/
private static class ExtractValues extends DoFn>, Iterable> {
#ProcessElement
public void processElement(ProcessContext c) {
KV> kv = c.element();
c.output(kv.getValue());
}
}
}

Java process physical memory increases continuously

I've run into a situation where a simple canvas
Drawing app shows a gradual increase in physical
Memory usage over time until it hits about 80% mark
And tries to maintain the process memory usage around
That number (that is the %MEM column when 'top' is run, 64bit Linux box). Of course by that point page swapping kicks in. I've tried to limit memory usage by using -Xmx10m -Xms10m and even -XX:+UseG1GC but nothing appears to make a difference.
Just a side note, I've even tried the StockLineChartApp example shipped with oracle jdk Javafx examples and it displays a physical memory usage increase as well. I'm using jdk1.8.0_102.
Any feedback regarding this issue would be very helpful. Thanks.
//
package test8;
import javafx.application.Application;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Group;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.layout.Pane;
import javafx.stage.Stage;
import javafx.scene.canvas.Canvas;
import javafx.scene.canvas.GraphicsContext;
import javafx.scene.paint.Color;
import javafx.util.Duration;
import javafx.animation.KeyFrame;
import javafx.animation.KeyValue;
import javafx.animation.Timeline;
import javafx.animation.Animation;
public class Test8 extends Application {
GraphicsContext gc;
double x;
double y;
#Override
public void start(Stage primaryStage) {
try {
Group root = new Group();
Scene s = new Scene(root, 600, 400, Color.BLACK);
final Canvas canvas = new Canvas(500,300);
gc = canvas.getGraphicsContext2D();
root.getChildren().add(canvas);
Pane pane = new Pane();
pane.getChildren().add(root);
Scene scene = new Scene(pane);
primaryStage.setTitle("canvas test");
primaryStage.setScene(scene);
primaryStage.show();
gc.beginPath();
x = 0.0;
y = 0.0;
EventHandler onFinished = new EventHandler<ActionEvent>() {
public void handle(ActionEvent t) {
y = (y<canvas.getHeight()) ? (y + 10.0) : 0.0;
x = (x<canvas.getWidth()) ? (x + 5.0) : 0.0;
gc.lineTo(x,y);
gc.stroke();
if (x==0.0) gc.beginPath();
}
};
final Timeline timeline = new Timeline();
timeline.setCycleCount(Animation.INDEFINITE);
timeline.setAutoReverse(true);
timeline.getKeyFrames().add(new KeyFrame(Duration.millis(40), onFinished));
timeline.play();
} catch (Exception e) {
System.out.println("Application start Exception");
} finally {
}
}
public static void main(String[] args) {
launch(args);
}
}

win:length(2) is fired after first event

I made a very simple test gui based on this brilliant article about getting started with Esper.
What surprises me is that this query is validated to true after the very first tick event is sent, if the price is above 6.
select * from StockTick(symbol='AAPL').win:length(2) having avg(price) > 6.0
As far as I understand, win:length(2) needs TWO ticks before an event is fired, or am I wrong?
Here is a SSCCE for this question, just press the "Create Tick Event" button and you will see the StockTick Event being fired at once.
It needs the following jars which comes bundled with Esper
esper\lib\antlr-runtime-3.2.jar
esper\lib\cglib-nodep-2.2.jar
esper\lib\commons-logging-1.1.1.jar
esper\lib\esper_3rdparties.license
esper\lib\log4j-1.2.16.jar
esper-4.11.0.jar
import javax.swing.JFrame;
import javax.swing.JSplitPane;
import javax.swing.SwingUtilities;
import java.awt.BorderLayout;
import java.awt.Dimension;
import javax.swing.JButton;
import javax.swing.JScrollPane;
import java.awt.TextArea;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.util.Date;
import java.util.Random;
import javax.swing.JRadioButton;
import javax.swing.JPanel;
import java.awt.GridLayout;
import javax.swing.JTextArea;
import javax.swing.ScrollPaneConstants;
import com.espertech.esper.client.Configuration;
import com.espertech.esper.client.EPAdministrator;
import com.espertech.esper.client.EPRuntime;
import com.espertech.esper.client.EPServiceProvider;
import com.espertech.esper.client.EPServiceProviderManager;
import com.espertech.esper.client.EPStatement;
import com.espertech.esper.client.EventBean;
import com.espertech.esper.client.UpdateListener;
import javax.swing.JTextField;
public class Tester extends JFrame {
/**
*
*/
private static final long serialVersionUID = 1L;
JButton createRandomValueEventButton;
private JPanel panel;
private JPanel southPanel;
private JPanel centerPanel;
private static JTextArea centerTextArea;
private static JTextArea southTextArea;
private static Random generator = new Random();
private EPRuntime cepRT;
private JSplitPane textSplitPane;
private JButton btnNewButton;
private static JTextField priceTextField;
public Tester() {
getContentPane().setLayout(new BorderLayout(0, 0));
JSplitPane splitPane = new JSplitPane();
createRandomValueEventButton = new JButton("Create Tick Event With Random Price");
splitPane.setLeftComponent(createRandomValueEventButton);
createRandomValueEventButton.addActionListener(new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
createTickWithRandomPrice();
}
});
panel = new JPanel();
splitPane.setRightComponent(panel);
panel.setLayout(new GridLayout(1, 0, 0, 0));
btnNewButton = new JButton("Create Tick Event");
panel.add(btnNewButton);
btnNewButton.addActionListener(new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
createTick();
}
});
priceTextField = new JTextField();
priceTextField.setText(new Integer(10).toString());
panel.add(priceTextField);
priceTextField.setColumns(4);
getContentPane().add(splitPane, BorderLayout.NORTH);
textSplitPane = new JSplitPane();
textSplitPane.setOrientation(JSplitPane.VERTICAL_SPLIT);
getContentPane().add(textSplitPane, BorderLayout.CENTER);
centerPanel = new JPanel();
centerPanel.setLayout(new BorderLayout(0, 0));
JScrollPane centerTextScrollPane = new JScrollPane();
centerTextScrollPane.setVerticalScrollBarPolicy(ScrollPaneConstants.VERTICAL_SCROLLBAR_ALWAYS);
centerTextArea = new JTextArea();
centerTextArea.setRows(12);
centerTextScrollPane.setViewportView(centerTextArea);
southPanel = new JPanel();
southPanel.setLayout(new BorderLayout(0, 0));
JScrollPane southTextScrollPane = new JScrollPane();
southTextScrollPane.setVerticalScrollBarPolicy(ScrollPaneConstants.VERTICAL_SCROLLBAR_ALWAYS);
southTextArea = new JTextArea();
southTextArea.setRows(5);
southTextScrollPane.setViewportView(southTextArea);
textSplitPane.setRightComponent(southTextScrollPane);
textSplitPane.setLeftComponent(centerTextScrollPane);
setupCEP();
}
public static void GenerateRandomTick(EPRuntime cepRT) {
double price = (double) generator.nextInt(10);
long timeStamp = System.currentTimeMillis();
String symbol = "AAPL";
Tick tick = new Tick(symbol, price, timeStamp);
System.out.println("Sending tick:" + tick);
centerTextArea.append(new Date().toString()+" Sending tick:" + tick+"\n");
cepRT.sendEvent(tick);
}
public static void GenerateTick(EPRuntime cepRT) {
double price = Double.parseDouble(priceTextField.getText());
long timeStamp = System.currentTimeMillis();
String symbol = "AAPL";
Tick tick = new Tick(symbol, price, timeStamp);
System.out.println("Sending tick:" + tick);
centerTextArea.append(new Date().toString()+" Sending tick: " + tick+"\n");
cepRT.sendEvent(tick);
}
public static void main(String[] args){
Tester tester = new Tester();
tester.setSize(new Dimension(570,500));
tester.setVisible(true);
}
private void createTickWithRandomPrice(){
SwingUtilities.invokeLater(new Runnable() {
public void run() {
GenerateRandomTick(getEPRuntime());
}
});
}
private void createTick(){
SwingUtilities.invokeLater(new Runnable() {
public void run() {
GenerateTick(getEPRuntime());
}
});
}
private void setupCEP(){
Configuration cepConfig = new Configuration();
cepConfig.addEventType("StockTick", Tick.class.getName());
EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEPEngine", cepConfig);
cepRT = cep.getEPRuntime();
EPAdministrator cepAdm = cep.getEPAdministrator();
EPStatement cepStatement = cepAdm.createEPL(
"select * from " +
"StockTick(symbol='AAPL').win:length(2) " +
"having avg(price) > 6.0");
cepStatement.addListener(new CEPListener());
//System.out.println("cepStatement.getText(): "+cepStatement.getText());
}
private EPRuntime getEPRuntime(){
public static class Tick {
String symbol;
Double price;
Date timeStamp;
public Tick(String s, double p, long t) {
symbol = s;
price = p;
timeStamp = new Date(t);
}
public double getPrice() {return price;}
public String getSymbol() {return symbol;}
public Date getTimeStamp() {return timeStamp;}
#Override
public String toString() {
return symbol+" Price: " + price.toString();
}
}
public static class CEPListener implements UpdateListener {
}
Actually aggregation and conditions are independent of how many events are in data window. There are functions you could use to check whether a data window is "filled": the "leaving", "count" or "prevcount" for example.
For anyone interested,
changing the query to this solved the problem
select * from StockTick(symbol='AAPL').win:length_batch(2) having avg(price) > 6.0 and count(*) >= 2
Now an event will be triggered for every consecutive tick with the price higher than 6, in batches of two.

Resources