RxJava2 has a doAfterNext operator that emits items downstream, and then invokes the consumer. It doesn't seem like Project Reactor has such an operator so I'd like to get some pointers on the best way to create my own to achieve the same thing.
The use case is freeing memory after the subscriber has received the item
Not sure if leavering doOnEach is a valid solution:
public class ByteBufferSafeReleaseConsumer implements Consumer<Signal<ByteBuffer<?>>> {
private final List<ByteBuffer<?>> elements = new ArrayList<>();
#Override
public void accept(Signal<ByteBuffer<?>> signal) {
if (signal.isOnNext()) {
ByteBuffer<?> next = signal.get();
if (next != null) {
elements.add(next);
}
}
if (signal.isOnComplete() || signal.isOnError()) {
for (ByteBuffer<?> buffer : elements) {
ByteBufferUtils.safeRelease(buffer);
}
}
}
}
ByteBufferSafeReleaseConsumer consumer = new ByteBufferSafeReleaseConsumer()
Flux.from(byteBufferPublisher).doOnEach(consumer)
Related
I am creating a batch data streamer in apache ignite, and need to control what happening after data receive.
My batch has a structure:
public class Batch implements Binarylizable, Serializable {
private String eventKey;
private byte[] bytes;
etc..
Then i trying to stream my data:
try (IgniteDataStreamer<Integer, Batch> streamer = serviceGrid.getIgnite().dataStreamer(cacheName);
StreamBatcher batcher = StreamBatcherFactory.create(event) ){
streamer.receiver(StreamTransformer.from(new BatchDataProcessor(event)));
streamer.autoFlushFrequency(1000);
streamer.allowOverwrite(true);
statusService.updateStatus(event.getKey(), StatusType.EXECUTING);
int counter = 0;
Batch batch = null;
IgniteFuture<?> future = null;
while ((batch = batcher.batch()) != null) {
future = streamer.addData(counter++, batch);
}
Object getted = future.get();
Just for test use lets get only the last future, and try to analyze this object. In the code above I'm using BatchDataProcessor, that look like this:
public class BatchDataProcessor implements CacheEntryProcessor<Integer, Batch, Object> {
private final Event event;
private final String eventKey;
public BatchDataProcessor(Event event) {
this.event = event;
this.eventKey = event.getKey();
}
#Override
public Object process(MutableEntry<Integer, Batch> mutableEntry, Object... objects) throws EntryProcessorException {
Node node = NodeIgniter.node(Ignition.localIgnite().cluster().localNode().id());
ServiceGridContainer container = (ServiceGridContainer) node.getEnvironmentContainer().getContainerObject(ServiceGridContainer.class);
ProcessMarshaller marshaller = (ProcessMarshaller) container.getService(ProcessMarshaller.class);
LocalProcess localProcess = marshaller.intoProccessing(event.getLambdaExecutionKey());
try {
localProcess.addBatch(mutableEntry);
} catch (IOException e) {
e.printStackTrace();
} finally {
return new String("111");
}
}
}
So after localProcess.addBatch(mutableEntry) I want to send back an information about the status of this particular batch, so I think that I should do this in IgniteFuture object, but I don't find any information how to control the future object that's received in addData function.
Can anybody help with understanding, where can I control future that receives in addData function or some other way to realize a callback to streamed batch?
When you do StreamTransformer.from(), you forfeit the result of your BatchDataProcessor, because
for (Map.Entry<K, V> entry : entries)
cache.invoke(entry.getKey(), this, entry.getValue());
// ^ result of cache.invoke() is discarded here
DataStreamer is for one-directional streaming of data. It is not supposed to return values as far as I know.
If you depend on the result of cache.invoke(), I recommend calling it directly instead of relying on DataStreamer.
BTW, be careful with fut.get(). You should do dataStreamer.flush() first, or DataStreamer's futures will wait indefinitely.
I'm new to reactive programming using rxjava and after going through the simpler examples I'm now trying to figure out how to work with continuous streams. The problem I have with the example below is that the program doesn't terminate after I've taken the 3 elements. My assumption is that I somehow need to unsubscribe to my observable but I don't fully grasp how to terminate the while loop and make the program exit.
I've come across the following post RxJava -- Terminating Infinite Streams but I still can't figure out what I'm missing.
class MyTwitterDataProvider {
/*
This example is written in Groovy
Instance variables and constructor omitted
*/
public Observable<String> getTweets() {
BufferedReader reader = new BufferedReader(new InputStreamReader(getTwitterStream()))
Observable.create({ observer ->
executor.execute(new Runnable() {
def void run() {
String newLine
while ((newLine = reader.readLine()) != null) {
System.out.println("printing tweet: $newLine")
observer.onNext(newLine)
}
observer.onCompleted()
}
})
})
}
def InputStream getTwitterStream() {
// code omitted
}
public static void main (String [] args) {
MyTwitterDataProvider provider = new MyTwitterDataProvider()
Observable<String> myTweetsObservable = provider.getTweets().take(3)
Subscription myTweetSubscription = myTweetsObservable.subscribe({tweet-> println("client prints: $tweet")})
// myTweetSubscription.unsubscribe()
}
}
You must add a check in your loop to see if the observer is still subscribed:
while ((newLine = reader.readLine()) != null && !observer.isUnsubsribed()) {
System.out.println("printing tweet: $newLine")
observer.onNext(newLine)
}
so I was asked this question:
Consider the ADT stack. In addition to the operations Push, Pop
and Top, we want to support a new operation FindMin, which returns the smallest
element in the stack. Design the data structure and algorithms to support these
operations such that each of the four operations (Push, Pop, Top and FindMin) takes
constant time. No need to check the over and under conditions and no need
to give the procedures for Empty and Full. [Hint: use an extra stack.]
So I have seen some answers but they all seemed to take O(n) time once the FindMin function is used. And I don't really get what the hint is trying to tell me...Please help me!!! Thankssss!!
import java.util.Stack;
public class StackWithMin extends Stack<Integer> {
private Stack<Integer> minStack;
public StackWithMin () {
minStack = new Stack<Integer>();
}
public void push(int value){
if (value <= min()) { // Note the '=' sign here
minStack.push(value);
}
super.push(value);
}
public Integer pop() {
int value = super.pop();
if (value == min()) {
minStack.pop();
}
return value;
}
public int min() {
if (minStack.isEmpty()) {
return Integer.MAX_VALUE;
} else {
return minStack.peek();
}
}
public static void main(String args[]) {
StackWithMin stackWithMin = new StackWithMin();
stackWithMin.push(7);
stackWithMin.push(5);
stackWithMin.push(6);
stackWithMin.push(7);
stackWithMin.push(4);
System.out.println(stackWithMin.min());
stackWithMin.pop();
System.out.println(stackWithMin.min());
}
}
I've just recently started using Rhino-Etl for very simple ETL processes and have had great success with it. I have a slightly more complicated scenario to address now and I didn't find the ConventionInputCommandOperation behaving the way I expected.
I've done up a very simplified example of what I'm trying to do. Basically I have two systems involved and I don't know what I want to get from system 2 until I first query system 1. I thought registering an InputOperation immediately after another InputOperation would behave like a loop. So that each row in operation 1 would be fed to operation 2. The below code fails with "Failed to execute operation DetailReader: Must declare the scalar variable #PlanetAbbrv." So my question is how are you meant to handle situations where the input operation is dependent a previous input operation?
Thanks,
Brian
using System;
using Rhino.Etl.Core;
using Rhino.Etl.Core.ConventionOperations;
namespace ETLTest
{
class Program
{
static void Main()
{
new MainProcess().Execute();
Console.ReadLine();
}
}
public class MainProcess : EtlProcess
{
protected override void Initialize()
{
Register(new MainReader());
Register(new DetailReader());
}
protected override void PostProcessing()
{
foreach (var exception in GetAllErrors())
{
throw exception;
}
}
}
public class MainReader : ConventionInputCommandOperation
{
public MainReader() : base("Galactic1")
{
Command = #"select * from Planet";
}
}
public class DetailReader : ConventionInputCommandOperation
{
public DetailReader() : base("Galactic2")
{
Command = #"select * from Delivery where DeliveryPlanetAbbrv = #PlanetAbbrv";
}
}
}
You'll need to have your DetailReader select all rows (take out the where operation).
Then use a JoinOperation to match the details to the main information.
Register(new JoinPlanets()
.Right(new MainReader())
.Left(new DetailReader()));
public class JoinPlanets: JoinOperation
{
protected override Row MergeRows(Row leftRow, Row rightRow)
{
Row row = leftRow.Clone();
foreach (var column in leftRow.Columns)
row[column] = leftRow[column];
return row;
}
protected override void SetupJoinConditions()
{
FullOuterJoin.Left("PlanetAbbrv")
.Right("DeliveryPlanetAbbrv");
}
}
I'm rolling my own ActivatableCollection<T> for db4o but cribbing heavily from the builtin ActivatableList<T> implementation. I'm running into the problem where transparent persistence doesn't seem to be working correctly. In the test code below:
[Fact]
void CanStoreActivatableCollection()
{
var planets = new ActivatableCollection<Planet>();
var pagingMemoryStorage = new PagingMemoryStorage();
var config = Db4oEmbedded.NewConfiguration();
config.Common.Add(new TransparentActivationSupport());
config.Common.Add(new TransparentPersistenceSupport());
config.File.Storage = pagingMemoryStorage;
var objectContainer = Db4oEmbedded.OpenFile(config, "Memory.yap");
planets.Add(new Planet("Mercury"));
objectContainer.Store(planets);
planets.Add(new Planet("Venus"));
planets.Add(new Planet("Earth"));
objectContainer.Commit();
objectContainer.Close();
config = Db4oEmbedded.NewConfiguration();
config.Common.Add(new TransparentActivationSupport());
config.Common.Add(new TransparentPersistenceSupport());
config.File.Storage = pagingMemoryStorage;
objectContainer = Db4oEmbedded.OpenFile(config, "Memory.yap");
planets = objectContainer.Query<ActivatableCollection<Planet>>().FirstOrDefault();
Assert.NotNull(planets);
Assert.Equal(3, planets.Count);
objectContainer.Close();
}
The planet "Mercury" is stored, but not "Venus" and "Earth". If I change from ActivatableCollection to ActivatableList, then all 3 planets are stored.
What am I missing? My ActivatableCollection is just minimal implementation of ActivatableList as best as I can tell.
Below is my implementation of ActivatableCollection:
public class ActivatableCollection<T>
: ICollection<T>
, IActivatable
, INotifyCollectionChanged
{
List<T> _list;
List<T> List
{
get
{
if (_list == null)
_list = new List<T>();
return _list;
}
}
public ActivatableCollection()
{
}
public int Count
{
get
{
ActivateForRead();
return List.Count;
}
}
public bool IsReadOnly
{
get
{
ActivateForRead();
return ((IList) List).IsReadOnly;
}
}
public void Add(T t)
{
ActivateForWrite();
List.Add(t);
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Add, t));
}
public void Clear()
{
ActivateForWrite();
List.Clear();
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Reset));
}
public bool Contains(T t)
{
ActivateForRead();
return List.Contains(t);
}
public void CopyTo(T[] array, int index)
{
ActivateForRead();
List.CopyTo(array, index);
}
public IEnumerator<T> GetEnumerator()
{
ActivateForRead();
return List.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public bool Remove(T t)
{
ActivateForWrite();
bool removed = List.Remove(t);
if (removed)
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Remove, t));
return removed;
}
[Transient]
private IActivator _activator;
public virtual void Bind(IActivator activator)
{
if (_activator == activator)
return;
if (activator != null && _activator != null)
throw new InvalidOperationException();
_activator = activator;
}
public virtual void Activate(ActivationPurpose purpose)
{
if (_activator == null)
return;
_activator.Activate(purpose);
}
protected virtual void ActivateForRead()
{
Activate(ActivationPurpose.Read);
}
protected virtual void ActivateForWrite()
{
Activate(ActivationPurpose.Write);
}
[Transient]
public event NotifyCollectionChangedEventHandler CollectionChanged;
protected virtual void OnCollectionChanged(NotifyCollectionChangedEventArgs e)
{
if (CollectionChanged != null)
CollectionChanged(this, e);
}
}
I've also tried copying the code from GenericTypeHandlerPredicate and registering my ActivatableCollection to use the GenericCollectionTypeHandler. That results in a crash in GenericTypeFor() throwing an InvalidOperationException() when "Mercury" is being stored.
Just want to mention my answers from the db4o forums also here, for people with a similar problem:
First part of the issue:
From db4o's point of view nothing has changed in the 'ActivatableCollection' object and therefore no changes are stored. This is what is happening:
When you add the items, the ActivatableCollection is marked as changed.
When you commit the changes are stored. However the ' ActivatableCollection' holds the reference to the same object. db4o only stores the changes in the ActivatableCollection-object, which is the reference to the List. Since it is the same, no actual change is stored.
The List of the ActivatableCollection is never updated, because it wasn't marked as 'changed'
So the transparent activation doesn't see the changes in the list. You can fix your issue simply by using an ActivatableList in you're ActivatableCollection implementation. Just change the List with a IList interface and instantiate a ActivatableList instead of an List.
The second part of the issue: Why doesn't it work even when registering the GenericCollectionTypeHandler for this type? Here we hit a implementation detail. The GenericCollectionTypeHandler has an internal list of supported types, which doesn't include the self made 'ActivatableCollection'. GenericCollectionTypeHandler is not really part of the public API and intendet for internal use only.
Workaround / Fix
Just use an ActivatableList<T> instead of a List<T>. then everything works fine.