I'm having an issue with gRPC streaming in Java but I believe this problem also exists in other "reactive" contexts.
If you have a method like this (Kotlin):
#Transactional
override fun get(
request: DummyRequest,
responseObserver: StreamObserver<DummyResponse>,
) {
val counter = AtomicInteger()
myStreamingDatabaseQuery().use {
it.iterator().forEach { responseObserver.onNext(it) }
}
responseObserver.onCompleted()
}
fun myStreamingDatabaseQuery(): Stream<Dummy> = ...
It will work fine as the transaction is opened, the stream is processed and closed, and then thetransaction is closed.
However, you might then run into memory issues and try to use some sort of flow control like this:
#Transactional
override fun get(
request: DummyRequest,
responseObserver: StreamObserver<DummyResponse>,
) {
val counter = AtomicInteger()
val iterator = myStreamingDatabaseQuery().iterator()
StreamObservers.copyWithFlowControl(
iterator,
responseObserver as CallStreamObserver<DummyResponse>,
)
}
fun myStreamingDatabaseQuery(): Stream<Dummy> = ...
This won't work because StreamObservers just sets an onReadyHandler and immediately returns. The stream will then be processed in this handler, after get() has returned - and therefore the transaction will have been closed and it can no longer read the stream.
How is this commonly solved? And how would I do it with grpc-java/Spring?
Related
i am using Jetpack Compose 1.2.0 and Room 2.4.3
everything works well and my state changes when i use Read, Insert, Delete but i don't know why it does not work with update (when i navigate back or re enter the screen Its okay and i will get updated data)
this is my DAO
#Dao
abstract class MessageDAO : BaseDao<Message>() {
#Query("SELECT * FROM messages")
abstract fun getAllMessages(): LiveData<List<Message>>
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract override fun insert(obj: Message): Long
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract override fun insert(obj: MutableList<Message>?): MutableList<Long>
#Update
abstract override fun update(obj: Message)
#Update
abstract override fun update(obj: MutableList<Message>?)
#Delete
abstract override fun delete(obj: Message)
#Query("delete from messages where id in (:messageIDs)")
abstract fun delete(messageIDs: List<Long>)
}
also this my viewModel
#HiltViewModel
class MessagesViewModel #Inject constructor(
private val application: Application,
private val messageRepository: MessageRepository
) : ViewModel() {
fun sendMessage(message: Message) =
CoroutineScope(Dispatchers.IO).launch {
message.localId = messageRepository.insert(message)
}
fun editMessage(message: Message) =
CoroutineScope(Dispatchers.IO).launch {
messageRepository.update(message)
}
fun deleteMessage(message: Message) {
CoroutineScope(Dispatchers.IO).launch {
messageRepository.delete(message)
}
}
}
and this is my Composable function to show data
#Composable
fun Messaging(
navController: NavController,
to_user_id: String,
messagesViewModel: MessagesViewModel,
currentUserId: Long,
) {
val messages: List<Message> by messagesViewModel.getConversationMessages(to_user_id.toLong())
.observeAsState(
listOf()
)
Column {
MessagingHeader(
navController,
profileViewModel,
to_user_id.toLong(),
selection
)
Column(Modifier.weight(1f)) {
Column(
Modifier.verticalScroll(
state = rememberScrollState(),
reverseScrolling = true
)
) {
messages.forEach { message ->
Message(
currentUserId,
message
)
}
}
}
}
UPDATE for getConversationMessages func:
fun getConversationMessages(targetUserId: Long) =
messageDAO.getMessagesByTargetUserId(targetUserId)
and this getMessagesByTargetUserId func for my MessageDAO
#Query("SELECT * FROM messages WHERE receiver = :targetUserId OR sender = :targetUserId ORDER BY createdAt")
abstract fun getMessagesByTargetUserId(targetUserId: Long): LiveData<List<Message>>
It sounds like the issue you're experiencing is that the data in your Composable function is not being updated when you call the "update" method in your ViewModel. The reason for this is that the LiveData returned by the "getConversationMessages" function in your repository is not being updated when you call "update" in your ViewModel.
One solution to this issue is to use the "postValue" method instead of "setValue" when updating the value of the LiveData object in your repository. When you call "postValue", it will trigger the observer in your Composable function and update the UI with the latest data.
Another solution is to use the Transformations.map method in your Messaging function to convert the LiveData returned by the getConversationMessages to another LiveData type, and then observe it, this way when the data change inside the map the observer will be triggered.
For example:
val conversationLiveData = messagesViewModel.getConversationMessages(to_user_id.toLong())
val messages: List<Message> by Transformations.map(conversationLiveData){it}.observeAsState(listOf())
It's also possible that the issue is caused by a problem with the Room database's ability to notify the observer when the data is updated. One way to check this is to add a log statement in the "update" method in your ViewModel to ensure that it is being called when you expect it to be.
It's also possible that the issue is related to the use of CoroutineScope with Dispatchers.IO in your ViewModel. In this case it might be useful to try calling the update method with Dispatchers.Main instead.
In summary, the issue you're experiencing is likely related to the LiveData object not being updated when the update method is called. There are several possible solutions to this issue, including using the "postValue" method instead of "setValue" when updating the value of the LiveData object, using Transformations.map, making sure the update method is being called when it should be, or trying to call the update method with Dispatchers.Main instead of Dispatchers.IO in your ViewModel.
I'm trying to aggregate (per key) a streaming data source in Apache Beam (via Scio) using a stateful DoFn (using #ProcessElement with #StateId ValueState elements). I thought this would be most appropriate for the problem I'm trying to solve. The requirements are:
for a given key, records are aggregated (essentially summed) across all time - I don't care about previously computed aggregates, just the most recent
keys may be evicted from the state (state.clear()) based on certain conditions that I control
Every 5 minutes, regardless if any new keys were seen, all keys that haven't been evicted from the state should be outputted
Given that this is a streaming pipeline and will be running indefinitely, using a combinePerKey over a global window with accumulating fired panes seems like it will continue to increase its memory footprint and the amount of data it needs to run over time, so I'd like to avoid it. Additionally, when testing this out, (maybe as expected) it simply appends the newly computed aggregates to the output along with the historical input, rather than using the latest value for each key.
My thought was that using a StatefulDoFn would simply allow me to output all of the global state up until now(), but it seems this isn't a trivial solution. I've seen hintings at using timers to artificially execute callbacks for this, as well as potentially using a slowly growing side input map (How to solve Duplicate values exception when I create PCollectionView<Map<String,String>>) and somehow flushing this, but this would essentially require iterating over all values in the map rather than joining on it.
I feel like I might be overlooking something simple to get this working. I'm relatively new to many concepts of windowing and timers in Beam, looking for any advice on how to solve this. Thanks!
You are right that Stateful DoFn should help you here. This is a basic sketch of what you can do. Note that this only outputs the sum without the key. It may not be exactly what you want, but it should help you move forward.
class CombiningEmittingFn extends DoFn<KV<Integer, Integer>, Integer> {
#TimerId("emitter")
private final TimerSpec emitterSpec = TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
#StateId("done")
private final StateSpec<ValueState<Boolean>> doneState = StateSpecs.value();
#StateId("agg")
private final StateSpec<CombiningState<Integer, int[], Integer>>
aggSpec = StateSpecs.combining(
Sum.ofIntegers().getAccumulatorCoder(null, VarIntCoder.of()), Sum.ofIntegers());
#ProcessElement
public void processElement(ProcessContext c,
#StateId("agg") CombiningState<Integer, int[], Integer> aggState,
#StateId("done") ValueState<Boolean> doneState,
#TimerId("emitter") Timer emitterTimer) throws Exception {
if (SOME CONDITION) {
countValueState.clear();
doneState.write(true);
} else {
countValueState.addAccum(c.element().getValue());
emitterTimer.align(Duration.standardMinutes(5)).setRelative();
}
}
}
#OnTimer("emitter")
public void onEmit(
OnTimerContext context,
#StateId("agg") CombiningState<Integer, int[], Integer> aggState,
#StateId("done") ValueState<Boolean> doneState,
#TimerId("emitter") Timer emitterTimer) {
Boolean isDone = doneState.read();
if (isDone != null && isDone) {
return;
} else {
context.output(aggState.getAccum());
// Set the timer to emit again
emitterTimer.align(Duration.standardMinutes(5)).setRelative();
}
}
}
}
Happy to iterate with you on something that'll work.
#Pablo was indeed correct that a StatefulDoFn and timers are useful in this scenario. Here is the with code I was able to get working.
Stateful Do Fn
// DomainState is a custom case class I'm using
type DoFnT = DoFn[KV[String, DomainState], KV[String, DomainState]]
class StatefulDoFn extends DoFnT {
#StateId("key")
private val keySpec = StateSpecs.value[String]()
#StateId("domainState")
private val domainStateSpec = StateSpecs.value[DomainState]()
#TimerId("loopingTimer")
private val loopingTimer: TimerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME)
#ProcessElement
def process(
context: DoFnT#ProcessContext,
#StateId("key") stateKey: ValueState[String],
#StateId("domainState") stateValue: ValueState[DomainState],
#TimerId("loopingTimer") loopingTimer: Timer): Unit = {
... logic to create key/value from potentially null values
if (keepState(value)) {
loopingTimer.align(Duration.standardMinutes(5)).setRelative()
stateKey.write(key)
stateValue.write(value)
if (flushState(value)) {
context.output(KV.of(key, value))
}
} else {
stateValue.clear()
}
}
#OnTimer("loopingTimer")
def onLoopingTimer(
context: DoFnT#OnTimerContext,
#StateId("key") stateKey: ValueState[String],
#StateId("domainState") stateValue: ValueState[DomainState],
#TimerId("loopingTimer") loopingTimer: Timer): Unit = {
... logic to create key/value checking for nulls
if (keepState(value)) {
loopingTimer.align(Duration.standardMinutes(5)).setRelative()
if (flushState(value)) {
context.output(KV.of(key, value))
}
}
}
}
With pipeline
sc
.pubsubSubscription(...)
.keyBy(...)
.withGlobalWindow()
.applyPerKeyDoFn(new StatefulDoFn())
.withFixedWindows(
duration = Duration.standardMinutes(5),
options = WindowOptions(
accumulationMode = DISCARDING_FIRED_PANES,
trigger = AfterWatermark.pastEndOfWindow(),
allowedLateness = Duration.ZERO,
// Only take the latest per key during a window
timestampCombiner = TimestampCombiner.END_OF_WINDOW
))
.reduceByKey(mostRecentEvent())
.saveAsCustomOutput(TextIO.write()...)
I am implementing a Heartbeat for a WebFlux SSE endpoint. To avoid a timeout in the client, I want to make sure that an element is emitted at least every, say, 10 seconds.
I came up with the following solution that emits a heartbeat element every 10 seconds regardless of whether a real element has been emitted or not:
originalFlux.mergeWith(Flux.interval(Duration.ofSeconds(10), Duration.ofSeconds(10)).map(ignored -> "heartbeat")
This is probably good enough for my use case but still I wonder if it is possible to emit the heartbeat only if no real element has been emitted in the last 10 seconds. I played around with the timeout operator which implements exactly the timing behavior I am looking for, but that emits an error and cancels the originalFlux instead of just emitting an extra element.
The following code using timeout passes my test but looks too complicated and as far as I understand could lose elements from the originalFlux if they are emitted between cancelling and re-subscribing to it:
ConnectableFlux<String> sharedOriginalFlux = originalFlux.publish();
CompletableFuture<Disposable> eventualSubscription = new CompletableFuture<>();
return addHeartbeat(sharedOriginalFlux)
.doOnSubscribe(ignored -> eventualSubscription.complete(sharedOriginalFlux.connect()))
.doFinally(ignored -> eventualSubscription.thenAccept(Disposable::dispose))
private Flux<String> addHeartbeat(Flux<String> sharedOriginalFlux) {
return sharedOriginalFlux.timeout(
Duration.ofSeconds(10),
Flux.mergeSequential(
Mono.just("heartbeat"),
Flux.defer(() -> addHeartbeat(sharedOriginalFlux))));
}
Is there a simple and safe way to do this?
It's not necessarily simpler, but another option could be to create a separate processor that can wrap the original Flux to provide a heartbeat (which shouldn't miss any elements):
public class HeartbeatProcessor<T> {
private final FluxProcessor<T, T> processor;
private final FluxSink<T> sink;
private final T heartbeatValue;
private final Duration heartbeatPeriod;
private Disposable d;
public HeartbeatProcessor(Flux<T> orig, T heartbeatValue, Duration heartbeatPeriod) {
this.heartbeatValue = heartbeatValue;
this.heartbeatPeriod = heartbeatPeriod;
this.processor = DirectProcessor.<T>create().serialize();
this.sink = processor.sink();
this.d = Mono.just(heartbeatValue).delayElement(heartbeatPeriod).subscribe(this::emit);
orig.subscribe(this::emit);
}
private void emit(T val) {
sink.next(val);
d.dispose();
this.d = Mono.just(heartbeatValue).delayElement(heartbeatPeriod).subscribe(this::emit);
}
public Flux<T> getFlux() {
return processor;
}
}
You could then call it as follows:
new HeartbeatProcessor<>(elements, "heartbeat", Duration.ofSeconds(10))
.getFlux()
.subscribe(System.out::println);
Flux.switchMap is a good candidate for this job: It switches to a new Publisher (and cancels the previous one) whenever the original Flux emits an item. In your case, the new Publisher is your heartbeat Flux.interval, prepended with the original item T:
public static Flux<String> addHeartbeat(Flux<String> originalFlux) {
return originalFlux
.startWith("heartbeat")
.materialize()
.switchMap(signal -> switch (signal.getType()) {
case ON_NEXT -> Flux.interval(Duration.ofSeconds(10))
.map(ignored -> "heartbeat")
.startWith(signal.get());
case ON_COMPLETE -> Mono.empty();
case ON_ERROR -> Mono.error(signal.getThrowable());
default -> Mono.error(new IllegalStateException());
});
}
Flux.switchMap is almost fit for the job, but it differs on two points from your desired solution:
It will only emit elements once the first element is received.
This means you have no heartbeat before the first item. This is solved by adding Flux.startWith("heartbeat"), which will emit "heartbeat" immediately on subscription, which then is processed by the switchMap into a heartbeat every 10 seconds.
The Publisher of the last element is never cancelled.
Since every generated Publisher is a Flux.interval that never completes, the onComplete signal will never reach the user. This is solved by transforming the onComplete signal into an emitted Signal item using Flux.materialize(), then map the onComplete Signal into an empty Publisher just to cancel the previous Publisher. This also creates onNext and onError Signals, which we have to handle each:
a. Signal.ON_NEXT can be processed as usual, retrieving the original with Signal.get()
b. Signal.ON_COMPLETE is mapped to an empty Mono that immediately completes.
c. Signal.ON_ERROR should relay the error downstream using Mono.error(Throwable).
d. The Signal enum contains more values, but they are not produced by Flux.materialize().
Here is the test to test this solution:
#Test
public void shouldAddHeartbeat() {
Flux<String> originalFlux = Flux.just(25, 15, 7, 5)
.concatMap(delay -> Mono.delay(Duration.ofSeconds(delay)).thenReturn(delay + " seconds delay"));
Flux<String> withHeartbeat = addHeartbeat(originalFlux);
StepVerifier.withVirtualTime(() -> withHeartbeat)
.expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(5)).expectNext("25 seconds delay")
.thenAwait(Duration.ofSeconds(10)).expectNext("heartbeat")
.thenAwait(Duration.ofSeconds(5)).expectNext("15 seconds delay")
.thenAwait(Duration.ofSeconds(7)).expectNext("7 seconds delay")
.thenAwait(Duration.ofSeconds(5)).expectNext("5 seconds delay")
.verifyComplete();
}
I have some initial state in my application and a few of policies that decorates this state with reactively fetched data (each of policy's Mono returns new instance of state with additional data). Eventually I get fully decorated state.
It basically looks like this:
public interface Policy {
Mono<State> apply(State currentState);
}
Usage for fixed number of policies would look like that:
Flux.just(baseState)
.flatMap(firstPolicy::apply)
.flatMap(secondPolicy::apply)
...
.subscribe();
It basically means that entry state for a Mono is result of accumulation of initial state and each of that Mono predecessors.
For my case policies number is not fixed and it comes from another layer of the application as a collection of objects that implements Policy interface.
Is there any way to achieve similar result as in the given code (with 2 flatMap), but for unknown number of policies? I have tried with Flux's reduce method, but it works only if policy returns value, not a Mono.
This seems difficult because you're streaming your baseState, then trying to do an arbitrary number of flatMap() calls on that. There's nothing inherently wrong with using a loop to achieve this, but I like to avoid that unless absolutely necessary, as it breaks the natural reactive flow of the code.
If you instead iterate and reduce the policies into a single policy, then the flatMap() call becomes trivial:
Flux.fromIterable(policies)
.reduce((p1,p2) -> s -> p1.apply(s).flatMap(p2::apply))
.flatMap(p -> p.apply(baseState))
.subscribe();
If you're able to edit your Policy interface, I'd strongly suggest adding a static combine() method to reference in your reduce() call to make that more readable:
interface Policy {
Mono<State> apply(State currentState);
public static Policy combine(Policy p1, Policy p2) {
return s -> p1.apply(s).flatMap(p2::apply);
}
}
The Flux then becomes much more descriptive and less verbose:
Flux.fromIterable(policies)
.reduce(Policy::combine)
.flatMap(p -> p.apply(baseState))
.subscribe();
As a complete demonstration, swapping out your State for a String to keep it shorter:
interface Policy {
Mono<String> apply(String currentState);
public static Policy combine(Policy p1, Policy p2) {
return s -> p1.apply(s).flatMap(p2::apply);
}
}
public static void main(String[] args) {
List<Policy> policies = new ArrayList<>();
policies.add(x -> Mono.just("blah " + x));
policies.add(x -> Mono.just("foo " + x));
String baseState = "bar";
Flux.fromIterable(policies)
.reduce(Policy::combine)
.flatMap(p -> p.apply(baseState))
.subscribe(System.out::println); //Prints "foo blah bar"
}
If I understand the problem correctly, then the most simple solution is to use a regular for loop:
Flux<State> flux = Flux.just(baseState);
for (Policy policy : policies)
{
flux = flux.flatMap(policy::apply);
}
flux.subscribe();
Also, note that if you have just a single baseSate you can use Mono instead of Flux.
UPDATE:
If you are concerned about breaking the flow, you can extract the for loop into a method and apply it via transform operator:
Flux.just(baseState)
.transform(this::applyPolicies)
.subscribe();
private Publisher<State> applyPolicies(Flux<State> originalFlux)
{
Flux<State> newFlux = originalFlux;
for (Policy policy : policies)
{
newFlux = newFlux.flatMap(policy::apply);
}
return newFlux;
}
In C# I am using the asynchronous versions of TcpListener/TcpClient, and I am chaining these via the callback method so that another Accept/Read is posted when the callback completes. Here is an example (untested):
public void Start()
{
TcpListener listener = new TcpListener(IPAddress.Any, 3000);
listener.Start();
PostAccept(listener);
}
private void PostAccept(TcpListener listener)
{
listener.BeginAcceptTcpClient(AcceptCallback, listener);
}
private void AcceptCallback(IAsyncResult ar)
{
var listener = ar.AsyncState as TcpListener;
if (listener == null)
{
return;
}
// get the TcpClient and begin a read chain in a similar manner
PostAccept(listener);
}
My question is how do I model something similar in F#? Would I use the async keyword? Is the code after BeginAcceptTcpClient, BeginRead, etc.. essentially the code that would be executed in the callback function? For example, is this correct?
let accept(listener:TcpListener) = async {
let! client = listener.AsyncAcceptTcpClient()
// do something with the TcpClient such as beginning a Read chain
accept(listener)
return()
}
The above code doesn't work because accept isn't defined, and marking it recursive technically isn't true as it isn't a recursive method?
#kvb's answer is correct and idiomatic.
Just wanted to also point out you can use (gasp) a loop:
let accept(t:TcpListener) =
let completed = ref false
async {
while not(!completed) do
let! client = t.AsyncAcceptTcpClient()
if client <> null then
Blah(client)
else
completed := true
}
(that was typing code into my browser, hopefully it compiles). Which is nice, since you certainly can't use a loop in the C# code (that has to span multiple methods with the begin/end callbacks).
I'm not sure what you mean by "it isn't a recusive method"; if you refer to a function from within its own definition, then it is a recursive function. I don't have much experience with the Sockets classes, but perhaps something along these lines is what you're looking for?
let rec accept(t : TcpListener) =
async {
let! client = t.AsyncAcceptTcpClient()
// do some stuff with client here
do! accept(t)
}