How to cancel an event in reactive stream? - stream

I'm new to reactive programming and have difficulties using "everything can be a stream" mantra. I'm considering the following scenario - I have a stream of websocket events definied like this:
Rx.Observable.create((observer) => {
io.on('connect', function(socket){
socket.on("enroll", function(player) {
observer.next({
event: 'enroll',
player,
socket
});
});
socket.on('resign', function(player){
observer.next({
event: 'resign',
player,
socket
});
});
});
return {
dispose: io.close
};
});
Then I can do something like
enrollmentStream = events$
.filter(find({ event: "enroll" }))
.map(pick('player'));
And likewise
resignationStream = events$
.filter(find({ event: "resign" }))
.map(pick('player'));
I would like to gather enrolled players in a stream which would batch them in 4's but obviously this should be done only for users which are in enrollment stream but are not in resignationStream or at least the last event was enrollment. How do I do this?
Here is the marble diagram.
There are 5 players which enroll. Game starts when there are 4 players enrolled. Note that the second player (violet one) enrolls but then resigns so the game does not start with blue marble but with the next - yellow - cause only after that there are really 4 players ready.
Probably there should be some stream operation like "without"... is there?

I think in this scenario you could use combineLatest() and scan() operators and then make a list of unresigned players yourself:
const bufferedEnrollment = enrollmentStream.scan((acc, val) => { acc.push(val); return acc; }, []);
const bufferedResignation = enrollmentStream.scan((acc, val) => { acc.push(val); return acc; }, []);
Observable.combineLatest(bufferedEnrollment, bufferedResignation)
.map(values => {
const enrolled = values[0];
const resigned = values[1];
// remove resigned players from `enrolled` array
return enrolled;
})
.filter(players => players.length === 4)
.subscribe(...)
The scan() operator is used only to collect players into an array. If you for example wanted to be able to reset the array you could merge it with another Observable.
enrollmentStream
.merge(resetStream)
.scan((acc, val) => {
if (!val) {
return [];
}
acc.push(val);
return acc;
}, []);
(for obvious reasons I didn't test this code).

Related

Broadcasting to Members of a Database Using Twilio Messaging Services

fine people of Stack Overflow.
I'm trying to solve a problem I'm having involving twilio functions, messaging services, and databases.
What I'm attempting to do is send a message to all members of a database at once.
My code is a mess, as Javascript isn't my native language and I'm rather new to twilio.
The problem I believe I'm having is with the async/await feature of javascript.
Here is my code so far:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
allItems = res.items;
// continue fetching until last is not seen
while (res.last){
res = await db.fetch({}, {last: res.last});
allItems = allItems.concat(res.items);
}
}
// Function to get total number of contacts.
async function ReturnNumberOfContacts(allItems) {
number_of_contacts = allItems.length;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message) {
allItems.forEach(contact => {
let users_name = contact.name
client.messages
.create({
body: `Hey ${users_name}! ${message}`,
messagingServiceSid: messaging_service,
to: contact.key
})
});
}
// Function to submit response to broadcaster.
async function SuccessResponse(user_name, number_of_contacts) {
responseObject = {
"actions": [
{
"say": `${user_name}, your broadcast has successfully sent to ${number_of_contacts} contacts.`
},
{
"listen": true
}
]
}
}
// Main Function
exports.handler = async function(context, event, callback) {
// Placeholder for number of contacts
let number_of_contacts;
// Place holder for object from database of all contacts
let allItems;
// Placeholder for users message
let message;
// Placeholder for response to user
let responseObject;
//Twilio and Deta, Etc Const
const client = require('twilio')(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messaging_service = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
await FetchDB(db, allItems).then(ReturnNumberOfContacts(allItems));
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message = memory.twilio.collected_data.broadcast_message.answers.message_input.answer;
}
// Check if verified and set name.
const current_user = await db.get(from);
// Get the current users name or set a default value
let user_name = current_user.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
responseObject = {
"actions": [
{
"say": `${user_name}, you have canceled your request and no messages have been sent.`
},
{
"listen": false
}
]
}
// Return Callback and end task
callback(null, responseObject);
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(message, client, messaging_service)
.then(SuccessResponse(user_name, number_of_contacts))
return callback(null, responseObject);
}
// The user is not authorized so return this.
}
return callback(null, {
"actions": [
{
"say": "You are not authorized to broadcast."
},
{
"listen": false
}
]
})
};
So when the Fetch() function is triggered, I want the database to load a list of everyone and have twilio send them the desired message saved in the message variable. I have the code working so that I can read from the database and get the proper values, and send a single text message with the desired message, but the problem I'm having now is integrating it all together.
Thanks if anyone can point me in the right direction here.
Again, I'm new to javascript and more specifically asynchronous programming.
Twilio developer evangelist here.
The issue from the error says that allItems is undefined when you call ReturnNumberOfContacts.
I think the issue comes from trying to use allItems as a sort of global variable, same for number_of_contacts. It would be better for FetchDB to resolve with the list of items and ReturnNumberOfContacts to resolve with the number of the items.
You also have some arguments missing when you call SendMessages in your function. I've updated it to the point that I think it will work:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message, client, messagingService) {
return Promise.all(
allItems.map((contact) => {
let usersName = contact.name;
return client.messages.create({
body: `Hey ${usersName}! ${message}`,
messagingServiceSid: messagingService,
to: contact.key,
});
})
);
}
// Main Function
exports.handler = async function (context, event, callback) {
// Placeholder for users message
let message;
//Twilio and Deta, Etc Const
const client = require("twilio")(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messagingService = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message =
memory.twilio.collected_data.broadcast_message.answers.message_input
.answer;
}
// Check if verified and set name.
const currentUser = await db.get(from);
// Get the current users name or set a default value
let userName = currentUser.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
// Return Callback and end task
callback(null, {
actions: [
{
say: `${userName}, you have canceled your request and no messages have been sent.`,
},
{
listen: false,
},
],
});
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(allItems, message, client, messagingService);
return callback(null, {
actions: [
{
say: `${userName}, your broadcast has successfully sent to ${numberOfContacts} contacts.`,
},
{
listen: true,
},
],
});
}
// The user is not authorized so return this.
}
return callback(null, {
actions: [
{
say: "You are not authorized to broadcast.",
},
{
listen: false,
},
],
});
};
What I did here was change FetchDB to only take the db as an argument, then to create a local allItems variable that collects all the contacts and then returns them.
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
This is then called in the main body of the function to assign a local variable. I also replaced the ReturnNumberOfContacts function with a simple assignment.
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
One thing you may want to consider is how many contacts you are trying to send messages to during this function. There are a few limits you need to be aware of.
Firstly, Function execution time is limited to 10 seconds so you need to make sure you can load and send all your messages within that amount of time if you want to use a Twilio Function for this.
Also, there are limits for the number of concurrent connections you can make to the Twilio API. That limit used to be 100 connections per account, but it may vary these days. When sending asynchronous API requests as you do in JavaScript, the platform will attempt to create as many connections to the API that it can in order to trigger all the requests asynchronously. If you have more than 100 contacts you are trying to send messages to here, that will quickly exhaust your available concurrent connections and you will receive 429 errors. You may choose to use a queue, like p-queue, to ensure your concurrent connections never get too high. The issue in this case is that it will then take longer to process the queue which brings me back to the original limit of 10 seconds of function execution.
So, I think the above code may work in theory now, but using it in practice may have other issues that you will need to consider.

Dart - Get the last or the first value of a stream

I have a stream and I need to use the last value of this stream, and if there is no value emitted by this stream I need to wait for the fist value. I only want to use this value once. What is the correct way to do it?
Sounds like you want the most recent event emitted by a stream (which is presumably a broadcast stream, because otherwise there is no events until you listen), or, if there has been no events before, you want the next event instead.
For a plain Dart Stream, that's impossible. It doesn't remember previous events. You need to have listened to that stream previously in order to know what the most recent event was (but if you do that, it doesn't have to be a broadcast stream anyway).
You can build your own memorizing stream wrapper fairly easily (but as always with asynchronous programming, you need to be careful about race conditions)
// Copyright 2021 Google LLC.
// SPDX-License-Identifier: Apache-2.0
import "dart:async";
/// Listens to [source] to returned stream.
///
/// Each listener on the returned stream receives the most recent
/// event sent on [source] followed by all further events of [source]
/// until they stop listening.
/// If there has been no events on [source] yet, only the further events
/// are forwarded.
Stream<T> mostRecentStream<T>(Stream<T> source) {
var isDone = false;
var hasEvent = false;
T? mostRecentEvent;
List<MultiStreamController>? pendingListeners;
var listeners = <MultiStreamController>[];
void forEachListener(void Function(MultiStreamController) action) {
var active = 0;
var originalLength = listeners.length;
for (var i = 0; i < listeners.length; i++) {
var controller = listeners[i];
if (controller.hasListener) {
listeners[active++] = controller;
if (i < originalLength) action(controller);
}
}
listeners.length = active;
}
source.listen((event) {
mostRecentEvent = event;
hasEvent = true;
forEachListener((controller) {
controller.addSync(event);
});
}, onError: (e, s) {
forEachListener((controller) {
controller.addErrorSync(e, s);
});
}, onDone: () {
isDone = true;
for (var controller in listeners) {
controller.close();
}
listeners.clear();
});
return Stream<T>.multi((controller) {
if (hasEvent) controller.add(mostRecentEvent as T);
if (isDone) {
controller.close();
} else {
listeners.add(controller);
}
});
}
With that, you can simply do var recentStream = mostRecentStream(yourStream) and then later do recentStream.first to get either the most recent event or, if there is none, the next event (if there is one, you get an error if the stream is completely empty).

When do Stream start publishing values to listeners?

After reading a bunch of documentation about Streams and StreamControllers in dart I tried to build a little example and was surprised of the results. All documentation I have read states that a stream starts emiting data as soon as a listener is registered. But this doesn't show any printed data:
class Order
{
String type;
Order(this.type);
}
class Pizza
{
}
void main()
{
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
})
.map((orderType) {
if(orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza)
{
print("We did a pizza");
},
onError: (error)
{
print(error);
});
sus.cancel();
sc.sink.add(new Order("pizza2"));
}
I was expecting this output:
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
When creating streams and adding data is all "sinked" data scheduled to be emited on the next application step?
Cheers.
You are right in that the documentation states that you listen on a stream to make it start generating events. However, streams are asynchronous so when you call the listen() method you are registering to receive events from the stream at some point in the future. Dart will then continue to run the remainder of your main function. Immediately after calling listen() you call cancel() to cancel the subscription which is why nothing is being printed.
If you remove or comment out the cancel and run it again you will see the expected output.
A slightly modified version of your code will hopefully highlight the run of events:
class Order {
String type;
Order(this.type);
}
class Pizza {}
void main() {
print("Main starts");
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
}).map((orderType) {
if (orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza) {
print("We did a pizza");
}, onError: (error) {
print(error);
});
// sus.cancel();
sc.sink.add(new Order("pizza2"));
print("Main ends");
}
Running this produces the output:
Main starts
Main ends
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
dude!, I don't know how to do that

RxJS5 - How can I cache the last value of a aggregate stream, without using Subjects and whilst excluding those that have completed?

I want a pubsub system, with producers and consumers of streams, via
a dataplane layer but without Subjects.
Many producers can multicast to the same stream name (e.g,
'filters.add'), and multiple consumers can subscribe to the stream.
Producers register with a name and stream to create a 'stream of published streams'.
If a new producer registers with a particular stream name, the
streams flowing to the consumers are dynamically updated such that the data received is a merging of all currently active published streams with the name requested.
One producer stream completing should not cause the aggregate stream to complete and is instead excluded (with all consumers updated) when this occurs.
A consumer getting a stream after a producer registers should receive the last emitted value of that stream (if there has been one to date).
Here is the code I have so far and the tests I wish to satisfy. I can satisfy the base case but not caching, since:
completed streams are being excluded even though they may have emitted the most recent value;
.cache(1) after stream.publish().refCount() repeats entry 4 from the first Observable, instead of 8 from the second;
even with pluckActiveStreams commented out, which also doesn't work, I just can't get it to work with any of the approaches commented out...
My questions are:
how can I implement caching of the aggregated stream? None of the commented out final lines of relevantStreams$ accomplishes this.
I think I need to remove completed streams from the aggregate stream received, so that the aggregate stream itself is not (ever) completed. This is (potentially) accomplished with pluckActiveStreams, which is based on the solution given here but still give late-arriving consumers the last emitted value of the stream, even if it may have been from a now-completed stream?
My current implementation already successfully accomplishes this using observables of ReplaySubjects internally as a 'bridge' between producers and consumers. But this way:
all producer streams are automatically made hot;
the ReplaySubject needs a hack to ensure it never completes if any of the streams multicasting to it do (otherwise it shuts down the stream for all consumers, as explained here);
Once created upon registration of the first producer, a ReplaySubject representing a stream name for a particular group of producers can never again be made cold, disposed if there are no current producers for that name, or re-instantiated with different arguments, as any previously-registered consumers of that stream would then hold an incorrect reference. Consumers instead therefore need a reference to the reactively updated, declarative aggregate of active streams for a particular name.
Code
const publishStream$ = new Subject();
function publishStream(name, stream) {
// publish streams as connectables to ensure a single source of truth
// (i.e., don't duplicate server requests)
publishStream$.next({ name, stream: stream.publish().refCount() });
}
const availableStreams$ = publishStream$
.startWith([])
.scan((streams, stream) => {
console.log('\nadding stream:', stream.name)
// (this would eventually be state operations to add/remove streams
// so this doesn't just keep growing)
return streams.concat(stream)
})
.publish();
availableStreams$.connect()
function getStream(name) {
const relevantStreams$ = availableStreams$
.do(res => console.log(`\nstream recalculates here for ${name}`))
.map(streams => streams.filter(streamObj => streamObj.name === name))
.filter(streams => streams.length)
.map(streamObjs => streamObjs.map(streamObj => streamObj.stream))
.map(pluckActiveStreams)
.switchMap(streams => Observable.merge(...streams))
// add caching for late subscribers
// per https://github.com/ReactiveX/rxjs/issues/1541?
// (none of the approaches below work):
//.multicast(() => new ReplaySubject()).refCount()
// .publishReplay(1).refCount()
//.share().repeat(1).publish()
//.cache(1)
return relevantStreams$;
}
function pluckActiveStreams(streams) {
const activeStreams = new Subject();
const elements = [];
const subscriptions = [];
streams.forEach(stream => {
var include = true;
const subscription = stream.subscribe(x => {
console.log('value', x)
}, x => {
console.log('error:', x)
}, x => {
console.log('completed', x)
include = false;
const i = elements.indexOf(stream);
if (i > -1) {
elements.splice(i, 1);
activeStreams.next(elements.slice());
}
});
if (include) {
elements.push(stream);
subscriptions.push(subscription);
}
});
activeStreams.next(elements.slice());
const pluckedStreams = Observable.using(
() => new Subscription(() => subscriptions.forEach(x => x.unsubscribe())),
() => activeStreams
);
return pluckedStreams;
}
Tests
var acc1 = [];
var acc2 = [];
// all streams published get publish().refCount() internally
var obs1 = Observable.of(1, 2, 3, 4);
var obs2 = Observable.empty();
var obs3 = Observable.of(5, 6, 7, 8);
// consumer before publish - will receive values
var sub1 = dp.getStream('foo').subscribe(function (i) {
console.log('subscription receives: ', i)
acc1.push(i);
});
var pub1 = dp.publishStream('foo', obs1);
console.log('end of publishStream1')
var pub2 = dp.publishStream('foo', obs2);
console.log('end of publishStream2')
var pub3 = dp.publishStream('foo', obs3);
console.log('end of publishStream3')
// consumer after publish - should receive last cached value
// from active aggregated stream
var sub3 = dp.getStream('foo').subscribe(function (i) {
console.log("\ncached value received (I also don't fire :( ", i)
acc2.push(i);
});
var i = setTimeout(function () {
expect(acc1).to.deep.equal([1, 2, 3, 4, 5, 6, 7, 8]);
expect(acc2).to.deep.equal([8]);
done();
}, 10);

Dart: how to append a transformer to an existing stream?

I'm looking a for a way to programmatically add a transformer to an existing stream that's already being listen to.
Example:
Stream numbers = new Stream.fromIterable([0,1,2,3]);
numbers.listen((number) => print(number));
Now in response to some UI event, I'd like to modify this stream by adding a mapping transformer, as if I originally wrote:
numbers.where((number) => number % 2 == 0);
All existing listeners should from now own only receive even numbers, without interruption. How can this be done?
Instead of thinking about it like "how do I dynamically insert a transformer into a stream", one possible way is to think about it like "how do I dynamically control a transformer that I already injected".
Here's an example of using a StreamTransformer:
var onlySendEvenNumbers = false; // controlled by some UI event handler
var originalStream = makeStreamOfStuff();
originalStream = originalStream.transform(new StreamTransformer.fromHandlers(
handleData: (int value, EventSink<int> sink) {
if (onlySendEvenNumber) {
if (value.isEven) {
sink.add(value);
}
} else {
sink.add(value);
}
}));
originalStream.listen(print); // listen on events like normal
One way I can think of doing that is filtering the Stream with a function that calls another function:
var filter = (n) => true;
Stream numbers = new String.fromIterable([0, 1, 2, 3]).where((n) => filter(n));
Then, when you want to change the filtering:
filter = (n) => n % 2 == 0;
A concrete example:
import 'dart:async';
main() {
var filter = (n) => true;
Stream numbers = new Stream.periodic(new Duration(seconds: 1), (n) => n)
.where((n) => filter(n));
numbers.listen((n) => print(n));
new Future.delayed(new Duration(seconds: 4)).then((_) {
filter = (n) => n % 2 == 0;
});
}
This will print:
0
1
2
3
4
6
8
10
12
And so on, for even numbers only, after 4 seconds.
What about rxdart's combineLatest2 ?
It combine two streams, and emit each time when changed both streams.
You can use Switch class for switch on/off with conditions.
class XsBloc {
Api _api = Api();
BehaviorSubject<List<X>> _xs = BehaviorSubject();
BehaviorSubject<Switcher> _switcher =
BehaviorSubject<Switcher>.seeded(Switcher(false, []));
XsBloc() {
Observable.combineLatest2<List<X>, Switcher, List<X>>(
_api.xs(), _switcher, (xs, s) {
if (s.isOn == true) {
return xs.where((x) => s.conditions.contains(x.id)).toList();
} else {
return xs;
}
}).listen((x) => _xs.add(x));
}
Stream<List<X>> get xs => _xs;
ValueObservable<Switcher> get switcher =>
_switcher.stream;
Function(Switcher) get setSwitcher => _switcher.sink.add;
}
class Switcher {
final bool isOn;
final List<String> conditions;
Switcher(this.isOn, this.conditions);
}
var bloc = XsBloc();
bloc.setSwitcher(true, ['A', 'B']);
bloc.setSwitcher(false, []);
bloc.setSwitcher(true, []);

Resources