create a circular stream in Dart - stream

I try to write a chat application to chat with a computer. The user can write a message and gets a response of the computer. A chat history might look like this:
user: Hi
computer: Hello
user: What's your name?
computer: Bot
...
My circular stream based design is inspired by the ideas of Cycle.js. I've got a stream of user messages, which get transformed to a stream of computer messages, which are in turn the input of the user message stream:
|----> user message stream ---->|
| |
transform transform
| |
|<-- computer message stream <--|
This code already works:
import 'dart:io';
import 'dart:async';
void main() {
cycle(computer, user);
}
typedef Stream<T> Transform<T>(Stream<T> input);
void cycle(Transform aToB, Transform bToA) {
var aProxy = new StreamController.broadcast();
var b = aToB(aProxy.stream);
var a = bToA(b);
aProxy.add('start'); // start with user
aProxy.addStream(a);
}
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
return computerMessages.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
Stream<String> computer(Stream<String> userMessages) {
var messages = <String, String>{
"Hi": "Hello",
"What's your name?": "Bot"
};
return userMessages.map((m) => messages.containsKey(m) ? messages[m] : 'What?');
}
There is only one problem. You need a start value to get a circular stream running. Therefore, I put this line in my function cycle:
aProxy.add('start'); // start with user
Actually, this logic belongs into my function user, since cycle shouldn't know the initial value(s). Moreover, I don't like to print the initial value. It should only trigger the user input stream. Thus, I changed cycle and user:
void cycle(Transform aToB, Transform bToA) {
var aProxy = new StreamController.broadcast();
var b = aToB(aProxy.stream);
var a = bToA(b);
aProxy.addStream(a);
}
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
var requestInput = new StreamController<String>.broadcast();
requestInput.add('start'); // start with user
requestInput.addStream(computerMessages); // continue on computer response
return requestInput.stream.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
But with this change my application terminates immediately with no message in stdout. What's wrong?

I found a solution. Create a normal instead of a broadcast StreamController in user:
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
var requestInput = new StreamController<String>();
requestInput.add('start'); // start with user
requestInput.addStream(computerMessages); // continue on computer response
return requestInput.stream.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
Even though I found a solution, I sadly can not really explain what's the problem with a broadcast StreamController. null is successfully added by requestInput.add(null); but strangely neither requestInput.addStream(computerMessages); completes nor events of computerMessages are added to requestInput. Thus, requestInput is closed and the application terminates. I would appreciate if someone could provide further explanation.

Related

Why aren't my messages sending using Twilio Messaging Service and Twilio Functions? [duplicate]

fine people of Stack Overflow.
I'm trying to solve a problem I'm having involving twilio functions, messaging services, and databases.
What I'm attempting to do is send a message to all members of a database at once.
My code is a mess, as Javascript isn't my native language and I'm rather new to twilio.
The problem I believe I'm having is with the async/await feature of javascript.
Here is my code so far:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
allItems = res.items;
// continue fetching until last is not seen
while (res.last){
res = await db.fetch({}, {last: res.last});
allItems = allItems.concat(res.items);
}
}
// Function to get total number of contacts.
async function ReturnNumberOfContacts(allItems) {
number_of_contacts = allItems.length;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message) {
allItems.forEach(contact => {
let users_name = contact.name
client.messages
.create({
body: `Hey ${users_name}! ${message}`,
messagingServiceSid: messaging_service,
to: contact.key
})
});
}
// Function to submit response to broadcaster.
async function SuccessResponse(user_name, number_of_contacts) {
responseObject = {
"actions": [
{
"say": `${user_name}, your broadcast has successfully sent to ${number_of_contacts} contacts.`
},
{
"listen": true
}
]
}
}
// Main Function
exports.handler = async function(context, event, callback) {
// Placeholder for number of contacts
let number_of_contacts;
// Place holder for object from database of all contacts
let allItems;
// Placeholder for users message
let message;
// Placeholder for response to user
let responseObject;
//Twilio and Deta, Etc Const
const client = require('twilio')(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messaging_service = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
await FetchDB(db, allItems).then(ReturnNumberOfContacts(allItems));
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message = memory.twilio.collected_data.broadcast_message.answers.message_input.answer;
}
// Check if verified and set name.
const current_user = await db.get(from);
// Get the current users name or set a default value
let user_name = current_user.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
responseObject = {
"actions": [
{
"say": `${user_name}, you have canceled your request and no messages have been sent.`
},
{
"listen": false
}
]
}
// Return Callback and end task
callback(null, responseObject);
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(message, client, messaging_service)
.then(SuccessResponse(user_name, number_of_contacts))
return callback(null, responseObject);
}
// The user is not authorized so return this.
}
return callback(null, {
"actions": [
{
"say": "You are not authorized to broadcast."
},
{
"listen": false
}
]
})
};
So when the Fetch() function is triggered, I want the database to load a list of everyone and have twilio send them the desired message saved in the message variable. I have the code working so that I can read from the database and get the proper values, and send a single text message with the desired message, but the problem I'm having now is integrating it all together.
Thanks if anyone can point me in the right direction here.
Again, I'm new to javascript and more specifically asynchronous programming.
Twilio developer evangelist here.
The issue from the error says that allItems is undefined when you call ReturnNumberOfContacts.
I think the issue comes from trying to use allItems as a sort of global variable, same for number_of_contacts. It would be better for FetchDB to resolve with the list of items and ReturnNumberOfContacts to resolve with the number of the items.
You also have some arguments missing when you call SendMessages in your function. I've updated it to the point that I think it will work:
// Boiler Plate Deta Code
const { Deta } = require("deta");
// Function to access database and get object of conta
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
// Function to send message to contact in database.
async function SendMessages(allItems, message, client, messagingService) {
return Promise.all(
allItems.map((contact) => {
let usersName = contact.name;
return client.messages.create({
body: `Hey ${usersName}! ${message}`,
messagingServiceSid: messagingService,
to: contact.key,
});
})
);
}
// Main Function
exports.handler = async function (context, event, callback) {
// Placeholder for users message
let message;
//Twilio and Deta, Etc Const
const client = require("twilio")(context.ACCOUNT_SID, context.AUTH_TOKEN);
const deta = Deta(context.DETA_PROJECT_KEY);
const db = deta.Base("users2");
const messagingService = context.MESSAGING_SERVICE;
// From Phone Number
const from = event.UserIdentifier;
// Parse memory
const memory = JSON.parse(event.Memory);
// Fetch all items from database and return total number of contacts.
// Update relavent variables
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
// Figure out if message came from short circuit broadcast or normal
if (memory.triggered) {
message = memory.message;
} else {
message =
memory.twilio.collected_data.broadcast_message.answers.message_input
.answer;
}
// Check if verified and set name.
const currentUser = await db.get(from);
// Get the current users name or set a default value
let userName = currentUser.name || "friend";
// Determine if user is an authorized broadcaster
if (from === context.BROADCAST_NUMBER) {
// Decide if the sending of a message should be cancelled.
if (message.toLowerCase() === "c" || message.toLowerCase() === "cancel") {
// Return Callback and end task
callback(null, {
actions: [
{
say: `${userName}, you have canceled your request and no messages have been sent.`,
},
{
listen: false,
},
],
});
}
// Move forward with sending a message.
else {
// Send message to users in database and send success message to broadcaster.
await SendMessages(allItems, message, client, messagingService);
return callback(null, {
actions: [
{
say: `${userName}, your broadcast has successfully sent to ${numberOfContacts} contacts.`,
},
{
listen: true,
},
],
});
}
// The user is not authorized so return this.
}
return callback(null, {
actions: [
{
say: "You are not authorized to broadcast.",
},
{
listen: false,
},
],
});
};
What I did here was change FetchDB to only take the db as an argument, then to create a local allItems variable that collects all the contacts and then returns them.
async function FetchDB(db) {
let res = await db.fetch();
let allItems = res.items;
// continue fetching until last is not seen
while (res.last) {
res = await db.fetch({}, { last: res.last });
allItems = allItems.concat(res.items);
}
return allItems;
}
This is then called in the main body of the function to assign a local variable. I also replaced the ReturnNumberOfContacts function with a simple assignment.
const allItems = await FetchDB(db);
const numberOfContacts = allItems.length;
One thing you may want to consider is how many contacts you are trying to send messages to during this function. There are a few limits you need to be aware of.
Firstly, Function execution time is limited to 10 seconds so you need to make sure you can load and send all your messages within that amount of time if you want to use a Twilio Function for this.
Also, there are limits for the number of concurrent connections you can make to the Twilio API. That limit used to be 100 connections per account, but it may vary these days. When sending asynchronous API requests as you do in JavaScript, the platform will attempt to create as many connections to the API that it can in order to trigger all the requests asynchronously. If you have more than 100 contacts you are trying to send messages to here, that will quickly exhaust your available concurrent connections and you will receive 429 errors. You may choose to use a queue, like p-queue, to ensure your concurrent connections never get too high. The issue in this case is that it will then take longer to process the queue which brings me back to the original limit of 10 seconds of function execution.
So, I think the above code may work in theory now, but using it in practice may have other issues that you will need to consider.

Dart - Get the last or the first value of a stream

I have a stream and I need to use the last value of this stream, and if there is no value emitted by this stream I need to wait for the fist value. I only want to use this value once. What is the correct way to do it?
Sounds like you want the most recent event emitted by a stream (which is presumably a broadcast stream, because otherwise there is no events until you listen), or, if there has been no events before, you want the next event instead.
For a plain Dart Stream, that's impossible. It doesn't remember previous events. You need to have listened to that stream previously in order to know what the most recent event was (but if you do that, it doesn't have to be a broadcast stream anyway).
You can build your own memorizing stream wrapper fairly easily (but as always with asynchronous programming, you need to be careful about race conditions)
// Copyright 2021 Google LLC.
// SPDX-License-Identifier: Apache-2.0
import "dart:async";
/// Listens to [source] to returned stream.
///
/// Each listener on the returned stream receives the most recent
/// event sent on [source] followed by all further events of [source]
/// until they stop listening.
/// If there has been no events on [source] yet, only the further events
/// are forwarded.
Stream<T> mostRecentStream<T>(Stream<T> source) {
var isDone = false;
var hasEvent = false;
T? mostRecentEvent;
List<MultiStreamController>? pendingListeners;
var listeners = <MultiStreamController>[];
void forEachListener(void Function(MultiStreamController) action) {
var active = 0;
var originalLength = listeners.length;
for (var i = 0; i < listeners.length; i++) {
var controller = listeners[i];
if (controller.hasListener) {
listeners[active++] = controller;
if (i < originalLength) action(controller);
}
}
listeners.length = active;
}
source.listen((event) {
mostRecentEvent = event;
hasEvent = true;
forEachListener((controller) {
controller.addSync(event);
});
}, onError: (e, s) {
forEachListener((controller) {
controller.addErrorSync(e, s);
});
}, onDone: () {
isDone = true;
for (var controller in listeners) {
controller.close();
}
listeners.clear();
});
return Stream<T>.multi((controller) {
if (hasEvent) controller.add(mostRecentEvent as T);
if (isDone) {
controller.close();
} else {
listeners.add(controller);
}
});
}
With that, you can simply do var recentStream = mostRecentStream(yourStream) and then later do recentStream.first to get either the most recent event or, if there is none, the next event (if there is one, you get an error if the stream is completely empty).

How to read part of a chunked stream

In Dart, stdin is a Stream<List<int>>. Bytes come in in chunks. What I want is a function that reads from stdin until I get some character (say '\0'), and then returns, so that future readers of stdin get the data after the '\0'.
Unfortunately because of the chunking, the '\0' byte might be in the middle of a chunk, so I kind of want to read a chunk, remove part of it, and then push it back to the start of the stream. But there isn't any way to do this.
Another option would be readByteSync() but reading bytes one at a time is going to be slow and this is in a GUI program so I can't use sync methods.
I think actually because a Stream<> can only ever be listened to once - even if a previous listener cancels its subscription - the only way is to have something permanently filtering the stream's events until the end of time. So you may as well just split the stream into two streams:
import 'dart:async';
class StdioPreambleSplitter {
StdioPreambleSplitter(this._input) {
var preambleFinished = false;
_input.listen((chunk) {
if (preambleFinished) {
_dataStream.add(chunk);
} else {
final nullByte = chunk.indexOf(0);
if (nullByte == -1) {
_dataStream.add(chunk);
} else {
preambleFinished = true;
_preambleStream.add(chunk.sublist(0, nullByte));
_dataStream.add(chunk.sublist(nullByte));
}
}
});
}
Stream<List<int>> preambleStream() {
return _preambleStream.stream;
}
Stream<List<int>> dataStream() {
return _dataStream.stream;
}
final Stream<List<int>> _input;
final StreamController<List<int>> _preambleStream = new StreamController();
final StreamController<List<int>> _dataStream = new StreamController();
}
Hopefully it doesn't add too much overhead.

When do Stream start publishing values to listeners?

After reading a bunch of documentation about Streams and StreamControllers in dart I tried to build a little example and was surprised of the results. All documentation I have read states that a stream starts emiting data as soon as a listener is registered. But this doesn't show any printed data:
class Order
{
String type;
Order(this.type);
}
class Pizza
{
}
void main()
{
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
})
.map((orderType) {
if(orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza)
{
print("We did a pizza");
},
onError: (error)
{
print(error);
});
sus.cancel();
sc.sink.add(new Order("pizza2"));
}
I was expecting this output:
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
When creating streams and adding data is all "sinked" data scheduled to be emited on the next application step?
Cheers.
You are right in that the documentation states that you listen on a stream to make it start generating events. However, streams are asynchronous so when you call the listen() method you are registering to receive events from the stream at some point in the future. Dart will then continue to run the remainder of your main function. Immediately after calling listen() you call cancel() to cancel the subscription which is why nothing is being printed.
If you remove or comment out the cancel and run it again you will see the expected output.
A slightly modified version of your code will hopefully highlight the run of events:
class Order {
String type;
Order(this.type);
}
class Pizza {}
void main() {
print("Main starts");
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
}).map((orderType) {
if (orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza) {
print("We did a pizza");
}, onError: (error) {
print(error);
});
// sus.cancel();
sc.sink.add(new Order("pizza2"));
print("Main ends");
}
Running this produces the output:
Main starts
Main ends
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
dude!, I don't know how to do that

async Future StreamSubscription Error

Could someone please explain what's wrong with the following code. I'm making two calls to the function fInputData. The first works ok, the second results in an error :
"unhandled exception"
"Bad state: Stream already has subscriber"
I need to write a test console program that inputs multiple parameters.
import "dart:async" as async;
import "dart:io";
void main() {
fInputData ("Enter Nr of Iterations : ")
.then((String sResult){
int iIters;
try {
iIters = int.parse(sResult);
if (iIters < 0) throw new Exception("Invalid");
} catch (oError) {
print ("Invalid entry");
exit(1);
}
print ("In Main : Iterations selected = ${iIters}");
fInputData("Continue Processing? (Y/N) : ") // this call bombs
.then((String sInput){
if (sInput != "y" && sInput != "Y")
exit(1);
fProcessData(iIters);
print ("Main Completed");
});
});
}
async.Future<String> fInputData(String sPrompt) {
async.Completer<String> oCompleter = new async.Completer();
stdout.write(sPrompt);
async.Stream<String> oStream = stdin.transform(new StringDecoder());
async.StreamSubscription oSub;
oSub = oStream.listen((String sInput) {
oCompleter.complete(sInput);
oSub.cancel();
});
return oCompleter.future;
}
void fProcessData(int iIters) {
print ("In fProcessData");
print ("iIters = ${iIters}");
for (int iPos = 1; iPos <= iIters; iPos++ ) {
if (iPos%100 == 0) print ("Processed = ${iPos}");
}
print ("In fProcessData - completed ${iIters}");
}
Some background reading:
Streams comes in two flavours: single or multiple (also known as
broadcast) subscriber. By default, our stream is a single-subscriber
stream. This means that if you try to listen to the stream more than
once, you will get an exception, and using any of the callback
functions or future properties counts as listening.
You can convert the single-subscriber stream into a broadcast stream
by using the asBroadcastStream() method.
So you've got two options - either re-use a single subscription object. i.e. call listen once, and keep the subscription object alive.
Or use a broadcast stream - note there are a number of differences between broadcast streams and single-subscriber streams, you'll need to read about those and make sure they suit your use-case.
Here's an example of reusing a subscriber to ask multiple questions:
import 'dart:async';
import 'dart:io';
main() {
var console = new Console();
var loop;
loop = () => ask(console).then((_) => loop());
loop();
}
Future ask(Console console) {
print('1 + 1 = ...');
return console.readLine().then((line) {
print(line.trim() == '2' ? 'Yup!' : 'Nope :(');
});
}
class Console {
StreamSubscription<String> _subs;
Console() {
var input = stdin
.transform(new StringDecoder())
.transform(new LineTransformer());
_subs = input.listen(null);
}
Future<String> readLine() {
var completer = new Completer<String>();
_subs.onData(completer.complete);
return completer.future;
}
}

Resources