How can I await a stream (or any other event queue) multiple times?
I tried Stream.first & Stream.single, both doesn't work.
What I want to do:
//next is a fake member
Future<void> xxx() async {
final eventStream = foo();
await eventStream.next; // wait for first event
//do some work
await eventStream.next; // wait for second event
//do some other differnet work
await eventStream.next; // wait for 3rd event
// another differnet work
return;
}
equvalent to:
Future<void> xxx() async {
final eventStream = foo();
int i=0;
await for (final _ in eventStream){
if(i==0);//do some work
else if(i==1);//do some other differnet work
else if(i==2){;break;}//another differnet work
++i;
}
return;
}
Try StreamQueue from package:async.
var q = StreamQueue(eventStream);
var v1 = await q.next;
var v2 = await q.next;
var v3 = await q.next;
// other work.
await q.cancel(); // If not listened to completely yet.
I end up with a custom class
class CompleterQueue<T> {
final _buf = <Completer<T>>[];
var _iWrite = 0;
var _iRead = 0;
int get length => _buf.length;
Future<T> next() {
if (_iRead == _buf.length) {
_buf.add(Completer<T>());
}
final fut = _buf[_iRead].future;
_iRead += 1;
_cleanup();
return fut;
}
void add(T val) {
if (_iWrite == _buf.length) {
final prm = Completer<T>();
_buf.add(prm);
}
_buf[_iWrite].complete(val);
_iWrite += 1;
_cleanup();
}
void _cleanup() {
final minI = _iWrite < _iRead ? _iWrite : _iRead;
if (minI > 0) {
_buf.removeRange(0, minI);
_iWrite -= minI;
_iRead -= minI;
}
}
}
I inherited a script to manage a deploy of Salesforce code to multiple orgs in one go, to ensure all orgs are on the same version. The code is maintained in a Github respository and the final step is the update of the main branch, so the deploy therefore has to be successful for all orgs before it updates the main branch. Currently we have 32 orgs for which the deploys run simultaneously (with more to be added).
The final step after the code has deployed successfully is to check all the Salesforce to Salesforce connections and mappings, since all the orgs update a 'hub' org. It is in this step that I've started getting Puppeteer timeouts. Sometimes it completes, sometimes it fails. It seems to be getting worse in that I have to rerun it 2 or 3 times to get it pass without timing out. I'm not experienced in Node or Puppeteer or scripts like these so don't know how to stop this happening. I've tried increasing the timeout from the default 30000 to 90000 but even then it fails sometimes so that is not a solution, obviously.
Interestingly a few of us have also been having problems lately with Chrome being dreadfully slow and timing out just in the browser (we run on the latest version of Chrome) and I read that Puppeteer uses Chrome. I tried googling but haven't found anything that helps me hence posting this query here.
I would appreciate any help to sort this out because running it multiple times for each deploy is not a viable solution, especially with the length of time it takes to complete.
This is the function from where it sets the timeout.
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000); // added this but it still times out
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
And this is the called function where I believe the timeout happens:
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
This is the error thrown (after running 2hrs 40min!)It is close to the end of the process so has completed most of the org checks at this stage. It doesn't always fail in the same place or on the same org checks so the timeout is not related to a specific connection.
The full script is here:
import puppeteer, { Browser, Page } from "puppeteer";
import { flatten } from "lodash";
import yargs from "yargs";
import pAll from "p-all";
import {
loginAndGetConnections,
Connection
} from "../page-objects/sf2sf-home.page-object";
import {
getSubscribedObjects,
ConnectionObject
} from "../page-objects/sf2sf-connection.page-object";
import {
SubscribedField,
SubscribedFieldOption,
getSubscribedFields,
gotoObject,
selectMapping,
getPicklistValues,
save as saveSubscribedFields,
getPicklistMappingLinks,
savePicklistMapping
} from "../page-objects/sf2sf-subscribed-fields.page-object";
import { SClusterConfig } from "../s-cluster-config";
class Config {
configFile: string;
clusterConfigFile: string;
dryRun: boolean;
strict: boolean;
concurrency: number;
skipPicklists: boolean;
constructor() {
// eslint-disable-next-line #typescript-eslint/no-explicit-any
const argv: any = yargs
.scriptName("publish-connections")
.describe("config-file", "The file configuring the SF2SF sync.")
.alias("config-file", "c")
.default("config-file", "./sf2sf.config.json")
.describe("cluster-config-file", "The file configuring the SF2SF sync.")
.alias("cluster-config-file", "f")
.string("cluster-config-file")
.required("cluster-config-file")
.describe(
"dry-run",
"don't make any changes, just print what you're going to do."
)
.boolean("dry-run")
.default("dry-run", false)
.describe("strict", "Prevents associations from being unassigned")
.boolean("strict")
.default("strict", false)
.number("concurrency")
.default("concurrency", 10)
.describe("skip-picklists", "Skip assigning the picklists")
.boolean("skip-picklists")
.default("skip-picklists", false).argv;
this.configFile = argv["config-file"];
this.clusterConfigFile = argv["cluster-config-file"];
this.dryRun = argv["dry-run"];
this.strict = argv["strict"];
this.concurrency = argv["concurrency"];
this.skipPicklists = argv["skip-picklists"];
}
}
interface SubscribedFieldUpdate {
connection: Connection;
connectionObject: ConnectionObject;
connectionField: SubscribedField;
newValue?: SubscribedFieldOption;
}
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000);
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
(async (): Promise<void> => {
const argv = new Config();
const { clusterConfigFile, concurrency } = argv;
const clusterConfig = await SClusterConfig.fromPath(clusterConfigFile);
const browser = await puppeteer.launch({});
const connections = flatten(
await pAll(
clusterConfig.usernames.map(
(username) => (): Promise<Connection[]> =>
loginAndGetConnections(browser, username)
),
{ concurrency }
)
).filter((conn) => conn.isActive);
const differences: SubscribedFieldUpdate[] = [];
await pAll(
connections.map(
(connection) => (): Promise<void> =>
checkDifferencesForConnectionSafely(
argv,
browser,
connection,
differences
)
),
{ concurrency }
);
const result = differences.map(
({ connection, connectionObject, connectionField, newValue }) => ({
username: connection.username,
connection: connection.name,
object: connectionObject.name,
field: connectionField.name,
oldValue: (connectionField.value && connectionField.value.name) || "",
newValue: (newValue && newValue.name) || ""
})
);
console.log(JSON.stringify(result, null, " "));
await browser.close();
})();
I'm having trouble canceling a stream that is created using the Stream.periodic constructor. Below is my attempt at canceling the stream. However, I'm having a hard time extracting out the 'count' variable from the internal scope. Therefore, I can't cancel the subscription.
import 'dart:async';
void main() {
int count = 0;
final Stream newsStream = new Stream.periodic(Duration(seconds: 2), (_) {
return _;
});
StreamSubscription mySubscribedStream = newsStream.map((e) {
count = e;
print(count);
return 'stuff $e';
}).listen((e) {
print(e);
});
// count = 0 here because count is scoped inside mySubscribedStream
// How do I extract out 'count', so I can cancel the stream?
if (count > 5) {
mySubscribedStream.cancel();
mySubscribedStream = null;
}
}
I'd rather use take(5) instead of checking > 5 and then cancel
final Stream newsStream = new Stream.periodic(Duration(seconds: 2), (_) => count++);
newsStream.map((e) {
count = e;
print(count);
return 'stuff $e';
}).take(5).forEach((e) {
print(e);
});
EDIT: Problem wasn't related to Timer or HttpServer, it was dart.io sleep function pausing everything. It is clearly described in documentation, my bad.
//
I have weird problem with HttpClient working in server code. I call
client.getUrl(Uri.parse(url)).then((HttpClientRequest response) => response.close()).then(HttpBodyHandler.processResponse).then((HttpClientResponseBody body) {
print(body.response.statusCode);
from Timer object and it never reach print step.
It is almost copy and paste code from previous version, which wasn't called from Timer but from HttpRequest. Working code is in my question [here][1].
It fails on the long line, I suspect that it is a last Future it never reach (HttpClientResponseBody).
Timer object is created like this (just test code):
main() {
t = new Timer.periodic(new Duration(minutes: period), (Timer t) => hit());
}
void hit() {
if (new DateTime.now().hour == 17) {
print("syncing rock");
loadUrlBody(furl + filter).then((content) {
print("content loaded");
//edit:
okay, here is the source, it might be some trivial problem..which I can't figure out for two days :-D
import 'dart:async';
import 'dart:io';
import 'package:http_server/http_server.dart';
import 'package:slack/slack_io.dart' as slack;
Timer t;
bool check;
final period = 1;
final furl = "https://****.tpondemand.com";
final filter = "somefilter";
main() {
t = new Timer.periodic(new Duration(minutes: period), (Timer t) => hit());
}
void hit() {
if (new DateTime.now().hour == 17) {
print("syncing rock");
loadUrlBody(furl + filter).then((content) {
print("content loaded");
Map parsedMap = content.body;
handleMap(parsedMap);
});
sleep(new Duration(minutes: 60));
} else {
print("no time to rock " + new DateTime.now().toString());
sleep(new Duration(minutes: period * 10));
}
}
Future loadUrlBody(String url) {
final c = new Completer();
HttpClient client = new HttpClient();
client.addCredentials(Uri.parse("https://****.tpondemand.com/api"), "tprealm", new HttpClientBasicCredentials("user", "password"));
client.getUrl(Uri.parse(url)).then((HttpClientRequest response) => response.close()).then(HttpBodyHandler.processResponse).then((HttpClientResponseBody body) {
print(body.response.statusCode);
c.complete(body);
});
return c.future;
}
void send2Slack(String m) {
slack.Message message = new slack.Message()..text = m;
slack.token = 'token';
slack.team = 'team';
slack.send(message);
}
void handleMap(Map valueMap) {
final Duration lostInTime = new Duration(days: 30);
var sb = new StringBuffer();
sb.write('K o m p o s t \n');
for (var item in valueMap["Items"]) {
if (item['CreateDate'] == null) item['CreateDate'] = '/Date(1403167885000+0100)/';
if (item['ModifyDate'] == null) item['ModifyDate'] = '/Date(1403167885000+0100)/';
if (item['LastCommentDate'] == null) item['LastCommentDate'] = '/Date(1403167885000+0100)/';
DateTime moonLanding = new DateTime.fromMillisecondsSinceEpoch(int.parse(item['CreateDate'].substring(6, 19)));
DateTime modifyLanding = new DateTime.fromMillisecondsSinceEpoch(int.parse(item['ModifyDate'].substring(6, 19)));
DateTime commentLanding = new DateTime.fromMillisecondsSinceEpoch(int.parse(item['LastCommentDate'].substring(6, 19)));
DateTime lastChangeLanding = (modifyLanding.isBefore(commentLanding)) ? commentLanding : modifyLanding;
Duration difference = new DateTime.now().difference(lastChangeLanding);
if (moonLanding.add(lostInTime).isBefore(new DateTime.now()) && difference.inDays > 4) {
sb
..write('<https://****.tpondemand.com/entity/')
..write(item['Id'])
..write('|')
..write(item['Name'])
..write('> last change: ')
..write(difference.inDays)
..write(' days ago \n');
}
;
}
send2Slack(sb.toString());
print("sent to Slack");
sb.clear();
}
I created similar code but I can't reproduce your problem.
So basically this does work when called from a Timer.
import 'dart:io';
import 'dart:async';
import 'package:http_server/http_server.dart';
Timer t;
final period = 1;
void main(args) {
t = new Timer.periodic(new Duration(minutes: period), (Timer t) => hit());
}
void hit() {
loadUrlBody('http://www.google.com')
.then((HttpClientResponseBody b) => print('hit: ${b.response.statusCode}'));
}
Future loadUrlBody(String url) {
print('executing');
HttpClient client = new HttpClient();
// commented out because I have no server where I can use it
// HttpClient client = new HttpClient()
// ..addCredentials(Uri.parse("https://****.tpondemand.com/api"), "tprealm", new HttpClientBasicCredentials("user", "password"));
return client.getUrl(Uri.parse(url)) // <== return is important here
.then((HttpClientRequest response) => response.close())
.then(HttpBodyHandler.processResponse)
.then((HttpClientResponseBody body) {
print('body: (${new DateTime.now()}) ${body.response.statusCode}');
return body; // <== this value is the value the next 'then' receives.
// for example x in: loadUrlBody('http://someurl').then(x) => doSomething(x));
});
}
You don't need to use a Completer. Completer are for more complicated used cases where for example one method returns a Completer and for example an eventHandler completes it.
You just have to ensure that you return a Future everywhere. then always returns a Future. The value of the returned Future is the value returned inside then.
I have already try to understand the API doc, the articles about them, and this post: How do you create a Stream in Dart
I'm making a simple web app using WebSocket. Actually, it's working well, but I want add a feature (enjoy learn).
This is my class (can be optimized I guess)
library Ask;
import 'dart:html';
import 'dart:async';
import 'dart:convert';
class Ask {
final String addr;
String _protocol;
String _port;
WebSocket _ws;
bool openned;
Map<int, Completer> _completer_list = {};
int _counter = 0;
static final Map<String, Ask> _cache = <String, Ask>{};
factory Ask(String addr) {
if (_cache.containsKey(addr)) {
return _cache[addr];
} else {
final ask_server = new Ask._internal(addr);
_cache[addr] = ask_server;
return ask_server;
}
}
Ask._internal(this.addr);
Future<bool> open() {
if (openned)
return true;
_completer_list[0] = new Completer();
if (window.location.protocol == 'http:') {
_port = ':8080/ws';
_protocol = 'ws://';
} else {
_port = ':8443/ws';
_protocol = 'wss://';
}
_ws = new WebSocket(_protocol + addr + _port);
_ws.onOpen.listen((e) {
_get_data();
_get_close();
openned = true;
_completer_list[0].complete(true);
});
return _completer_list[0].future;
}
Future<String> send(Map data) {
bool check = false;
int id;
_completer_list.forEach((k, v) {
if (v.isCompleted) {
id = data['ws_id'] = k;
_completer_list[k] = new Completer();
_ws.send(JSON.encode(data));
check = true;
}
});
if (!check) {
_counter++;
id = data['ws_id'] = _counter;
_completer_list[id] = new Completer();
_ws.send(JSON.encode(data));
}
return _completer_list[id].future;
}
void _get_data() {
_ws.onMessage.listen((MessageEvent data) {
var response = JSON.decode(data.data);
_completer_list[response['ws_id']].complete(response);
});
}
void _get_close() {
_ws.onClose.listen((_) {
print('Server have been lost. Try to reconnect in 3 seconds.');
new Timer(new Duration(seconds: 3), () {
_ws = new WebSocket(_protocol + addr + _port);
_get_data();
_get_close();
_ws.onOpen.listen((e) => print('Server is alive again.'));
});
});
}
}
Example of use:
void showIndex() {
Element main = querySelector('main');
Ask connect = new Ask('127.0.0.1');
Map request = {};
request['index'] = true;
connect.open().then((_) {
connect.send(request).then((data) {
main.setInnerHtml(data['response']);
});
});
}
I would replace the then by a listen who will be canceled when the message will completed. By this way, I can add a progress bar, I think...
So my question, my send function can be a stream and keep my concept of one websocket for all ? (yes, if my function is used when a request is in progress, it's sent and if she's finish before the first, I recovered her properly. Thank you ws_id).
Thank you.
I think what you need is a StreamController
https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart-async.StreamController