I inherited a script to manage a deploy of Salesforce code to multiple orgs in one go, to ensure all orgs are on the same version. The code is maintained in a Github respository and the final step is the update of the main branch, so the deploy therefore has to be successful for all orgs before it updates the main branch. Currently we have 32 orgs for which the deploys run simultaneously (with more to be added).
The final step after the code has deployed successfully is to check all the Salesforce to Salesforce connections and mappings, since all the orgs update a 'hub' org. It is in this step that I've started getting Puppeteer timeouts. Sometimes it completes, sometimes it fails. It seems to be getting worse in that I have to rerun it 2 or 3 times to get it pass without timing out. I'm not experienced in Node or Puppeteer or scripts like these so don't know how to stop this happening. I've tried increasing the timeout from the default 30000 to 90000 but even then it fails sometimes so that is not a solution, obviously.
Interestingly a few of us have also been having problems lately with Chrome being dreadfully slow and timing out just in the browser (we run on the latest version of Chrome) and I read that Puppeteer uses Chrome. I tried googling but haven't found anything that helps me hence posting this query here.
I would appreciate any help to sort this out because running it multiple times for each deploy is not a viable solution, especially with the length of time it takes to complete.
This is the function from where it sets the timeout.
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000); // added this but it still times out
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
And this is the called function where I believe the timeout happens:
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
This is the error thrown (after running 2hrs 40min!)It is close to the end of the process so has completed most of the org checks at this stage. It doesn't always fail in the same place or on the same org checks so the timeout is not related to a specific connection.
The full script is here:
import puppeteer, { Browser, Page } from "puppeteer";
import { flatten } from "lodash";
import yargs from "yargs";
import pAll from "p-all";
import {
loginAndGetConnections,
Connection
} from "../page-objects/sf2sf-home.page-object";
import {
getSubscribedObjects,
ConnectionObject
} from "../page-objects/sf2sf-connection.page-object";
import {
SubscribedField,
SubscribedFieldOption,
getSubscribedFields,
gotoObject,
selectMapping,
getPicklistValues,
save as saveSubscribedFields,
getPicklistMappingLinks,
savePicklistMapping
} from "../page-objects/sf2sf-subscribed-fields.page-object";
import { SClusterConfig } from "../s-cluster-config";
class Config {
configFile: string;
clusterConfigFile: string;
dryRun: boolean;
strict: boolean;
concurrency: number;
skipPicklists: boolean;
constructor() {
// eslint-disable-next-line #typescript-eslint/no-explicit-any
const argv: any = yargs
.scriptName("publish-connections")
.describe("config-file", "The file configuring the SF2SF sync.")
.alias("config-file", "c")
.default("config-file", "./sf2sf.config.json")
.describe("cluster-config-file", "The file configuring the SF2SF sync.")
.alias("cluster-config-file", "f")
.string("cluster-config-file")
.required("cluster-config-file")
.describe(
"dry-run",
"don't make any changes, just print what you're going to do."
)
.boolean("dry-run")
.default("dry-run", false)
.describe("strict", "Prevents associations from being unassigned")
.boolean("strict")
.default("strict", false)
.number("concurrency")
.default("concurrency", 10)
.describe("skip-picklists", "Skip assigning the picklists")
.boolean("skip-picklists")
.default("skip-picklists", false).argv;
this.configFile = argv["config-file"];
this.clusterConfigFile = argv["cluster-config-file"];
this.dryRun = argv["dry-run"];
this.strict = argv["strict"];
this.concurrency = argv["concurrency"];
this.skipPicklists = argv["skip-picklists"];
}
}
interface SubscribedFieldUpdate {
connection: Connection;
connectionObject: ConnectionObject;
connectionField: SubscribedField;
newValue?: SubscribedFieldOption;
}
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000);
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
(async (): Promise<void> => {
const argv = new Config();
const { clusterConfigFile, concurrency } = argv;
const clusterConfig = await SClusterConfig.fromPath(clusterConfigFile);
const browser = await puppeteer.launch({});
const connections = flatten(
await pAll(
clusterConfig.usernames.map(
(username) => (): Promise<Connection[]> =>
loginAndGetConnections(browser, username)
),
{ concurrency }
)
).filter((conn) => conn.isActive);
const differences: SubscribedFieldUpdate[] = [];
await pAll(
connections.map(
(connection) => (): Promise<void> =>
checkDifferencesForConnectionSafely(
argv,
browser,
connection,
differences
)
),
{ concurrency }
);
const result = differences.map(
({ connection, connectionObject, connectionField, newValue }) => ({
username: connection.username,
connection: connection.name,
object: connectionObject.name,
field: connectionField.name,
oldValue: (connectionField.value && connectionField.value.name) || "",
newValue: (newValue && newValue.name) || ""
})
);
console.log(JSON.stringify(result, null, " "));
await browser.close();
})();
I have DocumentReference locationDocumentRef; in my state.
I'm changing locationDocumentRef based on the references, whether I gather by querying or by adding new document.
So I have this function to check the documents, if there is one set its reference to the locationDocumentRef, or add a new one and set its ref to the locationDocumentRef. I'm resetting its value everytime by setting it to null, since I didn't want to get previous result. But it prints null.
So my question is, how can I resolve them and get the value? I think I'm resolving too early in my code, so I can't await a non-future value. How can I fix it?
void firestoreCheckAndPush() async {
setState(() {
locationDocumentRef = null;
});
bool nameExists = await doesNameAlreadyExist(placeDetail.name);
if (nameExists) {
print('name exist');
} else {
print('name will be pushed on firestore');
pushNameToFirestore(placeDetail);
}
var resolvedRef = await locationDocumentRef;
print(resolvedRef.documentID); // I get null here
}
These are the functions that I have used
Future<bool> doesNameAlreadyExist(String name) async {
QuerySnapshot queryDb = await Firestore.instance
.collection('locations')
.where("city", isEqualTo: '${name}')
.limit(1)
.getDocuments();
if (queryDb.documents.length == 1) {
setState(() {
locationDocumentRef = queryDb.documents[0].reference;
});
return true;
} else {
return false;
}
}
And the other
void pushNameToFirestore(PlaceDetails pd) async {
DocumentReference justAddedRef =
await Firestore.instance.collection('locations').add(<String, String>{
'city': '${pd.name}',
'image': '${buildPhotoURL(pd.photos[0].photoReference)}',
});
setState(() {
locationDocumentRef = justAddedRef;
});
}
there is two mistakes i saw first here
var resolvedRef = await locationDocumentRef;
why you await for locationDocumentRef,
second you dont wait for pushNameToFirestore(PlaceDetails pd) firestoreCheckAndPush() function which is weird since pushNameToFirestore(String) is sync and this means you wouldnt wait for it to finish so if you are adding a new name it would print null.
correct me if i am wrong.
you can find more about sync and future here https://www.dartlang.org/tutorials/language/futures
look at the graph at the middle of the page
Try this
Future<List<DocumentSnapshot>> doesNameAlreadyExist(String name) async {
QuerySnapshot data = await Firestore.instance
.collection('locations')
.where("city", isEqualTo: name)
.limit(1)
.getDocuments();
return data.documents;
}
void firestoreCheckAndPush() async {
var data = await doesNameAlreadyExist('yourname');
if (data.length > 0) {
print('name exist');;
print('Document id '+ data[0].documentID);
} else {
print('name will be pushed on firestore');
}
}
Take a look into following code.
void firestoreCheckAndPush() async {
DocumentReference documentReference;
var data = await doesNameAlreadyExist('yourname');
var dataRef = await doesNameAlreadyExist('yourname');
if (data.length > 0) {
print('name exist');
documentReference = dataRef[0].reference;
print('Document id ' + data[0].documentID);
documentReference = dataRef[0].reference;
print('Document reference ');
print(documentReference);
} else {
print('name will be pushed on firestore');
}
}
I am calling an api. I am getting a streamed response after sending the request. But i cannot parse the response and convert it to String/JSON. This is where I am calling the api.
static Future<String> callDeviceListFetchApi() async {
Completer completer = new Completer();
String jsonResponse;
String url = Constants.BASE_URL + Constants.DEVICE_REGISTER_URL;
var client = new http.Client();
var request = new http.Request('GET', Uri.parse(url));
request.headers[HttpHeaders.CONTENT_TYPE] = 'application/json';
request.headers[HttpHeaders.AUTHORIZATION] = '<auth code>';
await client.send(request).then((response) {
response.stream.bytesToString().then((value) {
print(value.toString());
jsonResponse = value.toString();
completer.complete(jsonResponse);
});
}).catchError((error) {
print(error.toString());
});
return completer.future;
}
I am getting the error,
Bad state: Stream has already been listened to Flutter error. Any idea why this is happening?
There's a couple of things wrong with your code. I think you have a slight misunderstanding about how Async and Futures work in dart - you should re-read the docs and this tutorial (part 1 and part 2).
Basically, the problem is that you were returning a 'Future' from an async function. If you return a future from an async function, it has issues (I don't know why the analyzer doesn't catch that).
Future<String> callDeviceListFetchApi() async {
Completer completer = new Completer();
String url = "<url>";
var client = new http.Client();
var request = new http.Request('GET', Uri.parse(url));
request.headers[HttpHeaders.CONTENT_TYPE] = 'application/json';
request.headers[HttpHeaders.AUTHORIZATION] =
'<auth string>';
var response = await client.send(request);
String jsonResponse;
try {
var value = await response.stream.bytesToString();
print(value.toString());
jsonResponse = value.toString();
} catch (error) {
print(error.toString());
}
return completer.complete(jsonResponse);
}
Or not async:
Future<String> callDeviceListFetchApiNotAsync() {
String url = "<url>";
var client = new http.Client();
var request = new http.Request('GET', Uri.parse(url));
request.headers[HttpHeaders.CONTENT_TYPE] = 'application/json';
request.headers[HttpHeaders.AUTHORIZATION] =
'<auth string>';
Completer completer = new Completer();
return client.send(request).then((response) {
return response.stream.bytesToString();
}).then((value) {
print(value.toString());
return value.toString();
}).catchError((error) {
print(error.toString());
// if you use catchError, whatever you return from it
// is the value you'll get wherever you resolve the future.
return null;
});
}
But unless you're trying to do something I'm not seeing, there's a way easier way to do this (assuming all you want to do is get a string from a server):
Future<String> getList() async {
var response = await http.get("<url>", headers: {
HttpHeaders.CONTENT_TYPE: 'application/json',
HttpHeaders.AUTHORIZATION: '<auth string>',
});
if (response.statusCode == 200) {
return response.body;
} else {
throw Error();
}
}
I have two classes, a user_api_manager and a base_api_manager. From user_api_manager i call the get method of base_api_manager which performs an http get request and returns a Future<String>. The getrequest is performed but i am not pass the result to my user_api_manager class. The callback result is always null.
This is my user_api_manager.dart
static Future<Map<String,dynamic>> forgotPasswordAPI(String email) async{
String url = Constants.BASE_URL + Constants.FORGOT_PASSWORD_URL + email;
await BaseApiManager.get(url: url).then((val) {
var response = JSON.decode(val);
var status = response['status'];
String message = '';
print(response);
switch (response['status']) {
case Constants.SUCCESS:
message = Constants.SUCCESS_RESPONSE;
break;
case Constants.SERVER_ERROR:
message = Constants.SERVER_ERROR_MESSAGE;
break;
case Constants.UNAUTHORISED:
message = Constants.UNAUTHORISED_MESSAGE;
break;
}
return {'status':status,'message':message};
});
}
and here is my base_api_manager.dart
static Future<String> get({url : String,
parameters : Map ,
headers: Map }) async {
var client = new http.Client();
Map<String,dynamic> resultJSON;
final c = new Completer();
await client.get(url).then((response) { //response is always null
resultJSON = {
'status' : response.statusCode,
'body' : JSON.decode(response.body)
};
c.complete(resultJSON.toString());
return c.future;
});
}
How to solve this issue?
Move the return c.future outside of the response processing, i.e you want to return this from your get otherwise you will return null.
You can simplify the code. That should make it easier to locate the problem
static Future<String> get({url : String, parameters : Map, headers: Map }) async {
var client = new http.Client();
final response = await client.get(url);
print(response.body);
var resultJSON = {
'status' : response.statusCode,
'body' : JSON.decode(response.body)
};
return resultJSON.toString()
}
What does that code print?
I have already try to understand the API doc, the articles about them, and this post: How do you create a Stream in Dart
I'm making a simple web app using WebSocket. Actually, it's working well, but I want add a feature (enjoy learn).
This is my class (can be optimized I guess)
library Ask;
import 'dart:html';
import 'dart:async';
import 'dart:convert';
class Ask {
final String addr;
String _protocol;
String _port;
WebSocket _ws;
bool openned;
Map<int, Completer> _completer_list = {};
int _counter = 0;
static final Map<String, Ask> _cache = <String, Ask>{};
factory Ask(String addr) {
if (_cache.containsKey(addr)) {
return _cache[addr];
} else {
final ask_server = new Ask._internal(addr);
_cache[addr] = ask_server;
return ask_server;
}
}
Ask._internal(this.addr);
Future<bool> open() {
if (openned)
return true;
_completer_list[0] = new Completer();
if (window.location.protocol == 'http:') {
_port = ':8080/ws';
_protocol = 'ws://';
} else {
_port = ':8443/ws';
_protocol = 'wss://';
}
_ws = new WebSocket(_protocol + addr + _port);
_ws.onOpen.listen((e) {
_get_data();
_get_close();
openned = true;
_completer_list[0].complete(true);
});
return _completer_list[0].future;
}
Future<String> send(Map data) {
bool check = false;
int id;
_completer_list.forEach((k, v) {
if (v.isCompleted) {
id = data['ws_id'] = k;
_completer_list[k] = new Completer();
_ws.send(JSON.encode(data));
check = true;
}
});
if (!check) {
_counter++;
id = data['ws_id'] = _counter;
_completer_list[id] = new Completer();
_ws.send(JSON.encode(data));
}
return _completer_list[id].future;
}
void _get_data() {
_ws.onMessage.listen((MessageEvent data) {
var response = JSON.decode(data.data);
_completer_list[response['ws_id']].complete(response);
});
}
void _get_close() {
_ws.onClose.listen((_) {
print('Server have been lost. Try to reconnect in 3 seconds.');
new Timer(new Duration(seconds: 3), () {
_ws = new WebSocket(_protocol + addr + _port);
_get_data();
_get_close();
_ws.onOpen.listen((e) => print('Server is alive again.'));
});
});
}
}
Example of use:
void showIndex() {
Element main = querySelector('main');
Ask connect = new Ask('127.0.0.1');
Map request = {};
request['index'] = true;
connect.open().then((_) {
connect.send(request).then((data) {
main.setInnerHtml(data['response']);
});
});
}
I would replace the then by a listen who will be canceled when the message will completed. By this way, I can add a progress bar, I think...
So my question, my send function can be a stream and keep my concept of one websocket for all ? (yes, if my function is used when a request is in progress, it's sent and if she's finish before the first, I recovered her properly. Thank you ws_id).
Thank you.
I think what you need is a StreamController
https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart-async.StreamController