I inherited a script to manage a deploy of Salesforce code to multiple orgs in one go, to ensure all orgs are on the same version. The code is maintained in a Github respository and the final step is the update of the main branch, so the deploy therefore has to be successful for all orgs before it updates the main branch. Currently we have 32 orgs for which the deploys run simultaneously (with more to be added).
The final step after the code has deployed successfully is to check all the Salesforce to Salesforce connections and mappings, since all the orgs update a 'hub' org. It is in this step that I've started getting Puppeteer timeouts. Sometimes it completes, sometimes it fails. It seems to be getting worse in that I have to rerun it 2 or 3 times to get it pass without timing out. I'm not experienced in Node or Puppeteer or scripts like these so don't know how to stop this happening. I've tried increasing the timeout from the default 30000 to 90000 but even then it fails sometimes so that is not a solution, obviously.
Interestingly a few of us have also been having problems lately with Chrome being dreadfully slow and timing out just in the browser (we run on the latest version of Chrome) and I read that Puppeteer uses Chrome. I tried googling but haven't found anything that helps me hence posting this query here.
I would appreciate any help to sort this out because running it multiple times for each deploy is not a viable solution, especially with the length of time it takes to complete.
This is the function from where it sets the timeout.
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000); // added this but it still times out
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
And this is the called function where I believe the timeout happens:
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
This is the error thrown (after running 2hrs 40min!)It is close to the end of the process so has completed most of the org checks at this stage. It doesn't always fail in the same place or on the same org checks so the timeout is not related to a specific connection.
The full script is here:
import puppeteer, { Browser, Page } from "puppeteer";
import { flatten } from "lodash";
import yargs from "yargs";
import pAll from "p-all";
import {
loginAndGetConnections,
Connection
} from "../page-objects/sf2sf-home.page-object";
import {
getSubscribedObjects,
ConnectionObject
} from "../page-objects/sf2sf-connection.page-object";
import {
SubscribedField,
SubscribedFieldOption,
getSubscribedFields,
gotoObject,
selectMapping,
getPicklistValues,
save as saveSubscribedFields,
getPicklistMappingLinks,
savePicklistMapping
} from "../page-objects/sf2sf-subscribed-fields.page-object";
import { SClusterConfig } from "../s-cluster-config";
class Config {
configFile: string;
clusterConfigFile: string;
dryRun: boolean;
strict: boolean;
concurrency: number;
skipPicklists: boolean;
constructor() {
// eslint-disable-next-line #typescript-eslint/no-explicit-any
const argv: any = yargs
.scriptName("publish-connections")
.describe("config-file", "The file configuring the SF2SF sync.")
.alias("config-file", "c")
.default("config-file", "./sf2sf.config.json")
.describe("cluster-config-file", "The file configuring the SF2SF sync.")
.alias("cluster-config-file", "f")
.string("cluster-config-file")
.required("cluster-config-file")
.describe(
"dry-run",
"don't make any changes, just print what you're going to do."
)
.boolean("dry-run")
.default("dry-run", false)
.describe("strict", "Prevents associations from being unassigned")
.boolean("strict")
.default("strict", false)
.number("concurrency")
.default("concurrency", 10)
.describe("skip-picklists", "Skip assigning the picklists")
.boolean("skip-picklists")
.default("skip-picklists", false).argv;
this.configFile = argv["config-file"];
this.clusterConfigFile = argv["cluster-config-file"];
this.dryRun = argv["dry-run"];
this.strict = argv["strict"];
this.concurrency = argv["concurrency"];
this.skipPicklists = argv["skip-picklists"];
}
}
interface SubscribedFieldUpdate {
connection: Connection;
connectionObject: ConnectionObject;
connectionField: SubscribedField;
newValue?: SubscribedFieldOption;
}
async function checkDifferencesForConnection(
argv: Config,
page: Page,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
await page.goto(connection.url);
const subscribedObjects = await getSubscribedObjects(page);
for (const object of subscribedObjects) {
await gotoObject(page, object);
const fields = await getSubscribedFields(page);
let changesMade = false;
for (const field of fields) {
field.isStrict = argv.strict;
if (field.selectedValueNeedsUpdate()) {
const newValue = field.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: field,
newValue
});
await selectMapping(page, field, newValue);
changesMade = true;
} else if (!field.value) {
const options = field.options.map((o) => o.name);
throw new Error(
`No value for ${connection.name} -> ${object.name} -> ${field.name}, ` +
`options: ${options.join(", ")}`
);
}
}
if (!argv.skipPicklists) {
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
await gotoObject(page, object);
changesMade = false;
}
const pickListMappings = await getPicklistMappingLinks(page);
for (const pickListMapping of pickListMappings) {
try {
await pickListMapping.click();
} catch (e) {
console.log(
`Failed ${connection.username} -> ${connection.name} -> ${object.name} -> ${pickListMapping.id}`,
e
);
throw e;
}
const picklistValues = await getPicklistValues(page);
for (const picklistValue of picklistValues) {
picklistValue.isStrict = argv.strict;
if (picklistValue.selectedValueNeedsUpdate()) {
const newValue = picklistValue.newValue();
changes.push({
connection,
connectionObject: object,
connectionField: picklistValue,
newValue
});
await selectMapping(page, picklistValue, newValue);
changesMade = true;
}
}
await savePicklistMapping(page);
}
}
if (!argv.dryRun && changesMade) {
await saveSubscribedFields(page);
}
}
}
async function checkDifferencesForConnectionSafely(
argv: Config,
browser: Browser,
connection: Connection,
changes: SubscribedFieldUpdate[]
): Promise<void> {
const page = await browser.newPage();
page.setDefaultNavigationTimeout(90000);
try {
console.log(`Checking ${connection.username} -> ${connection.name}`);
await checkDifferencesForConnection(argv, page, connection, changes);
console.log(`Finished ${connection.username} -> ${connection.name}`);
} catch (e) {
console.log(`Failed ${connection.username} -> ${connection.name}`, e);
throw e;
} finally {
await page.close();
}
}
(async (): Promise<void> => {
const argv = new Config();
const { clusterConfigFile, concurrency } = argv;
const clusterConfig = await SClusterConfig.fromPath(clusterConfigFile);
const browser = await puppeteer.launch({});
const connections = flatten(
await pAll(
clusterConfig.usernames.map(
(username) => (): Promise<Connection[]> =>
loginAndGetConnections(browser, username)
),
{ concurrency }
)
).filter((conn) => conn.isActive);
const differences: SubscribedFieldUpdate[] = [];
await pAll(
connections.map(
(connection) => (): Promise<void> =>
checkDifferencesForConnectionSafely(
argv,
browser,
connection,
differences
)
),
{ concurrency }
);
const result = differences.map(
({ connection, connectionObject, connectionField, newValue }) => ({
username: connection.username,
connection: connection.name,
object: connectionObject.name,
field: connectionField.name,
oldValue: (connectionField.value && connectionField.value.name) || "",
newValue: (newValue && newValue.name) || ""
})
);
console.log(JSON.stringify(result, null, " "));
await browser.close();
})();
In my Flutter Web application I am retrieving values from the map timeslots in Firestore.
This is what the data looks like:
But, instead of retrieving the whole list of values, I get a truncated list like this:
[Mo-Washing-(09:00-10:00, 10:00-11:00, 11:00-12:00, ..., 20:00-21:00, 21:00-22:00)]
Below I have included the 2 functions responsible for retrieving the data and adding it to the list object
static List object = [];
static Map<String, dynamic> timeDetails = {};
static Map<String, dynamic> userDetails = {};
checkExists(docuID) async {
return await firestore()
.collection('environments')
.doc(docuID)
.get()
.then((val) {
userDetails.addAll(val.data());
}).whenComplete(() async {
fs.DocumentSnapshot snapShot = await firestore()
.collection('environments')
.doc(docuID)
.collection('Washing')
.doc('monday')
.get();
if (snapShot == null || !snapShot.exists) {
print('does not exist');
} else {
await getData(docuID, 'Washing');
}
setState(() {});
});
}
getData(docuID, machineName) async {
return await firestore()
.collection('environments')
.doc(docuID)
.collection(machineName)
.doc('monday')
.get()
.then((val) {
timeDetails.addAll(val.data());
}).whenComplete(() {
object.add('Mo-$machineName-${timeDetails['timeslots'].values}');
print(object);
setState(() {});
});
}
This also happens in debugPrint. Would anyone know why this is happening and how I could solve it? Any help on this would be appreciated!
Neither the workaround as mentioned on Github nor debugPrint worked for me, but I managed to solve this by adding .toList() to my getData function:
getData(docuID, machineName) async {
return await firestore()
.collection('environments')
.doc(docuID)
.collection(machineName)
.doc('monday')
.get()
.then((val) {
timeDetails.addAll(val.data());
}).whenComplete(() {
//toList() is added here to .add
object.add('Mo-$machineName-${timeDetails['timeslots'].values.toList()}');
print(object);
setState(() {});
});
}
Output:
[Mo-Washing-[09:00-10:00, 10:00-11:00, 11:00-12:00, 12:00-13:00, 13:00-14:00, 14:00-15:00, 15:00-16:00, 16:00-17:00, 17:00-18:00, 18:00-19:00, 19:00-20:00, 20:00-21:00, 21:00-22:00]
I need to get all the document inside this collection which much these query - I'm receiving all the documents successfully but the future never ends.
I tried WhenComplete but still not working.
Future<Null> getOldVac(anId) async {
print("getOldVac");
await Firestore.instance
.collection("users")
.document(userId)
.collection("animals")
.document(anId)
.collection("anMedication")
.where("type", isEqualTo: "vac")
.where("result", isEqualTo: "")
.snapshots()
.forEach((onValue) {
print(onValue);
}).then((onValue) {
print("Done");
}).catchError((onError) {
print(onError);
});
}
I need to print "Done" once all the future is complete.
You need to return something, can be the null that you are using or a void or throw some error.
try to put return in the front of your await like this:
return await Firestore.instance
.collection("users")
.document(userId)
.collection("animals")
.document(anId)
.collection("anMedication")
.where("type", isEqualTo: "vac")
.where("result", isEqualTo: "")
.snapshots()
.forEach((onValue) {
print(onValue);
return null;
}).then((onValue) {
print("Done");
return null;
}).catchError((onError) {
print(onError);
throw onError;
});
You are mixing promises with async await!
My Problem is very similar to the one mentioned here and here, but for some reason these are not working for me.
Basically, I want to do some simple I/O operations (on a mobile), returning of list of strings (folder path) that contain a certain file format (let's assume for the sake of argument that I want to find all mp3 files).
This is the code I have
Future<List<String>> getFolders() async {
List<String> _dirs = new List();
await SimplePermissions.requestPermission(Permission.ReadExternalStorage);
_dirs = await findAllFolders();
return _dirs;
}
Future<List<String>> findAllFolders() async {
Directory _root = Directory("/sdcard");
bool _notInList = true;
List<String> _dirs = new List();
_root.list(recursive: true, followLinks: false)
.listen((FileSystemEntity entity) {
if(entity.toString().contains("mp3")) {
if(_dirs.length==0) {
_dirs.add(entity.parent.path.toString());
} else {
_notInList = true;
for (var i = 0; i < _dirs.length; ++i) {
if(_dirs[i] == entity.parent.path.toString()) {
_notInList = false;
}
}
if(_notInList) {
_dirs.add(entity.parent.path.toString());
}
}
}
});
return _dirs;
}
where I want to use _dirs outside of getFolders().
I know that findAllFolders() returns _dirs immediately, before my listen() event has finished (and so its length is always 0, although the actual method works fine, i.e. if I put print statements where I have _dirs.add() I can see that the correct directories are added, _dirs contains what I want but I have no idea how to return the finished _dirs list). I tried to do something in a similar way to the above mentioned post, where a Completer is used (to which I am getting an error message "Bad State: Future already completed"). The respective code would be
Future<List<String>> findAllFolders() async {
Directory _root = Directory("/sdcard");
bool _notInList = true;
List<String> _dirs = new List();
Completer<List<String>> _completer = new Completer<List<String>>();
_root.list(recursive: true, followLinks: false)
.listen((FileSystemEntity entity) {
if(entity.toString().contains("mp3")) {
if(_dirs.length==0) {
_dirs.add(entity.parent.path.toString());
} else {
_notInList = true;
for (var i = 0; i < _dirs.length; ++i) {
if(_dirs[i] == entity.parent.path.toString()) {
_notInList = false;
}
}
if(_notInList) {
_dirs.add(entity.parent.path.toString());
}
}
}
_completer.complete(_dirs);
});
return _completer.future;
}
The getFolders() function remains the same in this case. Could anyone point out where my logic is going wrong?
You're setting a listener, then immediately returning before any results are received - that's why your return is always empty. The body of findAllFolders() needs to wait for a response before returning. Try the below to replace _root.list().listen():
List<FileSystemEntity> files = await _root.list(recursive: true, followLinks: false).toList();
for (FileSystemEntity entity in files) {
// Do your filename logic and populate _dirs
Based on the SharedPreferences class, I try to retrieve a preference value like so:
String loadIPAddress() {
SharedPreferences.getInstance().then((SharedPreferences prefs) {
try {
var loadedValue = prefs.getString('serverIPAddress');
print('loadIPAddress <= ' + loadedValue);
return loadedValue; // [1]
} catch (e) {
print('loadIPAddress <= NOPE');
return '---'; [2]
}
});
}
Unfortunately, this doesn't return a value each time.
Q: Does the 1 and [2] return statements return the value of loadIPAddress()?
No, as you've guessed, those returns return from the then callback.
To return from loadIPAddress, refactor it like this:
Future<String> loadIPAddress() async {
SharedPreferences prefs = await SharedPreferences.getInstance();
try {
var loadedValue = prefs.getString('serverIPAddress');
print('loadIPAddress <= ' + loadedValue);
return loadedValue;
} catch (e) {
print('loadIPAddress <= NOPE');
return '---';
}
}
Note that having made loadIPAddress async, it now returns a Future, so you should call it like:
String ip = await loadIPAddress();
// or
loadIPAddress().then((String ip) {
// do something with ip - probably setState
});