I have a stream in gulp, but I want to split the stream into two, and put half in one destination, and the other half in another.
My thoughts is that I need to fork the stream twice, filter each of the new streams, use gulp.dest on each stream, then merge them back, and return them back to gulp.
I currently have the code,
function dumpCpuProfiles(profileDirectory) {
const _ = require('highland');
const stream = _();
const cpuProfiles = stream
.fork()
.pipe(filter('*.cpuprofile'))
.pipe(gulp.dest(profileDirectory));
const noCpuProfiles = _(cpuProfiles).filter(() => false);
const otherFiles = stream
.fork()
.pipe(filter(['**', '!**/*.cpuprofile']));
return _([noCpuProfiles, otherFiles]).merge();
}
However, I get the error,
TypeError: src.pull is not a function
at /home/user/project/node_modules/highland/lib/index.js:3493:17
at Array.forEach (native)
at pullFromAllSources (/home/user/project/node_modules/highland/lib/index.js:3492:15)
at Stream._generator (/home/user/project/node_modules/highland/lib/index.js:3449:13)
at Stream._runGenerator (/home/user/project/node_modules/highland/lib/index.js:949:10)
at Stream.resume (/home/user/project/node_modules/highland/lib/index.js:811:22)
at Stream._checkBackPressure (/home/user/project/node_modules/highland/lib/index.js:713:17)
at Stream.resume (/home/user/project/node_modules/highland/lib/index.js:806:29)
at Stream.pipe (/home/user/project/node_modules/highland/lib/index.js:895:7)
at Gulp.<anonymous> (/home/user/project/gulpfile.js:189:6)
The output is a stream, so I'm not too sure what the error is about. Any help would be massively useful! Thanks!
I use this little small monkeypatching trick to achieve it
Object.prototype.fork = function(_fn) {
_fn(this);
return this;
};
Stream is only event emitter, pipe method doesn't return old stream but a new one, so you can built fork functionality very easy.
Related
I have a Dart console app that is calling into a third-party library.
When my console app calls the third-party library the call to the method returns however my CLI app then 'hangs' for 10 seconds or so before finally shutting down.
I suspect that the library has some type of resource that it has created but has not closed/completed.
My best guess is that it is a non-completed future.
So I'm looking for ways to detect resources that haven't been freed.
My first port of call would be looking for a technique to detect futures that haven't been completed but solutions for other resource types would be useful.
I'm currently using a runZoneGuarded, passing in a ZoneSpecification to hook calls.
Edit: with some experimentation, I've found I can detect timers and cancel them. In a simple experiment, I've found that a non-cancelled timer will cause the app to hang. If I cancel the timers (during my checkLeaks method) the app will shut down, however, this isn't enough in my real-world app so I'm still looking for ways to detect other resources.
Here is the experimental code I have:
#! /usr/bin/env dcli
import 'dart:async';
import 'package:dcli/dcli.dart';
import 'package:onepub/src/pub/global_packages.dart';
import 'package:onepub/src/pub/system_cache.dart';
import 'package:onepub/src/version/version.g.dart';
import 'package:pub_semver/pub_semver.dart';
void main(List<String> arguments) async {
print(orange('OnePub version: $packageVersion '));
print('');
print(globals);
// await globals.repairActivatedPackages();
await runZonedGuarded(() async {
Timer(Duration(seconds: 20), () => print('timer done'));
unawaited(Future.delayed(Duration(seconds: 20)));
var completer = Completer();
unawaited(
Future.delayed(Duration(seconds: 20), () => completer.complete()));
// await globals.activateHosted(
// 'dcli_unit_tester',
// VersionConstraint.any,
// null, // all executables
// overwriteBinStubs: true,
// url: null, // hostedUrl,
// );
print('end activate');
}, (error, stackTrace) {
print('Uncaught error: $error');
}, zoneSpecification: buildZoneSpec());
print('end');
checkLeaks();
// await entrypoint(arguments, CommandSet.ONEPUB, 'onepub');
}
late final SystemCache cache = SystemCache(isOffline: false);
GlobalPackages? _globals;
GlobalPackages get globals => _globals ??= GlobalPackages(cache);
List<void Function()> actions = [];
List<Source<Timer>> timers = [];
int testCounter = 0;
int timerCount = 0;
int periodicCallbacksCount = 0;
int microtasksCount = 0;
ZoneSpecification buildZoneSpec() {
return ZoneSpecification(
createTimer: (source, parent, zone, duration, f) {
timerCount += 1;
final result = parent.createTimer(zone, duration, f);
timers.add(Source(result));
return result;
},
createPeriodicTimer: (source, parent, zone, period, f) {
periodicCallbacksCount += 1;
final result = parent.createPeriodicTimer(zone, period, f);
timers.add(Source(result));
return result;
},
scheduleMicrotask: (source, parent, zone, f) {
microtasksCount += 1;
actions.add(f);
final result = parent.scheduleMicrotask(zone, f);
return result;
},
);
}
void checkLeaks() {
print(actions.length);
print(timers.length);
print('testCounter $testCounter');
print('timerCount $timerCount');
print('periodicCallbacksCount $periodicCallbacksCount');
print('microtasksCount $microtasksCount');
for (var timer in timers) {
if (timer.source.isActive) {
print('Active Timer: ${timer.st}');
timer.source.cancel();
}
}
}
class Source<T> {
Source(this.source) {
st = StackTrace.current;
}
T source;
late StackTrace st;
}
I'm my real-world testing I can see that I do have hanging timers caused by HTTP connections. As I originally guessed this does seem to point to some other problem with the HTTP connections not being closed down correctly.
Active Timer: #0 new Source (file:///home/bsutton/git/onepub/onepub/bin/onepub.dart:105:21)
#1 buildZoneSpec.<anonymous closure> (file:///home/bsutton/git/onepub/onepub/bin/onepub.dart:68:18)
#2 _CustomZone.createTimer (dart:async/zone.dart:1388:19)
#3 new Timer (dart:async/timer.dart:54:10)
#4 _HttpClientConnection.startTimer (dart:_http/http_impl.dart:2320:18)
#5 _ConnectionTarget.returnConnection (dart:_http/http_impl.dart:2381:16)
#6 _HttpClient._returnConnection (dart:_http/http_impl.dart:2800:41)
#7 _HttpClientConnection.send.<anonymous closure>.<anonymous closure>.<anonymous closure> (dart:_http/http_impl.dart:2171:25)
#8 _rootRunUnary (dart:async/zone.dart:1434:47)
In general, it's impossible to find things that doesn't happen.
There is no way to find all futures in the program.
With a zone, you might be able to intercept all the callbacks being "registered" in the zone, but you can't know which of them must be called. A future can have both value handlers and an error handlers, and at most one of them will ever be called. So, just because a callback on a future isn't called, it doesn't mean the future didn't complete.
A future most likely won't keep the isolate alive, though.
An incompleted future will just be garbage collected if nothing important is hanging on to it.
The most likely culprits for keeping an isolate alive are timers and receive ports.
(The VM internal implementation of timers, and I/O, and sockets, all use receive ports, so it's really just the ports.)
Again, there is no way to find all open ports programmatically.
You need a debugger with memory inspection tools for that.
I'd recommend using the developer tools to look for instances of ReceivePort or RawReceivePort that are not being garbage collected, and see whether they are still alive.
Also be careful with runZonedGuarded.
Since runZonedGuarded introduces a new error zone (because it introduces an uncaught error handler in the new zone), an error future created inside the zone will not be seen to complete outside the zone.
That means that the code:
await runZonedGuarded(() async {
will not work if the body throws. The error of the future is handled by the zone instead of the await, so the await just sees a future which never completes.
I'm looking at automating a drag and drop scenario using Playwright. I found this code which is meant to do just this:
// Read your file into a buffer.
const buffer = readFileSync('./runtime_config/common/file.pdf');
// Create the DataTransfer and File
const dataTransfer = await scope.page.evaluateHandle((data) => {
const dt = new DataTransfer();
// Convert the buffer to a hex array
const file = new File([data.toString('hex')], 'file.pdf', { type: 'application/pdf' });
dt.items.add(file);
return dt;
}, buffer);
// Now dispatch
await page.dispatchEvent('YOUR_TARGET_SELECTOR', 'drop', { dataTransfer });
This code was found on this github post https://github.com/microsoft/playwright/issues/10667
When I try to use this code, I get an error that 'scope is not defined'
I can't see how this would work, nor does the post have any info. I've reached out to the poster but no reply yet, so thought I'd try here. Anyone know how this would work?
(I tried modify the line to be this.page.evaluateHandle((data) which doesn't error at runtime, but all I am getting is a 15B file called 'file.pdf' in the UI. I have console.log the buffer and it does appear to be valid.)
I'm trying to load a large-ish (1000 lines, 68k) text file using
final String enString = await rootBundle.loadString('res/string/string_en.json');
The Dart class function AssetBundle.loadString that loads the string is
Future<String> loadString(String key, { bool cache = true }) async {
final ByteData data = await load(key);
if (data == null)
throw FlutterError('Unable to load asset: $key');
// 50 KB of data should take 2-3 ms to parse on a Moto G4, and about 400 μs
// on a Pixel 4.
if (data.lengthInBytes < 50 * 1024) {
return utf8.decode(data.buffer.asUint8List());
}
// For strings larger than 50 KB, run the computation in an isolate to
// avoid causing main thread jank.
return compute(_utf8decode, data, debugLabel: 'UTF8 decode for "$key"');
}
Looking at the code above, if the file is bigger than 50k, as mine is, an isolate is used.
As a test, I cut my file in half (so 32k) and it loaded in a second (not using the isolate). But, unedited, the function hangs when the isolate is used.
My files is just a simple json file of key-value pairs. Here are the first few lines
{
"ctaButtonConfirm": "Confirm",
"ctaButtonContinue": "Continue",
"ctaButtonReview": "Review",
"balance": "Balance",
"totalBalance": "Total Balance",
"transactions": "Transactions",
:
Seem like it hangs when the isolate is used?
EDIT
Based on the loadString code above I wrote an extension function that doesn't use an isolate and it works fine, so it's looking like the isolate doesn't like my file?
extension AssetBundleX on AssetBundle {
Future<String> loadStringWithoutIsolate(String key) async {
final ByteData data = await load(key);
return utf8.decode(data.buffer.asUint8List());
}
}
You can't access rootBundle from spawned isolate.
So use main isolate instead.
Or in [docs](This is useful for operations that take longer than a few milliseconds, and which would therefore risk skipping frames. For tasks that will only take a few milliseconds, consider SchedulerBinding.scheduleTask instead.)
you can try SchedulerBinding.scheduleTask instead.
In a simple JavaScript Service Worker I want to intercept a request and read a value from IndexedDB before the event.respondWith
But the asynchronous nature of IndexDB does not seem to allow this.
Since the indexedDB.open is asynchronous, we have to await it which is fine. However, the callback (onsuccess) happens later so the function will exit immediately after the await on open.
The only way I have found to get it to work reliably is to add:
var wait = ms => new Promise((r, j) => setTimeout(r, ms));
await wait(50)
at the end of my readDB function to force a wait until the onsuccess has completed.
This is completely stupid!
And please don't even try to tell me about promises. They DO NOT WORK in this circumstance.
Does anyone know how we are supposed to use this properly?
Sample readDB is here (all error checking removed for clarity). Note, we cannot use await inside the onsuccess so the two inner IndexedDB calls are not awaited!
async function readDB(dbname, storeName, id) {
var result;
var request = await indexedDB.open(dbname, 1); //indexedDB.open is an asynchronous function
request.onsuccess = function (event) {
let db = event.target.result;
var transaction = db.transaction([storeName], "readonly"); //This is also asynchronous and needs await
var store = transaction.objectStore(storeName);
var objectStoreRequest = store.get(id); //This is also asynchronous and needs await
objectStoreRequest.onsuccess = function (event) {
result = objectStoreRequest.result;
};
};
//Without this wait, this function returns BEFORE the onsuccess has completed
console.warn('ABOUT TO WAIT');
var wait = ms => new Promise((r, j) => setTimeout(r, ms));
await wait(50)
console.warn('WAIT DONE');
return result;
}
And please don't even try to tell me about promises. They DO NOT WORK in this circumstance.
...
...
...
I mean, they do, though. Assuming that you're okay putting the promise-based IndexedDB lookups inside of event.respondWith() rather than before event.respondWith(), at least. (If you're trying to do this before calling event.respondWith(), to figure out whether or not you want to respond at all, you're correct in that it's not possible, since the decision as to whether or not to call event.respondWith() needs to be made synchronously.)
It's not easy to wrap IndexedDB in a promise-based interface, but https://github.com/jakearchibald/idb has already done the hard work, and it works quite well inside of a service worker. Moreover, https://github.com/jakearchibald/idb-keyval makes it even easier to do this sort of thing if you just need a single key/value pair, rather than the full IndexedDB feature set.
Here's an example, assuming you're okay with idb-keyval:
importScripts('https://cdn.jsdelivr.net/npm/idb-keyval#3/dist/idb-keyval-iife.min.js');
// Call idbKeyval.set() to save data to your datastore in the `install` handler,
// in the context of your `window`, etc.
self.addEventListener('fetch', event => {
// Optionally, add in some *synchronous* criteria here that examines event.request
// and only calls event.respondWith() if this `fetch` handler can respond.
event.respondWith(async function() {
const id = someLogicToCalculateAnId();
const value = await idbKeyval.get(id);
// You now can use `value` however you want.
const response = generateResponseFromValue(value);
return response;
}())
});
I'm using gulp.js. I want to pipe the resulting JSON data from gulp-yaml, straight into gulp-ejs. My logic says that I'm probably going to have to get synchronous at some point?
What I have tried thus far:
var yaml = require('gulp-yaml');
var ejs = require('gulp-ejs');
gulp.task('yaml', function(){
return gulp.src('./path/to/template.ejs')
.pipe( ejs( gulp.src('./path/to/data.yml').pipe( yaml() ) ) )
.pipe(gulp.dest('./path/to/dest'));
});
I realise that the above is an idiotic script as I'm essentially trying to pass a stream as an argument to ejs. And as predicted, it doesn't work. I've tried numerous other things but I'm guessing someone has done this before?
I reckon gulp-data will have something to do with the solution...
You probably don't need the gulp-data plugin to achieve this; like you said you're just passing the option synchronously to the ejs plugin, and you're only needing the one yaml config file. So this is probably better:
var yaml = require('js-yaml');
var ejs = require('gulp-ejs');
var fs = require('fs');
gulp.task('ejs-with-yaml', function () {
return gulp.src('path/to/template.ejs')
.pipe(ejs(yaml.safeLoad(fs.readFileSync('path/to/data.yml', 'utf-8'))))
.pipe(gulp.dest('path/to/dest'));
});
Okay, so after a bit of head-scratching I've been able to find a solution using js-yaml and gulp-data.
var yaml = require('js-yaml');
var ejs = require('gulp-ejs');
var data = require('gulp-data');
gulp.task('ejs-with-yaml', function(){
return gulp.src('path/to/data.yml'))
.pipe(data(function(file, cb){
// throughput the stream in case of bad/absent file data
if (file.isNull()) return cb( null, file );
if (file.isStream()) return cb( new Error("Streaming not supported") );
// try and convert yaml to json
try {
json = yaml.load(String(file.contents.toString('utf8')));
} catch(e) {
console.log(e);
}
gulp.src('path/to/template.ejs').pipe(ejs(json)).pipe(gulp.dest( 'path/to/dest' ));
}));
});
Hopefully this might help others that are stuck on this. If anyone has a better solution, feel free to comment.