I'm currently using an svg conversion library which wraps puppeteer:
https://github.com/etienne-martin/svg-to-img
After each call to its convert function it waits 500 ms and if there aren't any other calls then it closes the browser instance and on the subsequent call it will again call puppeteer.launch.
I'm using this inside of a docker container running in a Kubernetes cluster. I'm wondering how expensive it is to continually call puppeteer.launch versus connecting to an already running instance of headless chrome.
I'm considering instead just always having a docker container running an instance of headless chrome and connect to it from my docker container doing the svg conversion.
Before doing this though I wanted to get a sense of what is going on behind on the scenes of launch vs connect.
Short Answer:
Using puppeteer.connect() / browser.disconnect() whenever possible is best from a performance standpoint and is ≈ 146 times faster than using puppeteer.launch() / browser.close() (according to my benchmark tests).
Detailed Answer:
I ran some tests to compare the performance of calling puppeteer.connect() / browser.disconnect() versus puppeteer.launch() / browser.close().
Each method was tested 10,000 times, and the total time for all iterations and the average time for each iteration were recorded.
My tests found that using puppeteer.connect() / browser.disconnect() is approximately 146 times faster than using puppeteer.launch() / browser.close().
You can perform the tests on your own machine using the code provided below.
Benchmark (puppeteer.launch() / browser.close()):
'use strict';
const puppeteer = require('puppeteer');
const { performance } = require('perf_hooks');
const iterations = 10000;
(async () => {
let browser;
const start_time = performance.now();
for (let i = 0; i < iterations; i++) {
browser = await puppeteer.launch();
await browser.close();
}
const end_time = performance.now();
const total_time = end_time - start_time;
const average_time = total_time / iterations;
process.stdout.write (
'Total Time:\t' + total_time + ' ms\n'
+ 'Average Time:\t' + average_time + ' ms\n'
+ 'Iterations:\t' + iterations.toLocaleString() + '\n'
);
})();
Result:
Total Time: 1339075.0866550002 ms
Average Time: 133.90750866550002 ms
Iterations: 10,000
Benchmark (puppeteer.connect() / browser.disconnect()):
'use strict';
const puppeteer = require('puppeteer');
const { performance } = require('perf_hooks');
const iterations = 10000;
(async () => {
let browser = await puppeteer.launch();
const browserWSEndpoint = browser.wsEndpoint();
browser.disconnect();
const start_time = performance.now();
for (let i = 0; i < iterations; i++) {
browser = await puppeteer.connect({
browserWSEndpoint,
});
browser.disconnect();
}
const end_time = performance.now();
const total_time = end_time - start_time;
const average_time = total_time / iterations;
process.stdout.write (
'Total Time:\t' + total_time + ' ms\n'
+ 'Average Time:\t' + average_time + ' ms\n'
+ 'Iterations:\t' + iterations.toLocaleString() + '\n'
);
process.exit();
})();
Result:
Total Time: 9198.328596000094 ms
Average Time: 0.9198328596000094 ms
Iterations: 10,000
Puppeteer Source Code:
You can view what is happening behind the scenes by inspecting the source code of the functions in question:
puppeteer.connect() source code
browser.disconnect() source code
puppeteer.launch() source code
browser.close() source code
puppeteer.launch()
puppeteer.launch() starts the chromium instance and connects to it afterwards. The start of a chromium instance takes between 100 and 150ms depending on your hardware. The connect happens instantly (as it is a websocket on a local machine).
puppeteer.connect()
puppeteer.connect() only connects to an existing chromium instance.
If the instance you are connecting to is one the same machine as your script, this should happen instantly as before (<1ms).
If you run the chromium instance on a second machine, you will introduce a network delay for the puppeteer.connect() call and all following puppeteer function calls. The delay will depend entirely on the network, but if your machines are in the same location this should be below 10ms.
svg-to-img
Regarding the library you linked: It looks like the library you linked does not support connecting to a puppeteer instance. You could also put the library on a machine and offer an API that receives the SVG code and returns the image. That way you could keep the chromium instances running.
Related
I have tried but couldn't figure it out on my own.
I know MT4 provides Pipe and WebRequest(), as a means of communication, but WebSocket isn't built as part of the Programming. So for now, Pipe is the only thing available. But communication with Pipe breaks at some point. It skips some signals when sent.
How can I get around this please guys?
How can I get around this please guys ?
Free to use either a ZeroMQ or a nanomsg signalling / messaging framework
having been in such a need many years back, started to use ZeroMQ / MQL4-binding, so as to make MetaTrader Terminal work inside a distributed-computing QuantFX-analytics and ML-based augmented trading system.
No O/S localhost-only pipe, no file-based masquerades, but a fair, distributed, low-latency signalling/messaging, with:
remote keyboard / terminal system-console ( yes, added a DSL command language )
remote centralised logs ( avoids MQL4 execution get blocked from resource contentions )
distributed remote AI/ML-predictive engine, with latency under << 80 [ms] RTT
distributed remote automated trade-management processing
ZeroMQ is a way to go, if integration needs are to be kept under your own design controls. A brief sketch was presented here, in [ ZeroMQ hierarchy in less than a five seconds ] Section.
Feel free to read more posts on this and about the differences between WebSockets and ZeroMQ here, in the zeromq and other related posts.
I tried ZeroMQ but couldn't get it working properly.
I'm using WinSockets now and so far have no problems with it.
See: https://www.mql5.com/en/blogs/post/706665
Here you will find a way to use WinSockets in MQL as a server or as a client or both.
I know links can become dead but there is no way to attach files to this answer and the code does not format properly so I cannot include it.
You can then use any programming language that also supports sockets to communicate with your MQL EA. I'm using the build-in node implementation.
See: https://www.digitalocean.com/community/tutorials/how-to-develop-a-node-js-tcp-server-application-using-pm2-and-nginx-on-ubuntu-16-04
const net = require('net');
const port = 7070;
const host = '127.0.0.1';
const server = net.createServer();
server.listen(port, host, () => {
console.log('TCP Server is running on port ' + port + '.');
});
let sockets = [];
server.on('connection', function(sock) {
console.log('CONNECTED: ' + sock.remoteAddress + ':' + sock.remotePort);
sockets.push(sock);
sock.on('data', function(data) {
console.log('DATA ' + sock.remoteAddress + ': ' + data);
// Write the data back to all the connected, the client will receive it as data from the server
sockets.forEach(function(sock, index, array) {
sock.write(sock.remoteAddress + ':' + sock.remotePort + " said " + data + '\n');
});
});
// Add a 'close' event handler to this instance of socket
sock.on('close', function(data) {
let index = sockets.findIndex(function(o) {
return o.remoteAddress === sock.remoteAddress && o.remotePort === sock.remotePort;
})
if (index !== -1) sockets.splice(index, 1);
console.log('CLOSED: ' + sock.remoteAddress + ' ' + sock.remotePort);
});
});
Pipe in MQL is implemented through files, so you can use files instead of pipes - you will receive same or faster result, and no need to care of the communication
when using this very short script on the lua console on BizHawk (it's an emulator), Both the LUA console and BizHawk crashes at the same time.
I'd like to know if the error comes from my script or from BizHawk, her's the script: (What it is supposed to do is check if the player is not moving for a certain time period [TimeoutConstant] and if he is [cause he's dead, stuck or afk] the script loads a saved state called Filename and it starts again. Here's the script:
Filename = "yolo.state"
TimeoutConstant = 80
rightmost = 0
timeout = TimeoutConstant
function initializeRun()
savestate.load(Filename)
rightmost = 0
timeout = TimeoutConstant
end
function getPositions()
marioX = memory.read_s16_le(0x94)
marioY = memory.read_s16_le(0x96)
local layer1x = memory.read_s16_le(0x1A);
local layer1y = memory.read_s16_le(0x1C);
screenX = marioX-layer1x
screenY = marioY-layer1y
end
initializeRun()
while true do
getPositions()
if marioX > rightmost then
rightmost = marioX
timeout = TimeoutConstant
end
if timeout <= 0 then
initializeRun()
end
timeout = timeout - 1
end
I assume by "crash" you mean "freeze" which is not at all the same thing. It's freezing because your script is putting the emulator into a busy loop. You didn't do anything to advance time in the emulator. The final two lines of your script need to be:
emu.frameadvance();
end
BTW, with emulator lua scripts, the name of the game being scripted is essential information.
The examples in the RxJS README seem to suggest we have to subscribe to a source. In other words: we wait for the source to send events. In that sense, sources seem to be push-based: the source decides when it creates new items.
This contrasts, however, with iterators, where strictly speaking new items need only be created when requested, i.e., when a call is made to next(). This is pull-based behavior, also known as lazy generation.
For instance, a stream could return all Wikipedia pages for prime numbers. The items are only generated when you ask for them, because generating all of them upfront is quite an investment, and maybe only 2 or 3 of them might be read anyway.
Can RxJS also have such pull-based behavior, so that new items are only generated when you ask for them?
The page on backpressure seems to indicate that this is not possible yet.
Short answer is no.
RxJS is designed for reactive applications so as you already mentioned if you need pull-based semantics you should be using an Iterator instead of an Observable. Observables are designed to be the push-based counterparts to the iterator, so they really occupy different spaces algorithmically speaking.
Obviously, I can't say this will never happen, because that is something the community will decide. But as far as I know 1) the semantics for this case just aren't that good and 2) this runs counter to the idea of reacting to data.
A pretty good synopsis can be found here. It is for Rx.Net but the concepts are similarly applicable to RxJS.
Controlled observable from the page you referenced can change a push observable to pull.
var controlled = source.controlled();
// this callback will only be invoked after controlled.request()
controlled.subscribe(n => {
log("controlled: " + n);
// do some work, then signal for next value
setTimeout(() => controlled.request(1), 2500);
});
controlled.request(1);
A truly synchronous iterator is not possible, as it would block when the source was not emitting.
In the snippet below, the controlled subscriber only gets a single item when it signals, and it does not skip any values.
var output = document.getElementById("output");
var log = function(str) {
output.value += "\n" + str;
output.scrollTop = output.scrollHeight;
};
var source = Rx.Observable.timer(0, 1000);
source.subscribe(n => log("source: " + n));
var controlled = source.controlled();
// this callback will only be invoked after controlled.request()
controlled.subscribe(n => {
log("controlled: " + n);
// do some work, then signal for next value
setTimeout(() => controlled.request(1), 2500);
});
controlled.request(1);
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/2.5.2/rx.all.js"></script>
<body>
<textarea id="output" style="width:150px; height: 150px"></textarea>
</body>
I'm quite late to the party, but it's actually very simple to combine generators with observables. You can pull a value from a generator function by syncing it with a source observable:
const fib = fibonacci()
interval(500).pipe(
map(() => fib.next())
)
.subscribe(console.log)
Generator implementation for reference:
function* fibonacci() {
let v1 = 1
let v2 = 1
while (true) {
const res = v1
v1 = v2
v2 = v1 + res
yield res
}
}
I am running a JMeter job in Jenkins using performance plugin. I need to fail a job if the average response time < 3 seconds. I see the "duration assertion" in jmeter, but that works on each thread (each http request). Instead is it possible to do the duration assertion on average for each page?
This is the way I tried adding the BeanSehll Listener and Assertion.
Recording Controller
**Home Page**
BeanShell Listener
Debug Sampler
**Page1**
BeanShell Listener
Debug Sampler
Beanshell Assertion
View Results Tree
You can implement this check via some form of Beanshell scripting
Add a Beanshell Listener at the same level as all your requests live
Put the following code into Beanshell Listener's "Script" area
String requests = vars.get("requests");
String times = vars.get("times");
long requestsSum = 0;
long timesSum = 0;
if (requests != null && times != null) {
log.info("requests: " + requests);
requestsSum = Long.parseLong(vars.get("requests"));
timesSum = Long.parseLong(vars.get("times"));
}
long thisRequest = sampleResult.getTime();
timesSum += thisRequest;
requestsSum++;
vars.put("requests", String.valueOf(requestsSum));
vars.put("times", String.valueOf(timesSum));
long average = timesSum / requestsSum;
if (average > 3000){
sampleResult.setSuccessful(false);
sampleResult.setResponseMessage("Average response time is greater than threshold");
}
The code above will record sums of response times for each request and total number of requests into times and requests JMeter Variables
See How to use BeanShell: JMeter's favorite built-in component guide for comprehensive information on Beanshell scripting in Apache JMeter.
Based on the other answer, I managed to create something that works while using multiple threads.
Add the following code as a script to your JSR223 listener, you can also save it to file and load it from file (for easy reuse). I used a parameter in seconds to set the duration threshold.
import org.apache.jmeter.util.JMeterUtils;
int totalRequests = Integer.parseInt(ctx.getThreadGroup().getSamplerController().getProperty("LoopController.loops").getStringValue()) * ctx.getThreadGroup().getNumThreads();
long requestsCount = JMeterUtils.getPropDefault("requestsCount"+sampleResult.toString(),0);
long timesSum = JMeterUtils.getPropDefault("times"+sampleResult.toString(),0);
long thisRequestTime = sampleResult.getTime();
timesSum += thisRequestTime;
requestsCount++;
JMeterUtils.setProperty("requestsCount"+sampleResult.toString(), String.valueOf(requestsCount));
JMeterUtils.setProperty("times"+sampleResult.toString(), String.valueOf(timesSum));
long average = timesSum / requestsCount;
long threshold = Integer.parseInt(args[0])*1000;
if (requestsCount >= totalRequests) {
if(average > threshold){
sampleResult.setSuccessful(false);
sampleResult.setResponseMessage("Average response time is greater than threshold, average: " + String.valueOf(average) + ", threshold: " + threshold);
}
log.info("Average response time (" + sampleResult.toString() + "): " + String.valueOf(average) + ", threshold: " + threshold);
}
This stores it in the global properties, that are saved for the full JVM run. To keep consistency in between runs, I added a setup thread group with a JSR223 sampler, with this code:
import org.apache.jmeter.util.JMeterUtils;
JMeterUtils.getJMeterProperties().clear();
log.info("cleared properties");
I am using Jasmine with PhantomJS to run test cases.
In my typical test case, I make a service call, wait for response and confirm response.
Some requests can return in a few seconds and some can take up to a minute to return.
When ran through PhantomJS, the test case fails for the service call that is supposed to take a minute ( fails because the response is not yet received).
What's interesting is that the test passes when ran through Firefox.
I have tried looking at tcpdump and the headers are same for requests through both browsers, so this looks like a browser timeout issue.
Has anyone had a similar issue ? Any ideas as to where could the timeout be configured ? Or do you think the problem is something else ?
Ah the pain of PhantomJS.
Apparently it turned out that I was using javascript's bind function which is not supported in PhantomJS .
This was causing the test to fail which resulted in messing up state of some global variable( my fault) and hence the failure.
But the root cause was using bind.
Solution: try getting a shim for bind like this from https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Function/bind
if (!Function.prototype.bind) {
Function.prototype.bind = function (oThis) {
if (typeof this !== "function") {
// closest thing possible to the ECMAScript 5 internal IsCallable function
throw new TypeError("Function.prototype.bind - what is trying to be bound is not callable");
}
var aArgs = Array.prototype.slice.call(arguments, 1),
fToBind = this,
fNOP = function () {},
fBound = function () {
return fToBind.apply(this instanceof fNOP && oThis
? this
: oThis,
aArgs.concat(Array.prototype.slice.call(arguments)));
};
fNOP.prototype = this.prototype;
fBound.prototype = new fNOP();
return fBound;
};
}
I had exactly same issue. All you have to do is add setTimeout to exit
setTimeout(function() {phantom.exit();},20000); // stop after 20 sec ( add this before you request your webpage )
page.open('your url here', function (status) {
// operations here
});