NodeJS 0.3.1 app leaking memory - memory

I've got a fairly simple NodeJS app that is nothing but a shell that accepts requests and parses content using a form of Readability (based on https://github.com/arrix/node-readability). The problem is that I've noticed the app leaks memory - I've run node --trace-gc and the memory slowly builds up with each request with the occasional gc pass doing nothing.
$ node --trace-gc server.js
Scavenge 2.3 -> 2.3 MB, 1 ms.
Mark-sweep 3.5 -> 2.9 MB, 2 ms.
Scavenge 4.4 -> 3.8 MB, 0 ms.
16 Feb 10:57:51 - Server started on PORT 8000
Scavenge 5.9 -> 5.0 MB, 2 ms.
Mark-sweep 5.0 -> 4.2 MB, 3 ms.
Mark-compact 4.2 -> 4.1 MB, 7 ms.
Scavenge 5.2 -> 4.7 MB, 1 ms.
Scavenge 5.5 -> 5.1 MB, 1 ms.
Scavenge 5.9 -> 5.5 MB, 1 ms.
Scavenge 7.1 -> 6.3 MB, 2 ms.
Scavenge 7.8 -> 7.0 MB, 2 ms.
Mark-sweep 8.5 -> 7.6 MB, 10 ms.
Scavenge 11.7 -> 9.7 MB, 4 ms.
Scavenge 12.7 -> 11.2 MB, 5 ms.
Mark-sweep 14.2 -> 12.4 MB, 21 ms.
Scavenge 20.5 -> 16.5 MB, 10 ms.
Scavenge 22.5 -> 19.5 MB, 11 ms.
Mark-sweep 25.5 -> 22.4 MB, 38 ms.
Scavenge 36.6 -> 29.6 MB, 25 ms.
Scavenge 41.6 -> 35.6 MB, 24 ms.
Mark-sweep 46.8 -> 41.1 MB, 75 ms.
Scavenge 46.8 -> 44.0 MB, 33 ms.
Mark-sweep 57.2 -> 50.6 MB, 92 ms.
Scavenge 62.3 -> 56.5 MB, 26 ms.
Scavenge 68.5 -> 62.6 MB, 24 ms.
Scavenge 74.6 -> 68.6 MB, 26 ms.
Mark-sweep 80.6 -> 74.5 MB, 130 ms.
Scavenge 80.5 -> 77.6 MB, 25 ms.
Mark-sweep 77.6 -> 77.4 MB, 112 ms.
Mark-compact 77.4 -> 77.4 MB, 260 ms.
Even after the requests stop and the GC has time to run over and over the memory usage does not decrease.
I'm more or less convinced that I must be assigning a global variable somewhere but I can't figure out how to fix it. The code for this project lives here:
https://github.com/erskingardner/Readable
Does anyone have any suggestions (and reasoning) of what I should do?

I'll answer my own question here in hopes that it might help out other people with the same issue.
The problem was actually to do with the event.emitter's I'd used in creating the server. When just a single connection was being used there was no memory leak (think development) but when the app started to get hit a lot memory started to climb and climb and the GC wasn't doing anything. I had placed an event listener one step too high in the chain and because of that I was creating an event listener that was never being destroyed.
==
BROKEN CODE
Readable.prototype.createHTTPServer = function() {
var self = this;
var server = http.createServer(function(request, response) {
request.addListener('end', function() {
var location = url.parse(request.url, true)
,params = (location.query || request.headers)
,body = "";
if (location.pathname == '/' && request.method == "GET"){
if (params["url"] == null){
response.writeHead(200, {
'Content-Type': 'text/html'
});
response.end("Good to go, you might want to try adding a url param though.");
}
else if (params["url"] != null){
self.fetchAndParse(params["url"], params);
}
var listener = emitter.on("readability", function(result) {
response.writeHead(200, {
'Content-Type': 'text/html'
});
if (result == "error"){
response.end("error");
} else {
response.end(result.content);
}
});
}
});
});
return server
};
The listener variable was the part that was never being destroyed. I tried to force the delete of the variable but then the response was never making it to the browser. The answer was that all I had to do was move that listener var inside the else if statement and it was automatically GC'ed when the function was finished running.
==
PROPER CODE
Readable.prototype.createHTTPServer = function() {
var self = this;
var server = http.createServer(function(request, response) {
request.addListener('end', function() {
var location = url.parse(request.url, true)
,params = (location.query || request.headers)
,body = "";
if (location.pathname == '/' && request.method == "GET"){
if (params["url"] == null){
response.writeHead(200, {
'Content-Type': 'text/html'
});
response.end("Good to go, you might want to try adding a url param though.");
}
else if (params["url"] != null){
self.fetchAndParse(params["url"], params);
var listener = emitter.on("readability", function(result) {
response.writeHead(200, {
'Content-Type': 'text/html'
});
if (result == "error"){
response.end("error");
} else {
response.end(result.content);
}
});
}
}
});
});
return server
};

Like PartlyCloudy mentioned you should really update your node.js instance to latest greatest v0.4.0. Hopefully the bug has been fixed, or if not you could submit an issue over at github.
I don't think Ryan's is going to fix any bugs with that old unstable version.

Related

TSL237 sensor on ESP8266 Wemos D1 Mini

I'm trying to read a TSL237 light sensor using my ESP8266 Wemos D1 Mini board. I have got code for an Arduino Uno from here (see copied below) and loaded it on my board. I first tried pin D0 on my board (GPIO 16) for the sensor data input, then pin D1 (GPIO5). In both cases I get the same useless output; I've copied one loop of this output below. Any ideas what I'm doing wrong?
CODE:
#define TSL237 2
volatile unsigned long pulse_cnt = 0;
void setup() {
attachInterrupt(0, add_pulse, RISING);
pinMode(TSL237, INPUT);
Serial.begin(9600);
}
void add_pulse(){
pulse_cnt++;
return;
}
unsigned long Frequency() {
pulse_cnt = 0;
delay(10000);// this delay controlls pulse_cnt read out. longer delay == higher number
// DO NOT change this delay; it will void calibration.
unsigned long frequency = pulse_cnt;
return (frequency);
pulse_cnt = 0;
}
void loop() {
unsigned long frequency = Frequency();
Serial.println(frequency);
delay(5000);
}
OUTPUT:
14:26:59.605 -> ets Jan 8 2013,rst cause:2, boot mode:(3,6)
14:26:59.605 ->
14:26:59.605 -> load 0x4010f000, len 3460, room 16
14:26:59.605 -> tail 4
14:26:59.605 -> chksum 0xcc
14:26:59.605 -> load 0x3fff20b8, len 40, room 4
14:26:59.605 -> tail 4
14:26:59.605 -> chksum 0xc9
14:26:59.605 -> csum 0xc9
14:26:59.605 -> v00041fe0
14:26:59.605 -> ~ld

Extract text from wrk output

I'm running a load test with wrk2 as a job on Jenkins. I'd like to send the results of the load test to Graylog but I only want to store the Requests/Sec and average latency.
Here's what the output looks like:
Running 30s test # https://example.com
1 threads and 100 connections
Thread calibration: mean lat.: 8338.285ms, rate sampling interval: 19202ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 16.20s 6.17s 29.64s 65.74%
Req/Sec 5.00 0.00 5.00 100.00%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 15.72s
75.000% 20.81s
90.000% 24.58s
99.000% 29.13s
99.900% 29.66s
99.990% 29.66s
99.999% 29.66s
100.000% 29.66s
Detailed Percentile spectrum:
Value Percentile TotalCount 1/(1-Percentile)
4497.407 0.000000 1 1.00
7561.215 0.100000 11 1.11
11100.159 0.200000 22 1.25
12582.911 0.300000 33 1.43
14565.375 0.400000 44 1.67
15720.447 0.500000 54 2.00
16416.767 0.550000 60 2.22
17301.503 0.600000 65 2.50
18464.767 0.650000 71 2.86
19185.663 0.700000 76 3.33
20807.679 0.750000 81 4.00
21479.423 0.775000 84 4.44
22347.775 0.800000 87 5.00
22527.999 0.825000 90 5.71
23216.127 0.850000 93 6.67
23478.271 0.875000 95 8.00
23805.951 0.887500 96 8.89
24723.455 0.900000 98 10.00
25067.519 0.912500 99 11.43
25395.199 0.925000 101 13.33
26525.695 0.937500 102 16.00
26525.695 0.943750 102 17.78
26705.919 0.950000 103 20.00
28065.791 0.956250 104 22.86
28065.791 0.962500 104 26.67
28377.087 0.968750 105 32.00
28377.087 0.971875 105 35.56
28475.391 0.975000 106 40.00
28475.391 0.978125 106 45.71
28475.391 0.981250 106 53.33
29130.751 0.984375 107 64.00
29130.751 0.985938 107 71.11
29130.751 0.987500 107 80.00
29130.751 0.989062 107 91.43
29130.751 0.990625 107 106.67
29655.039 0.992188 108 128.00
29655.039 1.000000 108 inf
#[Mean = 16199.756, StdDeviation = 6170.105]
#[Max = 29638.656, Total count = 108]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
130 requests in 30.02s, 13.44MB read
Socket errors: connect 0, read 0, write 0, timeout 1192
Requests/sec: 4.33
Transfer/sec: 458.47KB
Does anyone know how I could go about extracting Requests/sec (at the bottom) and the latency average to send as JSON parameters?
The expected output would be: "latency": 16.2, "requests_per_second": 4.33
You didn't provide the expected output so your question isn't clear but is this what you want?
$ awk 'BEGIN{a["Latency"]; a["Requests/sec:"]} ($1 in a) && ($2 ~ /[0-9]/){print $1, $2}' file
Latency 16.20s
Requests/sec: 4.33
Updated based on you adding expected output to your question:
$ awk '
BEGIN { map["Latency"]="latency"; map["Requests/sec:"]="requests_per_second" }
($1 in map) && ($2 ~ /[0-9]/) { printf "%s\"%s\": %s", ofs, map[$1], $2+0; ofs=", " }
END { print "" }
' file
"latency": 16.2, "requests_per_second": 4.33

Why does app crash at NSData getBytes?

NSData is extended to determine the file type:
extension NSData {
var dataType: String? {
// Ensure data length is at least 1 byte
guard self.length > 0 else { return nil }
// Get first byte
var c = [UInt8](count: 1, repeatedValue: 0)
self.getBytes(&c, length: 1)
// Identify data type
switch (c[0]) {
case 0xFF:
return "jpg"
case 0x89:
return "png"
case 0x47:
return "gif"
case 0x49, 0x4D:
return "tiff"
default:
return nil //unknown
}
}
}
The method above is called on a NSData object from image data that is fetched from a server.
dispatch_async(dispatch_get_global_queue(QOS_CLASS_BACKGROUND, 0)) {
do {
// Fetch image synchronously from server
let query = PFQuery(className: <...>)
let result = try query.getFirstObject()
guard
let imageObject = result.objectForKey(<...>) as? PFFile,
let imageData = try? imageObject.getData(),
let image = imageData.dataType == "gif" ? UIImage.animatedImageWithAnimatedGIFData(imageData) : UIImage(data: imageData)
else {
return
}
<...>
} catch (let error as NSError) {
<...>
}
}
However the app very rarely crashes at line self.getBytes:
What is the reason for this?
The buffer of getBytes is &c, an UnsafeMutablePointer - do I have to take any special memory considerations because of that?
Update
The crashes still occur with the following variation of the code:
// Get first byte
var c: UInt8 = 0;
self.getBytes(&c, length: 1)
Update
The crashes still occur with the following variation of the code:
// Get first byte
var c = [UInt8](count: 1, repeatedValue: 0)
c.withUnsafeMutableBufferPointer {
buffer in
getBytes(buffer.baseAddress, length: 1)
}
guard c.indices.contains(0) else { return nil }
I got the following crash and included the whole thread, maybe someone can spot a hint:
Thread 18 Crashed:
0 libsystem_platform.dylib 0x21a8e198 _platform_memmove$VARIANT$CortexA9 + 92
1 Foundation 0x22512923 __34-[_NSDispatchData getBytes:range:]_block_invoke + 176
2 libdispatch.dylib 0x218d238d _dispatch_data_apply + 82
3 libdispatch.dylib 0x218d4a51 dispatch_data_apply + 26
4 Foundation 0x22512865 -[_NSDispatchData getBytes:range:] + 86
5 Foundation 0x2267730b -[_NSDispatchData getBytes:length:] + 24
6 MyAppName 0x00079ba0 partial apply forwarder for (extension in MyAppName):__ObjC.NSData.(dataType.getter : Swift.String?).(closure #1) (NSData+Extension.swift:54)
7 MyAppName 0x00079c14 partial apply forwarder for reabstraction thunk helper from #callee_owned (#inout Swift.UnsafeMutableBufferPointer<Swift.UInt8>) -> (#unowned (), #error #owned Swift.ErrorType) to #callee_owned (#inout Swift.UnsafeMutableBufferPointer<Swift.UInt8>) -> (#out (), #error #owned Swift.ErrorType) (NSData+Extension.swift:0)
8 MyAppName 0x00079cb8 generic specialization <Swift.UInt8, ()> of Swift.Array.withUnsafeMutableBufferPointer <A> ((inout Swift.UnsafeMutableBufferPointer<A>) throws -> A1) throws -> A1 (NSData+Extension.swift:0)
9 MyAppName 0x00079a70 (extension in MyAppName):__ObjC.NSData.dataType.getter : Swift.String? (NSData+Extension.swift:55)
10 MyAppName 0x00079948 #objc (extension in MyAppName):__ObjC.NSData.dataType.getter : Swift.String? (NSData+Extension.swift:0)
11 MyAppName 0x000d2264 MyAppName.DataManager.(fetchImagesFromServer (MyAppName.ImageSet) -> ()).(closure #1) (DataManager.swift:1214)
12 libdispatch.dylib 0x218cd823 _dispatch_call_block_and_release + 8
13 libdispatch.dylib 0x218dc5e9 _dispatch_root_queue_drain + 1558
14 libdispatch.dylib 0x218dbfcd _dispatch_worker_thread3 + 94
15 libsystem_pthread.dylib 0x21a91b29 _pthread_wqthread + 1022
16 libsystem_pthread.dylib 0x21a91718 start_wqthread + 6
Update
The crashes still occur with the following variation of the code:
// Get first byte
var c = UnsafeMutablePointer<UInt8>.alloc(1)
defer { c.dealloc(1) }
self.getBytes(c, length: 1)
switch (c[0]) { ...
With the help of an Apple engineer (via a TSI ticket) the issue was finally identified.
All code permutations above for reading the first byte are valid and working.
The issue was that the NSData object was created when a file was fetched from a server using the Parse iOS SDK which stores the data in a temporary file with file protection key NSFileProtectionCompleteUntilFirstUserAuthentication.
The file protection key allows reading data of the NSData object only after the user unlocks the device once after reboot. Although the data is not readable before unlocking, the NSData object can be created and even the NSData.length property is accessible. However, attempting to read the data would throw an exception.
I changed the code and added a check if the protected data is available before attempting to read it with UIApplication.sharedApplication().protectedDataAvailable.
You may wonder why a file was fetched by the app before the device was even unlocked. The app was started by a remote user notification. That explains why the crash happened so rarely.
Learned 2 things:
Always check your file protection key
Apple technical support gave a super in-depth explanation and is worth the money
A Swift array is more like a C++ std::vector than a C array: it has other contents besides the array elements. You can't get a pointer to the first element using &c. You need to ask the array for a pointer to its elements, like this:
var c = [UInt8](count: 1, repeatedValue: 0)
c.withUnsafeMutableBufferPointer { buffer in
getBytes(buffer.baseAddress, length: 1)
}
For your particular case, it seems overkill to use an array, though. Why not just do:
var c: UInt8 = 0;
self.getBytes(&c, length: 1)

Remote Connection like Browser Toolbox

I'm trying to interact with another open profile, which is a seperate process. Browser Toolbox does this. I was wondering how can I re-simulate this behavior? Without the prompt asking for "allow remote connection"?
My goal is to (1) find all open firefox process, (2) access each of its xpcom and figure out the profile name, (3) and if its a profile name Im interested in, Ill focus its most recent window.
I don't know but I'm getting somewhere by tracing it in MXR:
http://mxr.mozilla.org/mozilla-release/source/browser/devtools/framework/toolbox-process-window.js#11
11 let { debuggerSocketConnect, DebuggerClient } =
12 Cu.import("resource://gre/modules/devtools/dbg-client.jsm", {});
13 let { ViewHelpers } =
14 Cu.import("resource:///modules/devtools/ViewHelpers.jsm", {});
15
16 /**
17 * Shortcuts for accessing various debugger preferences.
18 */
19 let Prefs = new ViewHelpers.Prefs("devtools.debugger", {
20 chromeDebuggingHost: ["Char", "chrome-debugging-host"],
21 chromeDebuggingPort: ["Int", "chrome-debugging-port"]
22 });
23
24 let gToolbox, gClient;
25
26 function connect() {
27 window.removeEventListener("load", connect);
28 // Initiate the connection
29 let transport = debuggerSocketConnect(
30 Prefs.chromeDebuggingHost,
31 Prefs.chromeDebuggingPort
32 );
33 gClient = new DebuggerClient(transport);
34 gClient.connect(() => {
35 let addonID = getParameterByName("addonID");
36
37 if (addonID) {
38 gClient.listAddons(({addons}) => {
39 let addonActor = addons.filter(addon => addon.id === addonID).pop();
40 openToolbox({ addonActor: addonActor.actor, title: addonActor.name });
41 });
42 } else {
43 gClient.listTabs(openToolbox);
44 }
45 });
46 }
47
I ran the profile and it looks like the pref ..-host is localhost and ..-port is 6080. I'm not sure how this helps target a specific profile though. Maybe on start of the browser toolbox it opened port 6080 to the opener profile. I'm not sure, but if its true, then you'll have to run code from within the target profile to open a port maybe.
Totally not sure though.
But port is opened here:
http://mxr.mozilla.org/mozilla-release/source/browser/devtools/framework/ToolboxProcess.jsm#107
106
107 BrowserToolboxProcess.prototype = {
108 /**
109 * Initializes the debugger server.
110 */
111 _initServer: function() {
112 dumpn("Initializing the chrome toolbox server.");
113
114 if (!this.loader) {
115 // Create a separate loader instance, so that we can be sure to receive a
116 // separate instance of the DebuggingServer from the rest of the devtools.
117 // This allows us to safely use the tools against even the actors and
118 // DebuggingServer itself, especially since we can mark this loader as
119 // invisible to the debugger (unlike the usual loader settings).
120 this.loader = new DevToolsLoader();
121 this.loader.invisibleToDebugger = true;
122 this.loader.main("devtools/server/main");
123 this.debuggerServer = this.loader.DebuggerServer;
124 dumpn("Created a separate loader instance for the DebuggerServer.");
125
126 // Forward interesting events.
127 this.debuggerServer.on("connectionchange", this.emit.bind(this));
128 }
129
130 if (!this.debuggerServer.initialized) {
131 this.debuggerServer.init();
132 this.debuggerServer.addBrowserActors();
133 dumpn("initialized and added the browser actors for the DebuggerServer.");
134 }
135
136 this.debuggerServer.openListener(Prefs.chromeDebuggingPort);
137
138 dumpn("Finished initializing the chrome toolbox server.");
139 dumpn("Started listening on port: " + Prefs.chromeDebuggingPort);
140 },
141

Golang append memory allocation VS. STL push_back memory allocation

I compared the Go append function and the STL vector.push_back and found that different memory allocation strategy which confused me. The code is as follow:
// CPP STL code
void getAlloc() {
vector<double> arr;
int s = 9999999;
int precap = arr.capacity();
for (int i=0; i<s; i++) {
if (precap < i) {
arr.push_back(rand() % 12580 * 1.0);
precap = arr.capacity();
printf("%d %p\n", precap, &arr[0]);
} else {
arr.push_back(rand() % 12580 * 1.0);
}
}
printf("\n");
return;
}
// Golang code
func getAlloc() {
arr := []float64{}
size := 9999999
pre := cap(arr)
for i:=0; i<size; i++ {
if pre < i {
arr = append(arr, rand.NormFloat64())
pre = cap(arr)
log.Printf("%d %p\n", pre, &arr)
} else {
arr = append(arr, rand.NormFloat64())
}
}
return;
}
But the memory address is invarient to the increment of size expanding, this really confused me.
By the way, the memory allocation strategy is different in this two implemetation (STL VS. Go), I mean the expanding size. Is there any advantage or disadvantage? Here is the simplified output of code above[size and first element address]:
Golang CPP STL
2 0xc0800386c0 2 004B19C0
4 0xc0800386c0 4 004AE9B8
8 0xc0800386c0 6 004B29E0
16 0xc0800386c0 9 004B2A18
32 0xc0800386c0 13 004B2A68
64 0xc0800386c0 19 004B2AD8
128 0xc0800386c0 28 004B29E0
256 0xc0800386c0 42 004B2AC8
512 0xc0800386c0 63 004B2C20
1024 0xc0800386c0 94 004B2E20
1280 0xc0800386c0 141 004B3118
1600 0xc0800386c0 211 004B29E0
2000 0xc0800386c0 316 004B3080
2500 0xc0800386c0 474 004B3A68
3125 0xc0800386c0 711 004B5FD0
3906 0xc0800386c0 1066 004B7610
4882 0xc0800386c0 1599 004B9768
6102 0xc0800386c0 2398 004BC968
7627 0xc0800386c0 3597 004C1460
9533 0xc0800386c0 5395 004B5FD0
11916 0xc0800386c0 8092 004C0870
14895 0xc0800386c0 12138 004D0558
18618 0xc0800386c0 18207 004E80B0
23272 0xc0800386c0 27310 0050B9B0
29090 0xc0800386c0 40965 004B5FD0
36362 0xc0800386c0 61447 00590048
45452 0xc0800386c0 92170 003B0020
56815 0xc0800386c0 138255 00690020
71018 0xc0800386c0 207382 007A0020
....
UPDATE:
See comments for Golang memory allocation strategy.
For STL, the strategy depends on the implementation. See this post for further information.
Your Go and C++ code fragments are not equivalent. In the C++ function, you are printing the address of the first element in the vector, while in the Go example you are printing the address of the slice itself.
Like a C++ std::vector, a Go slice is a small data type that holds a pointer to an underlying array that holds the data. That data structure has the same address throughout the function. If you want the address of the first element in the slice, you can use the same syntax as in C++: &arr[0].
You're getting the pointer to the slice header, not the actual backing array. You can think of the slice header as a struct like
type SliceHeader struct {
len,cap int
backingArray unsafe.Pointer
}
When you append and the backing array is reallocated, the pointer backingArray will likely be changed (not necessarily, but probably). However, the location of the struct holding the length, cap, and pointer to the backing array doesn't change -- it's still on the stack right where you declared it. Try printing &arr[0] instead of &arr and you should see behavior closer to what you expect.
This is pretty much the same behavior as std::vector, incidentally. Think of a slice as closer to a vector than a magic dynamic array.

Resources