As we all know, memory layout has four segments as follows
Heap
Data Segment Read/Write permission
code segment Read/Execute permission
Stack
I would like to know the access permission for heap and stack segments.
I have written simple x86 OS to load elf image.
Which the roughly layout of the four segment is as:
Code segment
Data segment
Heap
Stack
You can get the layout in linux by:
sudo pmap <pid>
0000000000400000 384K r-x-- /usr/local/bin/tmux
0000000000660000 8K rw--- /usr/local/bin/tmux
0000000000662000 108K rw--- [ anon ]
0000000001242000 132K rw--- [ anon ]
0000000001263000 2868K rw--- [ anon ]
00000037ba400000 128K r-x-- /lib64/ld-2.12.so
...
00007fff4f411000 84K rw--- [ stack ]
00007fff4f55c000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
The first segment is for executable image. Second is for the read only data. Then goes the heap. The heap in linux which allocated by malloc was called Anonymous segment. In the address near 00008000 00000000 is the stack addresses.
Which stack grow downside, and heap grow upside from the address after the data segment of program.
Then you could found the permission of the heap and stack are all read/write permission.
Ps. I'm not quite sure why there are so many Anonymous segment as I haven't go to so detail of linux for a long time. But in my x86 simple OS, heap start after data segment with alignment and padding.
Related
I'm developing an iOS app for drawing on a simple canvas.
The app manages one canvas per time (layers are managed differently).
I didn't use 3rd party libraries and all components are Apple friendly (canvas is PencilKit for example) while the UI is all SwiftUI.
The app never crashed but while testers are playing on it, the dirty memory grows and the app is noticeably slower.
The Instruments record shows a grew in correct memory when the app starts, but the memory used never decreases, even when the action using memory is finished. This causes a constant increase in dirty memory.
I'm working on simulator and testers on real devices like iPad 12.9 4th generation.
I heavily work with Dispatchers, UIImages and CIImages.
Attached part of the Instruments VM Tracker of dirty memory.
VM Tracker
VIRTUAL RESIDENT DIRTY SWAPPED VOLATILE NONVOL EMPTY REGION
REGION TYPE SIZE SIZE SIZE SIZE SIZE SIZE SIZE COUNT (non-coalesced)
=========== ======= ======== ===== ======= ======== ====== ===== =======
Activity Tracing 256K 40K 40K 0K 0K 40K 0K 1
CG raster data 46.3M 46.2M 46.2M 140K 0K 0K 0K 21
CoreAnimation 93.5M 92.5M 92.5M 1036K 0K 108K 0K 37
CoreData 8K 8K 8K 0K 0K 0K 0K 1
CoreData Object IDs 4100K 4K 4K 0K 0K 0K 0K 2
CoreImage 114.6M 114.6M 114.6M 0K 0K 0K 0K 8
CoreUI image data 588K 424K 424K 164K 0K 0K 0K 3
Foundation 1040K 1028K 1028K 0K 0K 0K 0K 4
IOSurface 28.1M 23.7M 23.7M 0K 0K 23.7M 0K 10
Image IO 9952K 5984K 5984K 3968K 0K 0K 0K 4
Kernel Alloc Once 8K 8K 8K 0K 0K 0K 0K 1
MALLOC guard page 32K 0K 0K 0K 0K 0K 0K 8
MALLOC metadata 220K 160K 160K 20K 0K 0K 0K 11
MALLOC_LARGE 7756K 5520K 5520K 2116K 0K 0K 0K 25 see MALLOC ZONE table below
MALLOC_LARGE (empty) 8464K 7676K 7676K 520K 0K 0K 0K 13 see MALLOC ZONE table below
MALLOC_LARGE_REUSABLE 30.3M 27.2M 11.7M 0K 0K 0K 0K 10 see MALLOC ZONE table below
MALLOC_NANO 128.0M 5756K 5756K 1268K 0K 0K 0K 1 see MALLOC ZONE table below
MALLOC_NANO (empty) 384.0M 0K 0K 0K 0K 0K 0K 1 see MALLOC ZONE table below
MALLOC_SMALL 112.0M 6728K 6728K 388K 0K 0K 0K 14 see MALLOC ZONE table below
MALLOC_TINY 24.0M 2772K 2772K 8K 0K 0K 0K 24 see MALLOC ZONE table below
Performance tool data 448K 432K 432K 16K 0K 0K 0K 8 not counted in TOTAL below
SQLite page cache 512K 508K 508K 4K 508K 0K 0K 8
STACK GUARD 56.0M 0K 0K 0K 0K 0K 0K 7
Stack 11.0M 400K 364K 44K 0K 0K 0K 7
Stack (reserved) 520K 0K 0K 0K 0K 0K 0K 1 reserved VM address space (unallocated)
Stack Guard 4K 0K 0K 0K 0K 0K 0K 1
VM_ALLOCATE 17.8M 1360K 1360K 148K 0K 0K 0K 44
VM_ALLOCATE (reserved) 4K 0K 0K 0K 0K 0K 0K 1 reserved VM address space (unallocated)
__DATA 16.9M 9751K 3025K 664K 0K 0K 0K 397
__DATA_CONST 33.9M 29.8M 772K 742K 0K 0K 0K 391
__DATA_DIRTY 20K 19K 19K 0K 0K 0K 0K 6
__FONT_DATA 4K 0K 0K 0K 0K 0K 0K 1
__LINKEDIT 402.5M 176.6M 0K 0K 0K 0K 0K 15
__OBJC_RO 54.0M 30.0M 0K 0K 0K 0K 0K 1
__OBJC_RW 2592K 1610K 18K 0K 0K 0K 0K 1
__TEXT 392.7M 102.7M 28K 0K 0K 0K 0K 407
__UNICODE 588K 336K 0K 0K 0K 0K 0K 1
mapped file 268.3M 32.2M 24K 0K 0K 0K 0K 221
shared memory 233.1M 1532K 1532K 231.6M 0K 4K 0K 23
unused but dirty shlib __DATA 340K 115K 115K 226K 0K 0K 0K 279
=========== ======= ======== ===== ======= ======== ====== ===== =======
TOTAL 2.4G 726.0M 331.5M 242.7M 508K 23.8M 0K 2011
TOTAL, minus reserved VM space 2.4G 726.0M 331.5M 242.7M 508K 23.8M 0K 2011
I need to understand why memory is not deallocated when a SwiftUI's view disappears.
This is the simple flow:
start the app 20MB memory
open canvas 200MB memory
close canvas (should be 20MB again) 200 MB
So:
is there a way in Instruments to find exactly which Object retains dirty memory?
Thanks
I am seeing a leak in my program. It does not get caught with "valgrind memcheck" ( I confirmed this with summary report, it was no were near the top usage I can see). I could get something closer to my memory usage upon using "valgrind massif --pages-as-heap".
However it does not report a full Traceback for the portion thats does mmap and allocates big portions of memory and I can't do a examine of memory allocation also because I can collect massif output only after program is killed.
Another thing i tried was to inspect the memory blocks taking lot of RSS space. However I don't know how to look at the contents of the memory block reported by pmap. putting that addr on gdb dint help. I heard some address randomization is used by gdb. Can some one help me how to get the symbol that corresponds to the memory location reported by pmap output.
putting that addr on gdb dint help.
I don't know what you mean by "putting that addr on gdb", but doing that correctly will help.
I heard some address randomization is used by gdb.
You heard wrong: GDB doesn't do any randomization by itself, and it (by default) disables randomization that OS performs, so as to make debugging easier and more reproducible.
Can some one help me how to get the symbol that corresponds to the memory location reported by pmap output.
You are confused: heap allocated memory doesn't have any symbols by definition.
Ok, so let's work through example of examinining memory that is visible in pmap with GDB. Let's start by compiling this program, which builds a 1 million long linked list with some strings in it:
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
typedef struct Node { struct Node *next; char payload[64]; } Node;
int main()
{
int j;
Node *head = NULL;
for (j = 0; j < 1000000; j++) {
Node *n = malloc(sizeof(*n));
n->next = head;
sprintf(n->payload, "string %d", j);
head = n;
}
return 0;
}
gcc -Wall -g -std=c99 t.c && gdb -q ./a.out
(gdb) b 17
Breakpoint 1 at 0x4005e3: file t.c, line 17.
(gdb) r
Starting program: /tmp/a.out
Breakpoint 1, main () at t.c:17
17 return 0;
Now we can examine the program with pmap:
(gdb) info prog
Using the running image of child process 23785.
Program stopped at 0x4005e3.
It stopped at breakpoint 1.
Type "info stack" or "info registers" for more information.
(gdb) shell pmap 23785
23785: /tmp/a.out
0000000000400000 4K r-x-- a.out
0000000000600000 4K r---- a.out
0000000000601000 4K rw--- a.out
0000000000602000 78144K rw--- [ anon ]
00007ffff7a11000 1784K r-x-- libc-2.19.so
00007ffff7bcf000 2048K ----- libc-2.19.so
00007ffff7dcf000 16K r---- libc-2.19.so
00007ffff7dd3000 8K rw--- libc-2.19.so
00007ffff7dd5000 20K rw--- [ anon ]
00007ffff7dda000 140K r-x-- ld-2.19.so
00007ffff7fd1000 12K rw--- [ anon ]
00007ffff7ff6000 8K rw--- [ anon ]
00007ffff7ff8000 8K r---- [ anon ]
00007ffff7ffa000 8K r-x-- [ anon ]
00007ffff7ffc000 4K r---- ld-2.19.so
00007ffff7ffd000 4K rw--- ld-2.19.so
00007ffff7ffe000 4K rw--- [ anon ]
00007ffffffde000 132K rw--- [ stack ]
ffffffffff600000 4K r-x-- [ anon ]
total 82356K
It seems pretty clear that the anon space of 78MiB starting at 0x602000 must be where most of our data is. (You can also verify this by stepping a few times through the loop.)
How can we look at this data? Like so:
(gdb) x/30gx 0x602000
0x602000: 0x0000000000000000 0x0000000000000051
0x602010: 0x0000000000000000 0x3020676e69727473
0x602020: 0x0000000000000000 0x0000000000000000
0x602030: 0x0000000000000000 0x0000000000000000
0x602040: 0x0000000000000000 0x0000000000000000
0x602050: 0x0000000000000000 0x0000000000000051
0x602060: 0x0000000000602010 0x3120676e69727473
0x602070: 0x0000000000000000 0x0000000000000000
0x602080: 0x0000000000000000 0x0000000000000000
0x602090: 0x0000000000000000 0x0000000000000000
0x6020a0: 0x0000000000000000 0x0000000000000051
0x6020b0: 0x0000000000602060 0x3220676e69727473
0x6020c0: 0x0000000000000000 0x0000000000000000
0x6020d0: 0x0000000000000000 0x0000000000000000
0x6020e0: 0x0000000000000000 0x0000000000000000
Immediately you can notice that at 0x602018, at 0x602068 and at 0x6020b8 there are ASCII strings.
You can examine these strings like so:
(gdb) x/s 0x602018
0x602018: "string 0"
(gdb) x/s 0x602068
0x602068: "string 1"
(gdb) x/s 0x6020b8
0x6020b8: "string 2"
You can also notice that at 0x602060 there is a pointer to 0x602010, and at 0x6020b0 there is a pointer to 0x602060.
That gives you a guess that there is a Node at 0x602060, and another at 0x6020b0. You can confirm this guess:
(gdb) p *(Node*)0x602060
$1 = {next = 0x602010, payload = "string 1", '\000' <repeats 55 times>}
(gdb) p *(Node*)0x6020b0
$2 = {next = 0x602060, payload = "string 2", '\000' <repeats 55 times>}
And that's all there is to it.
We have a java application running on Mule. We have the XMX value configured for 6144M, but are routinely seeing the overall memory usage climb and climb. It was getting close to 20 GB the other day before we proactively restarted it.
Thu Jun 30 03:05:57 CDT 2016
top - 03:05:58 up 149 days, 6:19, 0 users, load average: 0.04, 0.04, 0.00
Tasks: 164 total, 1 running, 163 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.2%us, 1.7%sy, 0.0%ni, 93.9%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24600552k total, 21654876k used, 2945676k free, 440828k buffers
Swap: 2097144k total, 84256k used, 2012888k free, 1047316k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3840 myuser 20 0 23.9g 18g 53m S 0.0 79.9 375:30.02 java
The jps command shows:
10671 Jps
3840 MuleContainerBootstrap
The jstat command shows:
S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
37376.0 36864.0 16160.0 0.0 2022912.0 1941418.4 4194304.0 445432.2 78336.0 66776.7 232 7.044 17 17.403 24.447
The startup arguments are (sensitive bits have been changed):
3840 MuleContainerBootstrap -Dmule.home=/mule -Dmule.base=/mule -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=256m -Djava.endorsed.dirs=/mule/lib/endorsed -XX:+HeapDumpOnOutOfMemoryError -Dmyapp.lib.path=/datalake/app/ext_lib/ -DTARGET_ENV=prod -Djava.library.path=/opt/mapr/lib -DksPass=mypass -DsecretKey=aeskey -DencryptMode=AES -Dkeystore=/mule/myStore -DkeystoreInstance=JCEKS -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dmule.mmc.bind.port=1521 -Xms6144m -Xmx6144m -Djava.library.path=%LD_LIBRARY_PATH%:/mule/lib/boot -Dwrapper.key=a_guid -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=10744 -Dwrapper.version=3.5.19-st -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang
Adding up the "capacity" items from jps shows that only my 6144m is being used for java heap. Where the heck is the rest of the memory being used? Stack memory? Native heap? I'm not even sure how to proceed.
If left to continue growing, it will consume all memory on the system and we will eventually see the system freeze up throwing swap space errors.
I have another process that is starting to grow. Currently at about 11g resident memory.
pmap 10746 > pmap_10746.txt
cat pmap_10746.txt | grep anon | cut -c18-25 | sort -h | uniq -c | sort -rn | less
Top 10 entries by count:
119 12K
112 1016K
56 4K
38 131072K
20 65532K
15 131068K
14 65536K
10 132K
8 65404K
7 128K
Top 10 entries by allocation size:
1 6291456K
1 205816K
1 155648K
38 131072K
15 131068K
1 108772K
1 71680K
14 65536K
20 65532K
1 65512K
And top 10 by total size:
Count Size Aggregate
1 6291456K 6291456K
38 131072K 4980736K
15 131068K 1966020K
20 65532K 1310640K
14 65536K 917504K
8 65404K 523232K
1 205816K 205816K
1 155648K 155648K
112 1016K 113792K
This seems to be telling me that because the Xmx and Xms are set to the same value, there is a single allocation of 6291456K for the java heap. Other allocations are NOT java heap memory. What are they? They are getting allocated in rather large chunks.
Expanding a bit more details on Peter's answer.
You can take a binary heap dump from within VisualVM (right click on the process in the left-hand side list, and then on heap dump - it'll appear right below shortly after). If you can't attach VisualVM to your JVM, you can also generate the dump with this:
jmap -dump:format=b,file=heap.hprof $PID
Then copy the file and open it with Visual VM (File, Load, select type heap dump, find the file.)
As Peter notes, a likely cause for the leak may be non collected DirectByteBuffers (e.g.: some instance of another class is not properly de-referencing buffers, so they are never GC'd).
To identify where are these references coming from, you can use Visual VM to examine the heap and find all instances of DirectByteByffer in the "Classes" tab. Find the DBB class, right click, go to instances view.
This will give you a list of instances. You can click on one and see who's keeping a reference each one:
Note the bottom pane, we have "referent" of type Cleaner and 2 "mybuffer". These would be properties in other classes that are referencing the instance of DirectByteBuffer we drilled into (it should be ok if you ignore the Cleaner and focus on the others).
From this point on you need to proceed based on your application.
Another equivalent way to get the list of DBB instances is from the OQL tab. This query:
select x from java.nio.DirectByteBuffer x
Gives us the same list as before. The benefit of using OQL is that you can execute more more complex queries. For example, this gets all the instances that are keeping a reference to a DirectByteBuffer:
select referrers(x) from java.nio.DirectByteBuffer x
What you can do is take a heap dump and look for object which are storing data off heap such as ByteBuffers. Those objects will appear small but are a proxy for larger off heap memory areas. See if you can determine why lots of those might be retained.
I want to dump the memory pages of a process once it finishes execution. I'm trying to use gdb for that, First I set break points at exit and _exit then I run the process inside gdb, once the process breaks I use info proc mappings to get the memory map of the process. it looks like the following:
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x400000 0x415000 0x15000 0x0 /path/workspace/freqmine
0x614000 0x615000 0x1000 0x14000 /path/workspace/freqmine
0x615000 0x616000 0x1000 0x15000 /path/workspace/freqmine
0x616000 0x129b000 0xc85000 0x0 [heap]
0x7ffff71f4000 0x7ffff720a000 0x16000 0x0 /lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff720a000 0x7ffff7409000 0x1ff000 0x16000 /lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff7409000 0x7ffff740a000 0x1000 0x15000 /lib/x86_64-linux-gnu/libgcc_s.so.1
0x7ffff740a000 0x7ffff750f000 0x105000 0x0 /lib/x86_64-linux-gnu/libm-2.19.so
0x7ffff750f000 0x7ffff770e000 0x1ff000 0x105000 /lib/x86_64-linux-gnu/libm-2.19.so
0x7ffff770e000 0x7ffff770f000 0x1000 0x104000 /lib/x86_64-linux-gnu/libm-2.19.so
0x7ffff770f000 0x7ffff7710000 0x1000 0x105000 /lib/x86_64-linux-gnu/libm-2.19.so
0x7ffff7710000 0x7ffff78cb000 0x1bb000 0x0 /lib/x86_64-linux-gnu/libc-2.19.so
0x7ffff78cb000 0x7ffff7acb000 0x200000 0x1bb000 /lib/x86_64-linux-gnu/libc-2.19.so
0x7ffff7acb000 0x7ffff7acf000 0x4000 0x1bb000 /lib/x86_64-linux-gnu/libc-2.19.so
0x7ffff7acf000 0x7ffff7ad1000 0x2000 0x1bf000 /lib/x86_64-linux-gnu/libc-2.19.so
0x7ffff7ad1000 0x7ffff7ad6000 0x5000 0x0
0x7ffff7ad6000 0x7ffff7bbc000 0xe6000 0x0 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
0x7ffff7bbc000 0x7ffff7dbb000 0x1ff000 0xe6000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
0x7ffff7dbb000 0x7ffff7dc3000 0x8000 0xe5000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
0x7ffff7dc3000 0x7ffff7dc5000 0x2000 0xed000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
0x7ffff7dc5000 0x7ffff7dda000 0x15000 0x0
0x7ffff7dda000 0x7ffff7dfd000 0x23000 0x0 /lib/x86_64-linux-gnu/ld-2.19.so
0x7ffff7fce000 0x7ffff7fd3000 0x5000 0x0
0x7ffff7ff7000 0x7ffff7ffa000 0x3000 0x0
0x7ffff7ffa000 0x7ffff7ffc000 0x2000 0x0 [vdso]
0x7ffff7ffc000 0x7ffff7ffd000 0x1000 0x22000 /lib/x86_64-linux-gnu/ld-2.19.so
0x7ffff7ffd000 0x7ffff7ffe000 0x1000 0x23000 /lib/x86_64-linux-gnu/ld-2.19.so
0x7ffff7ffe000 0x7ffff7fff000 0x1000 0x0
0x7ffffffdd000 0x7ffffffff000 0x22000 0x0 [stack]
0xffffffffff600000 0xffffffffff601000 0x1000 0x0 [vsyscall]
Now I have two questions, first: getconf PAGESIZE on my machine returns 4096 which is equal to 0x1000, some of these memory spaces have different sizes though. how is that possible? are these spaces memory pages or just logical spaces? if these are not memory pages, how can I view the addresses of memory pages, or even directly dump memory pages to files?
my second question is the following: these addresses are supposed to be virtual addresses viewed by the program (not physical addresses), so why doesn't the program space start at 0? if I try to dump the memory starting from address 0 I get the following error: Cannot access memory at address 0x0. also why are there some regions in between these memory spaces that cannot be accessed (the region after the heap for example)? shouldn't the virtual space of a process be contiguous?
some of these memory spaces have different sizes though. how is that possible?
Easy: they span multiple pages (note that all of their sizes are multiples of 0x1000).
are these spaces memory pages or just logical spaces?
They are spans of one or more pages that have the same underlying mapping (the same file) and the same protections. I am not sure what exactly you call "logical spaces", but chances are you can call them that.
these addresses are supposed to be virtual addresses viewed by the program (not physical addresses),
Correct.
so why doesn't the program space start at 0?
Because long time ago VAX machines used to map something at address 0, and that made finding NULL pointer dereferences hard (they didn't crash). This was decided to be a bad idea, so later UNIX variants do not map zero page, and any attempt to dereference a NULL pointer causes SIGSEGV, helping you to debug your programs.
I have a dedicated server that is only doing ASTERISK and nothing else, however, when asterisk runs my cpu goes max immediately, but memory still the same. Is there anyway I can make asterisk use some memory instead of CPU?
Here's what I see:
top - 08:40:36 up 26 days, 15:39, 2 users, load average: 2.16, 2.16, 2.38
Tasks: 137 total, 3 running, 132 sleeping, 2 stopped, 0 zombie
Cpu(s): 99.7%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3922684k total, 1643672k used, 2279012k free, 123968k buffers
Swap: 4063224k total, 31788k used, 4031436k free, 847928k cached
You have rewrite your control application to use less i/o and cpu, but most likly it will nto help you much. Asterisk is network/cpu appliacation.
To see what is taking up cpu use
ps xau | sort -n -k3
To see what is taking up memory use
ps xau | sort -n -k3
For my trimmed down vps I did the following, this is for a vps not a server so some services you might need.
see
http://forums.asterisk.org/viewtopic.php?f=13&t=87399