It is already the second time that I notice one of my Go program taking alot of memory (much more that I would expect) and that I do not understand why, so here I am.
I decided to profile the memory with pprof and the result of the top5 pprof memory profiling is as such:
1140.28MB of 1169.97MB total (97.46%)
Dropped 61 nodes (cum <= 5.85MB)
Showing top 5 nodes out of 15 (cum >= 33.89MB)
My problem is the following. In the profile we see that the program consumed roughly 1.2GB of memory (which is affordable for what I am doing, parsing and indexing logs). However, when I do a "top command" and look at the resident memory used by my program, it is rather around 10GB to 11GB of memory taken by the program which is quite a huge difference with the memory profiling.
So where are those Gigs of memory that I don't see in the profile ?
And why ?
How to troubleshout ?
Thanks by advance,
It's likely that extra memory usage is from the file system, esp. since you are probably scanning over a lot of disk.
See: http://www.linuxatemyram.com/
I am using the "mark generation" button of the allocation instrument.
Every snapshot generation adds about 200 KB to the growth column.
I can't find anything suspicious so could it be that the growth is due to the system libs or maybe some core data caches?
It's possible that system libraries generates some new MBs to the system heap. What is more, they should be doing this! And you cannot do anything abut that...
You could try looking at this tutorial which I was inspecting a few days ago to give myself a easy-start with Instruments.
I've just been analyzing an ipad app I'm developing using Instruments. In particular I was interested in the memory usage, as I have been receiving some memory warnings.
First of all the Activity monitor reports overall some 40MB of memory used just after starting the application. This really seams like a lot to me. Especially as after the startup nothing really fancy is going on.
So I have been analyzing the app in the VM tracker.
First of all can somebody explain how to interpret the dirty memory? I mean the ipad doesn't really have virtual memory, in the sense that there is no swapping etc.
Ok the really weird thing is that I have some 40 MB of dirty memory, that is resident! Some 38MB are listed under IOKit. Under IOKit there is no further information, what that actually means.
So what exactly does IOKit do?
What could be causing this insane those huge values?
Any kind of hint is appreciated! :)
Try Heapshot Analysis, bbum has a great tutorial here.
Basically you take a Heapshot, run some procedure, take another Heapshot for several iterations. This will help find memory that lost but not a leak. I use this method often,
I have used Heapshot many times to great advantage, many thanks to bum.
What is dirty memory?
According to this session.
memory written by an app
all heap allocations
decoded image buffers
VM profile shows some info of the dirty memory
like dirty memory size. They are anonymous.
vmmap --summary App.memgraph
In this session, this Apple dev uses heap to get more info about the object sizes.
heap App.memgraph
One old application started to consume memory a lot after server update. Memory usage seems to rise with out limit until program hangs.
According to FastMM4 and EurekaLog, there's no memory leak (except 28 bytes), so I assume all memory is freed when application is shutdown.
Are there any tools or strategies suitable for tracking this kind of memory problem?
Since September 2012, there is a very simple and comfortable way to find this type of "run-time only" memory leaks.
FastMM4991 introduced a new method, LogMemoryManagerStateToFile:
Added the LogMemoryManagerStateToFile call. This call logs a summary of
the memory manager state to file: The total allocated memory, overhead,
efficiency, and a breakdown of allocated memory by class and string type.
This call may be useful to catch objects that do not necessarily leak, but
do linger longer than they should.
To discover the leak at run time, you only need these steps
add a call to LogMemoryManagerStateToFile('memory.log', '') in a place where it will be called in intervals
run the application
open the log file with a tail program (for example BareTail), which will auto-refresh when the file content changes
watch the first lines of the file, they will contain the memory allocations which occupy the highest amount of memory
if you see a class or memory type constantly has a growing number of instances, this can be the reason of your leak
The growing memory consumption is an application issue. It is not a bug, which can discover FastMM4 or EurekaLog. As from they point of view - application just correctly uses the memory.
Using AQTime, MemProof (hard to find, D7 is last supported version (?)), SleuthQA (similar to MemProof) or similar memory profilers, you can track the memory usage outside of application in real-time.
Using FastMM4, GetMemoryManagerState / GetMemoryManagerUsageSummary you can track memory usage from application. Output this information into trace file and analyze it after run. Or make simple wrapping function for one of the above procedures, which will return curent memory usage. And call it from IDE Debugger Evalute / Modify, add to Watches or call OutputDebugString, and see the current memory usage.
Note, if memory is eated by some DLL then you may not see her memory usage using (3). Use (2).
Analyzing the memory usage and the tasks performed by the application, you may discover what leads to raised memory usage.
AQTime (a commercial tool which is quite expensive) can report your memory usage, down to the line of source code that allocated each object. In the case of very large memory usage scenarios, you might want the AQTime functionality that can show the number of objects and the size (total plus individual instance size) for each object. AQTime worked great for me, starting with Delphi 7, and all later versions, including your version (2006) and the latest versions (XE and XE2).
As the program memory usage grows, AQTime can be used to grab "snapshots" of the runtime heap, you can use to understand memory usage of your application; What is being created, and how many of each object exists. Even when no leaks exist, understanding the runtime behaviour of your application in terms of the objects it creates and manages, is very important, and AQTime is the most powerful tool I know of for Delphi users.
If you are willing to upgrade to Delphi XE/XE2, you might have an included light version of AQTime already, if so, check it out. If not, I recommend you try their demo. I am unaware of any free or open source alternatives that can provide the same functionality.
Lesser functionality could be cobbled together manually by writing lots of trace messages, or using the FastMM full-debug-mode. If you could write a complete dump of your memory usage into a very large file, you might be able to write some tools to parse, and create a summary. The problem I have with FastMM in this case, is that you will be drowned in detail information, without the ability to extract exactly the summary information that helps you understand your situation. So, you can try to write your own tool to summarize the memory usage. In one application I had that used a series of components that I knew would use a lot of memory, I wrote a dialog box into my application that showed current memory usage by these large memory-blob-of-data objects.
Have you ever think about the Leak that is causing the IDE... it is so huge!!!
In my case (2GB of RAM) i do the next...
1. Open the IDE
2. Leave it minimized for near six hours
3. See how Physical memory is getting used
The result:
While IDE is oppened (remember i also do the test having it minimized) it is getting more and more RAM... till no more ram free.
It gets all 2GB RAM + all Pagefile hard disk space (i have it configured to a mas of 4GB)
In less that six hours (doing nothing on IDE) it tries to use more than 6GB.
That is called a Memory Leak casused by the IDE... i do not type any letter on IDE, do not compile anything, do not even open any project... just open IDE and minimize it... leave the computer without doing anything on it for about six hours and IDE is consuming 6GB of memory.
Of course, after that, the IDE start with annoying messages of SystemOutOfMemory... and i must kill it... then all that 6GB are freed!!!
When on the hell will this get fixed?
Please note i have all patches applied, i also tested without applying each patch/hotfix, etc...
The best i got was dissabling some options on Tools, like the one that underlines bad code, etc... so why on the hell that option has any influence... i am not typing anything on the IDE (on the tests)... and if i have it dissabled the memory leak gets reduced a lot...
Of course, if i use the IDE (write code on an opened project) without even compiling / running it... the thing goes much more worst... memory leak upto 6GB can got reached on less than an hour, sometimes occurs after 15 minutes of Copy/Paste source code.
Seems there will not be a solution in a short time!!!
So i got the next solution that works perfect:
-Close the IDE an reopen it each 15 minutes or less
Ugly solution, i know... but works!!!
I am developing a application which receives 600-700 KB of XML data from the server. I have to do some manipulations in that data so once received the data the memory increases to 600 KB to 2 M.B. Already view occupied 4 M.B of memory in the application.
So while processing the XML data i m doing some manipulation(pre-parsing) and the memory increases to 600 K.B to 2 M.B and finally decreases to 600 K.B. due to increase in memory, application gives the memory warning. While getting memory warning i m releasing all the views in the navigation controller but it releases only 1 M.B of memory. Even though I release all the views the application is crashing.
Please help me out in this issue. It happens in iPad 4.2.
Thanks in advance
There's no magical answer here. You're using too much memory and you need to figure out how to use less. Without knowing more about your application it's difficult to be specific, though clearly loading in nearly 1Mb of data and playing around with it isn't helping.
Maybe you can stream the data rather than loading it all into memory? There's an open source library that helps: StreamingXMLParser.
Also, your view sounds huge (over a megabyte!). I'm sure there's some optimisation that can be performed there. Use Instruments to see where your memory is being used.
Maybe only 1MB is released because of a parameter value which can be altered or you may need to manually start a garbage collection operation during your development session, if relevant to the language in use. You could sectionalise the xml input if possible or you could invoke [your own] compact or compress of the xml when stored if you have access to the script or code in a way that allows it.