I am getting an OutOfMemoryError. How do I solve this?
Error: java.lang.OutOfMemoryError: Java heap space
You need to either increase the available heap space (with the Java -Xmx flag) or use less memory in your application.
I'd recommend that you try to use less memory. There are plenty of good profiling tools out there that you can use to discover where your code uses a lot of memory. It's also worth checking that you are not misusing any HashMaps or ObjectOutputStreams. Those two classes are notorious for soaking up memory if not used properly.
Related
This is a follow-up to this question: What could explain the difference in memory usage reported by FastMM or GetProcessMemoryInfo?
My Delphi XE application is using a very large amount of memory which sometimes lead to an out of memory exception. I'm trying to understand why and what is causing this memory usage and while FastMM is reporting low memory usage, when requesting for TProcessMemoryCounters.PageFileUsage I can clearly see that a lot of memory is used by the application.
I would like to understand what is causing this problem and would like some advise on how to handle it:
Is there a way to know what is contained in that memory and where it has been allocated ?
Is there some tool to track down memory usage by line/procedure in a Delphi application ?
Any general advise on how to handle such a problem ?
EDIT 1 : Here are two screenshots of FastMMUsageTracker indicating that memory has been allocate by the system.
Before process starts:
After process ends:
Legend: Light red is FastMM allocated and dark gray is system allocated.
I'd like to understand what is causing the system to use that much memory. Probably by understanding what is contained in that memory or what line of code or procedure did cause that allocation.
EDIT 2 : I'd rather not use the full version of AQTime for multiple reasons:
I'm using multiple virtual machines for development and their licensing system is a PITA (I'm already a registered user of TestComplete)
LITE version doesn't provide enough information and I won't waste money without making certain the FULL version will give me valuable information
Any other suggestions ?
Another problem might be heap fragmentation. This means you have enough memory free, but all the free blocks are to small. You might see it visually by using the source version of FastMM and use the FastMMUsageTracker.pas as suggested here.
You need a profiler, but even that won't be enough in lots of places and cases. Also, in your case, you would need the full featured AQTime, not the lite version that comes with Delphi XE and XE2. (AQTIME is extremely expensive, and annoyingly node-locked, so don't think I'm a shill for SmartBear software.)
The thing is that people often mistake AQTime Allocation Profiler as only a way to find leaks. It can also tell you where your memory goes, at least within the limits of the tool. While running, and consuming lots of memory, I click Run -> Get Results.
Here is one of my applications being profile in AQTime with its Allocation Profiler showing exactly what class is allocating how many instances on the heap and how much memory those use. Since you report low Delphi heap usage with FastMM, that tells me that most of AQTime's ability to analyze by delphi class name will also be useless to you. However by using AQTime's events and triggers, you might be able to figure out what areas of your application are causing you a "memory usage expense" and when those occur, what the expense is. AQTime's real-time instrumentation may be sufficient to help you narrow down the cause even though it might not find for you what function call is causing the most memory usage automatically.
The column names include "Object Name" which includes things like this:
* All delphi classes, and their instance count and heap usage.
* Virtual Memory blocks allocated via Win32 calls.
It can detect Delphi and C/C++ library allocations on the heap, and can see certain Windows-API level memory allocations.
Note the live count of objects, the amount of memory from the heap that is used.
I usually try to figure out the memory cost of a particular operation by measuring heap memory use before, and just after, some expensive operation, but before the cleanup (freeing) of the memory from that expensive operation. I can set event points inside AQTime and when a particular method gets hit or a flag gets turned on by me, I can measure before, and after values, and then compare them.
FastMM alone can not even detect a non-delphi allocation or an allocation from a heap that is not being managed by FastMM. AQTime is not limited in that way.
I'm developing a logger/sniffer using Delphi. During operation I get hugh amounts of data, that can accumulate during stress operations to around 3 GB of data.
On certain computers when we get to those levels the application stops functioning and sometimes throws exceptions.
Currently I'm using GetMem function to allocate the pointer to each message.
Is there a better way to allocate the memory so I could minimize the chances for failure? Keep in mind that I can't limit the size to a hard limit.
What do you think about using HeapAlloc, VirtualAlloc or maybe even mapped files? Which would be better?
Thank you.
Your fundamental problem is the hard address space limit of 4GB for 32 bit processes. Since you are hitting problems at 3GB I can only presume that you are using /LARGEADDRESSAWARE running on 64 bit Windows or 32 bit Windows with the /3GB boot switch.
I think you have a few options, including but not limited to the following:
Use less memory. Perhaps you can process in smaller chunks or push some of the memory to disk.
Use 64 bit Delphi (just released) or FreePascal. This relieves you of the address space constraint but constrains you to 64 bit versions of Windows.
Use memory mapped files. On a machine with a lot of memory this is a way of getting access to the OS memory cache. Memory mapped files are not for the faint hearted.
I can't advise definitively on a solution since I don't know your architecture but in my experience, reducing your memory footprint is often the best solution.
Using a different allocator is likely to make little difference. Yes it is true that there are low-fragmentation allocators but they surely won't really solve your problem. All they could do would be make it slightly less likely to arise.
I'm finding AQTime hard to use because it interferes with the original program too much. If I have a program that uses, for example, 300MB of ram I can use AQTime's allocation profiler without a problem, and find out where most of the memory is being used. However I notice that running under AQTime, the original program uses more like 1GB while it's being profiled.
Right now I'm trying to reduce memory usage in a program which is using 1.4GB of memory. If I run it under AQTime, then the original program uses all of the 2GB address space and crashes. I can of course invent a smaller set of test data and estimate how the memory usage will scale with the full data set - but the reason I'm using a profiler in the first place is to try to avoid this sort of guesswork.
I already have AQTime set to 'Collect stack information - None' and all the check boxes to do with checking memory integrity are switched off, and I've tried restricting the area being profiled to just a few classes but this doesn't seem to improve anything. Is there a way to use AQTime that produces a smaller overhead? Or failing that, what other approaches are there to get a good idea of the memory being used?
The app is written in Delphi 2010 and I'm using AQTime 6.
NB: On top of the increased memory usage, running under AQTime slows the app down an awful lot, making the whole exercise not just impossible but impractical too :-P
AFAIK the allocation profiler will track memory block allocation regardless of profiling areas. Profiling areas are used to track classes instantiation. Of course memory-profiling an application that allocates a large amount of memory is a issue, you may try to use the LARGE_ADRESS_AWARE flag, and the /3GB boot switch, or use a 64 bit system (as long as you have at least 4GB of memory, or more). Also you can take snapshot of the application state before it crashes, to see where the memory is allocated. Profiling takes time, anyway, you may have to let it run for a while.
I mostly work on C language for my work. I have faced many issues and spent lot time in debugging issues related to dynamically allocated memory corruption/overwriting. Like malloc(A) A bytes but use write more than A bytes. Towards that i was trying to read few things when i read about :-
1.) An approach wherein one allocates more memory than what is needed. And write some known value/pattern in that extra locations. Then during program execution that pattern should be untouched, else it indicated memory corruption/overwriting. But how does this approach work. Does it mean for every write to that pointer which is allocated using malloc() i should be doing a memory read of the additional sentinel pattern and read for its sanity? That would make my whole program very slow.
And to say that we can remove these checks from the release version of the code, is also not fruitful as memory related issues can happen more in 'real scenario'. So can we handle this?
2.) I heard that there is something called HEAP WALKER, which enables programs to detect memory related issues? How can one enable this.
thank you.
-AD.
If you're working under Linux or OSX, have a look at Valgrind (free, available on OSX via Macports). For Windows, we're using Rational PurifyPlus (needs a license).
You can also have a look at Dmalloc or even at Paul Nettle's memory manager which helps tracking memory allocation related bugs.
If you're on Mac OS X, there's an awesome library called libgmalloc. libgmalloc places each memory allocation on a separate page. Any memory access/write beyond the page will immediately trigger a bus error. Note however that running your program with libgmalloc will likely result in a significant slowdown.
Memory guards can catch some heap corruption. It is slower (especially deallocations) but it's just for debug purposes and your release build would not include this.
Heap walking is platform specific, but not necessarily too useful. The simplest check is simply to wrap your allocations and log them to a file with the LINE and FILE information for your debug mode, and most any leaks will be apparent very quickly when you exit the program and numbers don't tally up.
Search google for LINE and I am sure lots of results will show up.
Is the a VS2005 C++ compiler flag like the Xmx???M java flag so I can limit the heap size of my application running on Windows.
I need to limit the heap size so I can fill the memory to find out the current free memory. (The code also runs on an embedded system where this is the best method to get the memory usage)
You can set the heap size for your program by setting the size in:
Linker -> System -> Heap Reserve Size
It can also be set at the compiler command line using /HEAP:reserve
You might want to look into whether the gflags utility (in the Windows Debugging Tools) can do this. It can do a lot of other interesting things with the heap of native applications.
The heap size depends on the allocator used. There might also be some Windows API call that limits the amount of memory a process can allocate, but I'm not aware of one and I don't feel like looking for it right now, sorry. But in general, if you write your own allocator (maybe just wrap around the compiler-provided malloc() or new operator) you can artificially limit the heap size that way.
Alternatively, if you have your own allocator, even if just a wrapper, you can keep track of how much memory has been allocated in total. If you know the amount available you can just do some subtraction and be done with getting the total. You might also be able to get fragmentation statistics then, like largest free block.