I am using Lucene.Net-2.3.2.1 in my project. My project also supporting multithreading environment. Lucene Indexing service is working as Windows Service. Problem is when the service is running, it's memory blockage is gradually increasing. So after some hours, it shows a memory of 150 mb in Task Manager where as it start with 13 mb.so it has a memory increasing behavior. I identified by dotTrace Profiler that in Lucene.Net there are some methods and objects that increased the memory. From Call Tree one of my dotTrace out identify that Index(), Segment() related functions hold's memory increased as long as the service perform. So it at a time, it will crash the system.
Please help me how i can recover my application from this memory leakage.
Increasing memory usage doesn't necessarily implies a memory leak. Memory leaks in .NET are not that common, but there are a few options you should check
Events. Make sure that all event listeners are detached from the publisher as soon as they are no longer used. Failing to do so will keep the listeners alive as long as the publisher is alive.
If the code uses any disposable resource that holds handles to native code, be sure to call Dispose on these as soon as they are no longer needed.
A blocking finalizer will prevent other finalizable objects from being garbage collected, so make sure finalizers don't do any more than they have to (and in many cases they are probably not needed anyway).
If you want to examine which objects are being kept alive as well as why they are not collected, I recommend using WinDbg + Sos.
Related
Let's say that I have an OS that implements malloc by storing a list of segments that the process points to in a process control block. I grab my memory from a free list and give it to the process.
If that process dies, I simply remove the reference to the segment from the process control block, and move the segment back to my free list.
Is it possible to create an idempotent function that does this process cleanup? How is it possible to create a function such that it can be called again, regardless of whether it was called many times before or if previous calls died in the middle of executing the cleanup function? It seems to me that you can't execute two move commands atomically.
How do modern OS's implement the magic involved in culling memory from processes that randomly die? How do they implement it so that it's okay for even the process performing the cull to randomly die, or is this a false assumption that I made?
I'll assume your question boils down to how the OS culls a process's memory if that process crashes.
Although I'm self educated in these matters, I'll give you two ways an OS can make sure any memory used by a process is reclaimed if the process crashes.
In a typical modern CPU and modern OS with virtual memory:
You have two layers of allocation. Whenever the process calls malloc, malloc tries to satisfy the request from already available memory pages the kernel gave the process. If not enough pages are available, malloc asks the kernel to allocate more pages.
In this case, whenever a process crashes or even if it exits normally, the kernel doesn't care what malloc did, or what memory the process forgot to release. It only needs to free all the pages it gave the process.
In a simpler OS that doesn't care much about performance, memory fragmentation or virtual memory and maybe not even about memory protection:
Malloc/free is implemented completely on the kernel side (e.g: system calls). Whenever a process calls malloc/free, the kernel does all the work, and therefore knows about all the memory that needs to be freed. Once the process crashes or exits, the kernel can cleanup. Since the kernel is never supposed to crash, and keep a record of all the allocated memory per process, it's trivial.
Like I said, I'm self educated, and I didn't check how for example Linux or Windows implement it.
We have some problems with Dart. It seems like after some period of time the garbage collector can't clear the memory in VM, so application hangs. Anyone with this issue? Are there any memory limits?
You should reuse your objects instead of creating new ones. You should use pool pattern:
http://en.wikipedia.org/wiki/Object_pool_pattern
Be careful about canvas and it's proper destruction.
Another GC performance papers:
http://blog.tojicode.com/2012/03/javascript-memory-optimization-and.html
http://qt-project.org/doc/qt-5/qtquick-performance.html
Are there any memory limits?
Yes. Dart apparently runs with a maximum sizes that can be configured at launch time:
How to run a dart program with big memory?
(The following applies to all garbage-collected languages ...)
If your application starts to run out of space (i.e. the heap is slowly filing with objects that the GC can't remove) then you may get into a nasty situation where the GC runs more and more frequently, and manages to reclaim less and less memory each time. Eventually you run out of memory, but before that happens the application gets really slow.
The solution is typically to do one or both of the following:
Find what is causing the memory to run out. It is typically not that you are allocating too many objects. Rather, the typical cause is that the unwanted objects are all still reachable ... via some data structure that your application has built.
Set the "quick death" tuning option for the GC .... if available. For example, Java garbage collectors can be configured to measure the time spent garbage collecting. (The GC overhead.) When the GC overhead exceeds a preset ratio, the Java virtual machine throws an OutOfMemoryError to "pull the plug".
We have a multi-threaded client - server project, we recently upgraded the server side applications on 64 - bit architecture. Solved many problems, our application now works steadily week under a heavy load. But after this period, the application on the server crashes with "Out of memory" error. At this time, free memory is available in large quantities, it seems that the problem of memory fragmentation. Is there any possibility to defragment memory, some tools? Or may be some other reasons "Out of memory" in a similar situation?
Memory allocation:
Total amount of memory : 96Gb
Physical : 48Gb
Virtual : 48Gb
The amount of free physical memory at the time of the collapse : 3GB
The amount of free virtual memory at the time of collapse : 45Gb
Maximum size of memory allocated per thread : 1GB
As you have surmised, your problem is fragmentation. There's nothing you can do to defragment memory--any tool that would attempt to do so would have to have a complete map of every pointer in your program. Note that even .NET's garbage collector can't accomplish this with the large object heap, I have crashed a 32-bit net application with only 100mb actually used.
Instead, what you need to do is avoid the fragmentation in the first place. Generally that means object pools, saving old objects for reuse rather than freeing them and later reallocating them.
Another option (if the service isn't time critical and needs to be online 100% of every second) is to restart your service every 24 hours or so (either via the task scheduler or from within your own program).
If from within your own program, you can do it in one of two ways, depending on what type of service you have (if there can exist two instances of your service at the very same - brief - time):
1) Execute a second instance of your service from within your currently running service and then terminate
2) Execute a tiny helper program that waits f.ex. 5 seconds and then (re)starts your service, then terminate your currently running service
But the best way would be to avoid the fragmentation in the first place, as Loren Pechtel writes.
I am trying to debug memory usage in a large application using Delphi 7. I was able to installed fastmm debug full dll and with it solve some leak problems.
I also installed the memory usage tracker, allowing me to see which blocks were allocated and of what size they are.
My question is, is there a way to find out where the blocks were allocated? I know it is possible because if the memory wasn't freed a stack trace gets printed. Is there a way to 'poke' at fastmm to get it to print the stack trace for a given allocation?
Side question: if the start address of an allocation is known, is there a way to find out which class the object is? (assuming that the allocation was for a object.
You can either:
try to use LogAllocatedBlocksToFile procedure. If its ALastAllocationGroupToLog param is less than AFirstAllocationGroupToLog or is zero, then all blocks along with their allocation call stacks are logged. However, if your app has many memory allocations, prepare to long waiting. I experienced about 4 hrs wait time and 1.5Gb resulting file. (Side-note: use glogg to view such large files)
modify FastMM4.pas so implementation's LogCallStack will be visible in interface. Or you can try to use it directly from FastMM_FullDebugMode.dll
On the side question: try to use DetectClassInstance function.
If you use FullDebugMode and enable the conditionals that will write the data to a file, then you should get exactly what you're asking for. It will write out a stack trace for every leaked allocation when the program shuts down. (If you're debugging a program with a lot of memory leaks, this can take a while. I've seem it make shutdowns last for 10 minutes or more if the leaking item is a container that holds lots of other objects.)
Considering you said in a comment that the app's memory gets cleaned up nicely on app's closing for me sounds you're looking for logical memory leaks - in other words: objects that are alive more time than needed but when the time arrive for finish the app they are cleaned because exist code to clean them.
Example:
Create an TForm using the Application as owner and the variable that references it is the global one that Delphi creates when create the form's unit.
Config the Form's CloseAction to caHide (in the OnClose event)
Show the form, operate on it
Close the form and never more use it until the app closes
Close the app, which makes Application clear all the objects it owns
So you have an logical memory leak but not an physical memory leak - which is the kind that FastMM can easily detect. Since you not intended that our hypothetical TForm live until the application finish, it semantically leaked but since it is referenced and there is code that destroys it at the end of application, to FastMM is a normal allocation.
That said appears you need not the memory dump of the memory manager, but an memory profiler.
I'm using NSOperationQueue to manage a phase of an iOS application which is quite long so I would like to manage it asynchronously. Inside that phase I allocate big arrays in C by using directly calloc functions.
With big I mean a 1024x256 bidimensional array of floats and similar things.
If everything resides on the main thread than the app locks up while computing but everything goes fine, if, instead, I move the heavy part to a NSInvocationOperation then I got many strange results, with debugger sometimes I get a strange message in console stating
No memory available to program now: unsafe to call malloc
so I was wondering if threads managed by an operation queue have some different restrictions compared to main thread, and in case what is better to do to get around this issue.
There's no restrictions that I know of.. however, you may be hitting the edge of available RAM. Since iOS doesn't do virtual memory, when memory gets low, it'll send a warning to other apps to free up RAM. That may be the source of your issue.
Use instruments to profile how much RAM you're using. If it's more than about 20MB or so, you're probably in danger of being terminated due to excessive memory usage anyway.