I want try to understand better the problem of synchronization of shared memory. I have understood that interprocess synchronization can work differently on different operating system. The most difference is what's happening when a process that has locked the shared memory crash. Windows free locked named mutex after a process crash whereas linux don't free it. Can someone explain me better the problem and which are the vantages and disadvantages? How is possible under linux free a named mutex or a interprocess semaphore after a process crash? I have searched on internet but I didn't find someone that explain good the problems and the solutions.
I hope somebody can help me.
Sorry for my English.
The advantage of Windows is that the waiting thread is freed to continue. The disadvantage is that it has no idea what the state of the shared memory is—the crashed process may have been part way through updates. (Windows indicates this by the wait on the mutex returning WAIT_ABANDONED rather than WAIT_OBJECT_0 (or offsets from these if waiting on multiple objects).
In practice the only safe thing to do is to reset the shared memory in some way (assuming that can be done meaningfully) or to fail.
Related
From my research I understand that if two processes communicate through shared memory then if the shared segment gets damaged, both processes will most likely be affected.
What I would like to know is whether or not a damaged process has the ability to corrupt a healthy process memory just by passing a bad file descriptor or sending a corrupted message through IPC methods like unix sockets or dbus. In case it matters, I am asking about corruption due to programming errors and not purposeful exploitation.
I apologize if my question is too broad and I presume that the answer will be obvious to an experienced programmer, however this is something that has been bugging me for some time and it is very hard to find a satisfying answer on the web.
Definitely yes.
If the receiving process is assumes a certain memory layout and you corrupt it, it can easily behave wrong.
Say for instance I write a program which allocates a bunch of large objects when it is initialized. Then the program runs for awhile, perhaps indefinitely, and when it's time to terminate, each of the large initialized objects are freed.
So my question is, will it take longer to manually deallocate each block of memory separately at the end of the program's life or would it be better to let the system unload the program and deallocate all of the virtual memory given to the program by the system at the same time.
Would it be safe and/or faster? Also, if it is safe, does the compiler do this when set to optimise anyway?
1) Not all systems will free a memory for you when application terminates. Of course most of the modern desktop systems will do this, so if you are going to run your program only on Linux or Mac(or Windows), you can leave the deallocation to the system.
2) Often it is needed to make some operations with the data on termination, not just to free the memory. So if you are going to develop such program design that makes it hard to deallocate objects at the end manually, then it can happen that later you will need to perform some code before exiting and you will face up with hard problem.
2') Sometimes even if you think that your program will need some objects all the way until dead, later you may want to make a library from you program or change a project to load and unload you big objects and the poor design of your program will make this hard or impossible.
3) Moreover, the program deallocation performance depends on the implementation of the allocator you are going to use in your program. The system deallocation depends on the system memory management and even for a single system there can be several implementations. So if you face with allocation/deallocation performance problems - you would like to develop better allocator rather then hope on the system.
4) So my opinion is: When you deallocate memory manually at the end - you are always on a right way. When you don't do this, perhaps you can get some ambiguous benefits in several cases, but likely you will just face with the problems sooner or later.
Well most OS will free the memory at exit if the program, but the bigger question is why would you want it to have to?
Is it faster? Hard to say with memory sometimes. I would guess not really and definitely not worth breaking good coding practices anyway.
Is it safe? Define safe... Will your OS crash? Probably not. Will your code be susceptible to memory leaks or other problems? Absolutely, it will. In fact you are basically telling it you want memory leaks.
Best practice is to always free your memory when you are done with it. With C and C++, every malloced or new block of memory should have a corresponding free or delete.
It is a bad idea to rely on the OS to free your memory because it not only makes your code look bad and makes it less portable, but if the program was ever integrated into another program, then you will likely be tracking down memory leaks for hours.
So, short answer, always do it manually.
Programs with a short maintenance life time are good candidates for memory deallocation by "exit() and let the kernel sort them out." However, if the program will last more than a few months you have to consider the maintenance burden.
For instance, consider that someone may realize that a subsequent stage is required in the program, and some of the data is not needed, or not needed in memory. They now have to go and find out how to deallocate the memory, properly removing stale references, etc.
I use a lot of components in my Delphi 7 Service application, Indy, Synapse, Zeolibs, etc.
My application is generally stable, I use Eurekalog 6 to capture exceptions, but in rare situations, some threads hang because a 3rd party function it calls has hung, e.g. Indy gets stuck when trying to send email.
In many cases, the application that hung are my customer place, I've no access to their computer, so it is not possible for me to do a live debug. My application requires high availability so even if it hangs once a year, that is not acceptable to my users.
I am now looking for the best way to deal with such a situation where debugging is not feasible but I will still need the application recover by itself. Is it possible for a thread to terminate if a function it calls hangs? Alternatively, I can also restart the entire service when that happens. How about a Watchdog and what is the best way to implement it? Thanks.
I think you are being rather defeatist. Find and fix the bugs. It might be tricky, but it's the right solution.
Killing threads whose behviour you don't understand is never the solution. If you start killing threads you'll likely make things worse. That can lead to other runtime errors, deadlock and so on. Once you start killing threads you've lost control.
Now, it would be safe to kill the process (rather than a specific thread) and rely on a watchdog service to restart the process. But that's a really dire solution.
You should certainly use a tool like madExcept, EurekaLog etc. to debug unexpected exceptions. I see you are already using EurekaLog - that's good.
Deadlocks (it sounds like you have deadlock) can be more tricky to chase down. One good way to debug a deadlock is to get your client to produce a crash dump (e.g. from Process Explorer). Then debug it in WinDbg using map2dbg to produce symbolic stack traces. That will tell you which threads are blocking and that reveals the deadlock. And then fix the bugs.
For more details on this deadlock debugging technique see here: http://capnbry.net/blog/?p=18
I'm not familiar with EurekaLog since I use madExcept, but I would expect EurekaLog has a facility to allow generation of thread stack traces for a hung process. If so then that would most likely be the best approach for you.
Your question is rather too vague. If you don't know which of the various components you're using you wish to blame, then you have zero hope of fixing it. The most likely thing is you're doing something wrong, or that you don't understand how these components work. I very much doubt that it's purely a bug in the components themselves, but hey, either way it's all on you to find what's having a problem, and your job to fix it.
A deadlock that you've created, or a deep process corruption issue, that is happening, may prevent MadExcept from giving you any information, but it's worth trying.
To find out which one is freezing, if any at all, then the madexcept comment is the best suggestion yet. It will time-out (after a configurable # of seconds) and raise an artificial exception for you, interrupting your hung process. This works for user code, and for places where the thread is blocked in a Win32 or kernel function. For example, it's possible that you've set up Indy for infinite timeouts, as that's the default these days in Indy 10, and that what you're experiencing is a timeout related freeze, where network activity that you expected to complete but which never will complete, is causing your program to "hang". The cure here is to change your timeouts.
However, until you figure out WHERE the problem is, I doubt you'll be able to fix it. And so, for that, again, Marcus is right, you should be looking into madExcept. I can't live without it.
Secondly, you should really be adding trace logic to your program, so you know where it's going and what it was doing just before it had a problem. If you really need help doing that, you could try CodeSite, from Raize. Personally I find that OutputDebugString combined with the free Microsoft DebugView utility (formerly from SysInternals) tool is more than enough to debug such problems on a client computer.
Any program with background threads that does not have trace logging, is a badly designed program. Heck, any non-trivial single threaded application that might ever fail or have problems, needs trace logging.
Logging is always going to help, even when MadExcept or other exception tools don't. Trace-Logging is usually a roll-your-own solution, although CodeSite is also quite popular.
I found an article on About.com that tells you how you can manage your apps memory.
Here is the code:
procedure TrimAppMemorySize;
var
MainHandle : THandle;
begin
try
MainHandle := OpenProcess(PROCESS_ALL_ACCESS, false, GetCurrentProcessID) ;
SetProcessWorkingSetSize(MainHandle, $FFFFFFFF, $FFFFFFFF) ;
CloseHandle(MainHandle) ;
Log('Trimmed Memory Successfull!');
except
Log('Failed to trim Memory!');
end;
Application.ProcessMessages;
end;
I tried it out, works perfectly - Even when my app is doing something, and I fire buttonclicks, etc, it still does its thing, and it works like a charm. I look at my apps Memory usage in the Resource Monitor, and as far as I can see, its all good.
So.. Whats the catch? We all deal with memory issues, but is the solution really that simple? Can anyone tell me if doing this every 60 seconds is a bad thing?
I will reboot and try to run my program, and post a screenshot of my Resource Monitor.
Yes, it's a bad thing. You're telling the OS that you know more about memory management than it does, which probably isn't true. You're telling to to page all your inactive memory to disk. It obeys. The moment you touch any of that memory again, the OS has to page it back into RAM. You're forcing disk I/O that you don't actually know you need.
If the OS needs more free RAM, it can figure out which memory hasn't been used lately and page it out. That might be from your program, or it might be from some other program. But if the OS doesn't need more free RAM, then you've just forced a bunch of disk I/O that nobody asked for.
If you have memory that you know you don't need anymore, free it. Don't just page it to disk. If you have memory that the OS thinks you don't need, it will page it for you automatically as the need arises.
Also, it's usually unwise to call Application.ProcessMessages unless you know there are messages that your main thread needs to process that it wouldn't otherwise process by itself. The application automatically processes messages when there's nothing else to do, so if you have nothing to do, just let the application run itself.
The "catch" as it were is that you have just told the operating system to remove pages from your working set that are actually in RAM. Assuming the OS is removing pages that don't have data you will ever access again, there's no problem. But if it's paging out data that your process will need in the future, you've just told Windows "More page faults, please."
The main issue with this code is that you're essentially sacrificing your own process's performance for the sake of the rest of the system (though thrashing actually harms the whole system.) That's somewhat noble, but clearly not "catch" free.
This is the moral equivalent of pretending to the operating system that your machine is an permanent state of RAM crisis. The system knows how to manage its memory far better than you do, just let it get on with its job.
It is, sadly, a very common mistake for people to worry when their system is using all of its RAM and all of its CPU. In reality you should be concerned if your system fails to make full use of its resources!
Actually when using GlobalAlloc/GlobalFree Windows does tend to keep those memory blocks attached to your process, I have had process "appear" to be using 400MB of memory when it was only using 40MB because GlobalFrees do not really release the memory back (we are talking along lived process running in the background as long as the machine is running). In this case I found it very useful to have the ability to tell Windows compact the process memory. I do use the GetProcessMemoryInfo and check the current .WorkingSetSize, if larger than a certain amount (like 100MB) the memory is compacted. Yes, this does incur page faults for memory being actively used but memory is freed back to the kernel for use by other processes.
So, in some cases I found that Windows does not do a good job "garbage collecting" and returning resources. Glad this call is available, it is very useful.
I believe that you can't force a running Memcached instance to de-allocate memory, short of terminating that Memcached instance (and freeing all of the memory it held). Does anyone know of a definitive piece of documentation, or even a mailing list or blog posting from a reliable source, that can confirm or deny this impression?
As I understand it, a Memcached process initially allocates a chunk of memory (the exact initial allocation size is configurable), and then monotonically increases its memory utilization over its lifetime, limited by the daemon's maximum memory allocation size (also configurable). At no point does the Memcached daemon ever free any memory, regardless of whether the daemon has any ongoing need for the memory it holds.
I know that this question might sound a little whiny, with a tone of "I DEMAND that open source project X support my specific need!" That's not it, at all--I'm purely interested in the exact technical answer, here, and I swear I'm not harshing on Memcached. For the curious, this question came out of a discussion about possible methods for gracefully juggling multiple Memcached instances on a single server, given an application where the cost of a cache flush can be quite high.
However, I'd appreciate it if you save your application suggestions/advice for a different question (re-architecting my application, using a different caching implementation, etc.). I do appreciate a good brainstorm, but I think this question will be most valuable if it stays focused on the technical specifics of how Memcached does and does not work. If you don't have the answer to this specific question, there is probably still value in what you have to say, but I'd guess that there's a different, better place to post the more speculative comments/suggestions/advice.
This is probably the hardest problem we have to solve for memcached currently (well, a variation of it, anyway).
Freeing a chunk of memory requires us to know that a) nothing within the chunk is in use and b) nothing will start using it while we're in the process of purging it for reuse/freeing. I've heard some really good ideas for how we might solve our slab rebalancing problems which is basically the same, except we're not trying to free the memory, but to give it to something else (a common problem in a few large installations).
Also, whether free actually reduces the RSS of your process is implementation dependent. In many cases, a malloc/fill/free will leave the memory mapped in (unless your allocator uses mmap instead of sbrk).
I'm pretty sure this isn't possible with memcached. I don't see any technical reason why it couldn't be implemented though. Lock cache operations, expire enough keys to reach the desired size, update the size, unlock. (I'm sure there's nicer ways to avoid blocking the server during that time.)
The standard and default mechanism of memory management in memcached is slab allocator. It means that memory is being allocated for the process and never released to the operating system. Basically, when memory is no longer used to store some data, it is being held by the process in order to be reused later, when needed. However, the operating system releases memory allocated by the process when it is finished. That is why memory is being released when you kill/stop the memcached.
There is a compile-time option in memcached to enable malloc/free mechanism. So that when free() is called, memory might be released to operating system (this depends on C standard library implementation). But doing so might hurt a good fragmentation and performance.
Please read more about the issue here:
Why not use malloc/free
Memcached memory management