Running cvs up -Ad, I'm getting the following error thrown:
cvs [update aborted]: out of memory; can not allocate 333685120 bytes
But upon investigation, (running top), there appears to be enough free memory after all.
Mem: 1025528k total, 521660k used, 503868k free, 48736k buffers
Does anyone know of any common CVS issues that might cause this error to be thrown as something of a red herring? If memory isn't the real problem, any ideas as to what I can do to find out what is?
There are at least 3 solutions for this bug:
A) Uninstall CVSNT, then install CVSNT-x64. Because this bug is already fixed in 64-bit version of CVSNT.
B) Fix this bug in your currently installed (32-bit) cvs.exe binary by using editbin.exe tool from Visual Studio distribution. Use this command:
editbin.exe /LARGEADDRESSAWARE cvs.exe
C) If you can't reinstall or modify your CVSNT, then as a temporary solution, try updating separate sub-folders and/or separate files in the folder you are trying to update, one-by-one, to make CVSNT allocate less memory.
Related
My Jenkins server keeps crashing, so I generated a heap dump which I then put through visualVM. It shows most of the memory is being used up by the class java.util.concurrent.concurrenhashmapnode.
My understanding is loads objects are being referenced, which are unable to be GC'd. As a result, most the memory is being used up by this. Any idea how to resolve this? I'm new to system admin stuff, so not the most technically proficient sorry.
TIA
I've recently came across OutOfMemoryError which crashed my Jenkins every 2 days. It was due to Ldap bug in old version of java: Ldap Error and java fixed versions matrix
In my case updating java fixed the problem.
Anyway to investigate OutOfMemoryError I did as follows:
restarted jenkins after crash,
took incremental thread dumps every half an hour, can be taken from <jenkinsUrl>/threadDump,
comparing thread dumps pointed me to memory leak on ldap threads.
In general I'd also suggest to:
update: java, Jenkins and it's plugins, other problematic tools,
investigate Jenkins logs, dumps, profile heap (what you already did).
I am running plink software through a PBS batch job. This error occurs when I run the job:
*** glibc detected *** /software/plink: double free or corruption (out): 0x000000018dfafca0 ***
======= Backtrace: =========
[0x7d7691]
[0x7d8bea]
[0x45f5ed]
[0x47bb11]
[0x40669a]
[0x7bdb2c]
[0x400209]
However it only occur with one of my files (bw 30-60 gb files) and each rerun shows the exact same back trace map. I tried running it not through the batch scheduler and received the same error again, with the same backtrace map. I am just using the software (plink), and didn't write it, so most of the answers online are about writing and freeing memory in your program.
Any ideas on
what is causing this error, and
how I can fix it?
what is causing this error, and
A double-free or heap corruption in the plink
how I can fix it?
You can't. You can do one of two things, depending on how much you know and understand.
First, build the newest version of plink from source, and see if the problem persists.
If it does not, you are done (or at least you might hope that someone else found and fixed this problem).
If it does, you'll have to debug the problem sufficiently for either you, or plink developers to fix it. Some tools that should help: Valgrind and Address Sanitizer (note: in addition to Clang, Address Sanitizer is also included in GCC-4.8).
Once you have a good report (where the memory was allocated, and where it got corrupted), you should either fix it and submit your fix to plink developers, or give them a bug report with the allocation and corruption location and stack traces.
I'm hunting an elusive memory problem in a Delphi 5 program, where memory gets randomly overwritten at the customer site. After trying a lot of things with no result so far I now want to use the FastMM4 output from the LogAllocatedBlocksToFile() to find out which objects are allocated immediately before the overwritten area. The program uses a timer to write allocated block information to a new file every 30 minutes. Unfortunately my test run of the program (DEBUG build) crashed after about 23 hours with an EOutOfMemory exception, using allocated memory of 1.83 GB according to MadExcept.
From SysInternals Process Explorer it does look like each call of LogAllocatedBlocksToFile() allocates but does not free memory:
The red spikes in the CPU Usage graph are the LogAllocatedBlocksToFile() calls. I have added calls to LogMemoryManagerStateToFile() immediately before and after, and the data for the last spike (increse of the private bytes from about 183 MB to about 218 MB) looks like this:
55054K Allocated
47911K Overhead
53% Efficiency
and this:
55055K Allocated
47910K Overhead
53% Efficiency
so FastMM4 seems not to be aware of the additional memory the program consumes according to Process Explorer.
I'm using version 4.991 of FastMM4, downloaded today from SourceForge. The test program runs in DEBUG mode, with the following defines set:
FullDebugMode
UseCustomFixedSizeMoveRoutines
UseCustomVariableSizeMoveRoutines
NoDebugInfo
ASMVersion
DetectMMOperationsAfterUninstall
RawStackTraces
LogErrorsToFile
LogMemoryLeakDetailToFile
AlwaysAllocateTopDown
SuppressFreeMemErrorsInsideException
EnableMemoryLeakReporting
HideExpectedLeaksRegisteredByPointer
RequireDebuggerPresenceForLeakReporting
EnableMMX
ForceMMX
EnableBackwardCompatibleMMSharing
UseOutputDebugString
Questions:
Is there any known problem with those functions? Am I not using them properly, are they not intended to be called multiple times in one debugging session? Is there a way to get that memory released again?
Short version:
I have tracked this down to be a version mismatch of the support library FastMM_FullDebugMode.dll.
An older version of the library works with the newer version compiled into the executable. There seems to be no check that versions do match. However, modules don't really work together at run-time.
Long version:
The project originally uses the older version 4.97 of FastMM4, which I have checked in here together with the support library (file version 1.44.0.4, product version 1.42).
While trying to find the bug in the program I have upgraded FastMM4 to version 4.991. I also remember to have copied the new support library (file version 1.61.0.6, product version 1.60) to the build directory. However, some time later I must have deleted it from the directory, or I copied it into the wrong directory to begin with, because two hours ago I checked the modules loaded by the application and found that the app had picked up the old version of the support library from another directory, as it was not in the build directory.
Since copying it there and restarting the app the problem seems to be gone. Memory usage doesn't increase when LogAllocatedBlocksToFile() is called.
Maybe this helps someone, so I answer this instead of deleting the question.
On with debugging...
Users have been reporting problems/crashes/bugs that I can't reproduce on my machine. I'm finding these problems difficult to fix.
I've started using EurekaLog (fantastic!) and SmartInspect. Both these tools have helped greatly but I'm still finding it difficult to catch some problems.
I've just purchased Debugging by David Agans (and waiting for it to arrive).
Are there any other tools or techniques specific to Delphi that will help with catching these hard to find remote problems? The kinds of problems I'm finding difficult to track down are those that don't raise exceptions or have a clear cause. EurekaLog catches exceptions and SmartInspect is pretty good once I have a theory to check. But in some cases it is a seemingly random crash and there are several thousand lines of code that may may be at fault. How to narrow down to the root cause?
MadExcept is what I use, and it is fabulous. I have also used EurekaLog and find the functionality almost exactly identical, except that I have more experience and time using MadExcept. it's free for non-commercial use, and reasonably priced for commercial use.
Update: MadExcept 4 is now out and even supports 64 bit Delphi XE2 apps, and has memory-leak checking too.
When nothing blows up, I rely on heavy use of trace logging. I have a TraceMessage(integer,string) function which I call throughout all my apps, and when someone has problems I get them to click a menu item that turns up the debug trace level to the most verbose level; it gives me a complete history of everything my application did, and this has helped me even more than madExcept, to solve problems at customer sites. Customers get a crash, and that crash report sent by madexcept contains a log file (created by my app) that is attached automatically. I believe you can do this equally well with madExcept and EurekaLog. If you need a logging system you could download Log4D, or you could write your own, it's pretty simple.
For always-free, try JclDebug, which requires more work to set up, but which has worked fabulously for me, also.
For help with heap problems, learn more about fastMM (full version) debug options.
And you shouldn't forget that Delphi itself supports Remote debugging, if you can reproduce a crash on machines in your office that don't have delphi installed, use remote debug across the office network instead of installing a complete RAD Studio installation on that other machine at your work. You could also use remote debug to connect to a client PC computer across the internet, but I have not tried remote debug across the internet yet, so I can't say whether it works great over the internet or not. I do know that since remote debug doesn't support automatic deploy of the EXE file you built (you have to do that part yourself), remote debug over internet, to a client PC is more work.
You might also find lots of your problems by fixing all your hints and warnings, and then going through with CodeHealer or Pascal Analyzer (PAL) from Peganza. These static analysis tools can help you find real code problems.
If performance and memory usage are your problems, get the full version of AQTime, and use it to profile and watch your system operate. It will help you fix your memory leaks, and understand your app's runtime behaviour and memory usage, not just leaks but bottlenecks for memory and CPU usage. Some of those bottlenecks can also be the source of some odd problems. I have even used AQTime to help me find deadlocks, since it can generate traces of execution, that can help me figure out what code is running, and locate deadlocks. Update: AQTime is not installable on machines other than your main dev machine, without violating the newly modified license terms for AQTime. These terms were never this restrictive in the good old days.
If you gave more exact idea of what your problems are, I'm sure other people could give you some more ideas that are specific, but all of the above are general techniques that have served me well.
One of the best way is to use the Remote Debugger that comes with Delphi, so you can debug directly the application running on the remote machine. THe remote debugger is somewhat buggy in some Delphi releases, and requires to follow the instructions carefully to make it working, but when needed it's a tool to consider. Also check if there are updates available for your version, they could come in a separate installer for deployment on "remote" systems. Otherwise first install the remote debugger, than check if the files installed has newer versions in your local installation, and the copy tehm on the remote machine.
CodeSite has helped me a lot in these situations. Since XE it is bundled with Delphi.
Logging is the key, in this matter.
Take a look at our TSynLog class available in our Open Source SynCommons library.
It does have the JCL Debug / MadExcept features, with some additional (like customer-side profiling, and logging):
logging with a set of levels;
fast, low execution overhead;
can load .map file symbols to be used in logging;
compression of .map into binary .mab (900 KB -> 70 KB);
inclusion of the .map/.mab into the .exe;
reading of an external .map to add unit names and line numbers to a log file without .map available information at execution;
exception logging (Delphi or low-level exceptions) with unit names and line numbers;
optional stack trace with units and line numbers;
methods or procedure recursive tracing, with Enter and auto-Leave using interfaces;
high resolution time stamps, for customer-side profiling of the application execution;
set / enumerates / TList / TPersistent / TObjectList / TContainer / dynamic array JSON serialization;
per-thread or global logging;
multiple log files on the same process;
integrated log archival (in zip or any other format);
Open Source, works from Delphi 5 up to XE.
I have a program that runs fine on MacOS and Linux and cross-compiles to Windows with mingw. Recently I made the program multi-threaded.
The current design of the program has memory allocated in the main thread and freed in the slave "worker" threads. That's not a problem on MacOS and Linux because the malloc/free system is multi-threaded.
I'm concerned about the cross-compiling, however. The version of mingw that I'm using is built from MacOS ports. It's a pretty ancient version of G++ (version 3.4.5) from 2004. I've been unsuccessful in my attempts to build a more recent version (I'd like to build a 64-bit version, but gave up). I'm getting pthreads from http://sourceware.org/pthreads-win32.
My concern is that the malloc & free system in 3.4.5 is not multi-threaded.
Questions:
Should I rewrite my program so that the blocks of memory to be freed are passed back to the main thread to be freed there?
Should I try to upgrade to a more recent mingw?
Is there any way to find these concurrency problems other than massive amounts of testing? That just doesn't feel good to me.
Thanks!
Why do you say malloc & free are not multithreaded?
mingw32 by default will link with msvcrt.dll which is a multithread dll. See [1]. There was[2] a single-threaded library provided by Microsoft, but it was only available for static linking.
PS: You mention that you are cross-compiling but you seem instead to be compiling the windows program in windows. In such case, Why don't you dowload the binaries from www.mingw.org? (it's a pain to figure out in their downloads the files needed, though)
1- http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.71%29.aspx
2- See [1]. Removed in Visual Studio 2005 http:// msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.80%29.aspx
I would avoid this. It sounds like you're trying to dodge the main issue.
Yes, that would be a good idea in any case...
One way to detect concurrency problems related to memory allocation/deallocation is a memory leak detector. I'm not sure if valgrind works on cygwin.