I have a program that runs fine on MacOS and Linux and cross-compiles to Windows with mingw. Recently I made the program multi-threaded.
The current design of the program has memory allocated in the main thread and freed in the slave "worker" threads. That's not a problem on MacOS and Linux because the malloc/free system is multi-threaded.
I'm concerned about the cross-compiling, however. The version of mingw that I'm using is built from MacOS ports. It's a pretty ancient version of G++ (version 3.4.5) from 2004. I've been unsuccessful in my attempts to build a more recent version (I'd like to build a 64-bit version, but gave up). I'm getting pthreads from http://sourceware.org/pthreads-win32.
My concern is that the malloc & free system in 3.4.5 is not multi-threaded.
Questions:
Should I rewrite my program so that the blocks of memory to be freed are passed back to the main thread to be freed there?
Should I try to upgrade to a more recent mingw?
Is there any way to find these concurrency problems other than massive amounts of testing? That just doesn't feel good to me.
Thanks!
Why do you say malloc & free are not multithreaded?
mingw32 by default will link with msvcrt.dll which is a multithread dll. See [1]. There was[2] a single-threaded library provided by Microsoft, but it was only available for static linking.
PS: You mention that you are cross-compiling but you seem instead to be compiling the windows program in windows. In such case, Why don't you dowload the binaries from www.mingw.org? (it's a pain to figure out in their downloads the files needed, though)
1- http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.71%29.aspx
2- See [1]. Removed in Visual Studio 2005 http:// msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.80%29.aspx
I would avoid this. It sounds like you're trying to dodge the main issue.
Yes, that would be a good idea in any case...
One way to detect concurrency problems related to memory allocation/deallocation is a memory leak detector. I'm not sure if valgrind works on cygwin.
Related
I'm hunting an elusive memory problem in a Delphi 5 program, where memory gets randomly overwritten at the customer site. After trying a lot of things with no result so far I now want to use the FastMM4 output from the LogAllocatedBlocksToFile() to find out which objects are allocated immediately before the overwritten area. The program uses a timer to write allocated block information to a new file every 30 minutes. Unfortunately my test run of the program (DEBUG build) crashed after about 23 hours with an EOutOfMemory exception, using allocated memory of 1.83 GB according to MadExcept.
From SysInternals Process Explorer it does look like each call of LogAllocatedBlocksToFile() allocates but does not free memory:
The red spikes in the CPU Usage graph are the LogAllocatedBlocksToFile() calls. I have added calls to LogMemoryManagerStateToFile() immediately before and after, and the data for the last spike (increse of the private bytes from about 183 MB to about 218 MB) looks like this:
55054K Allocated
47911K Overhead
53% Efficiency
and this:
55055K Allocated
47910K Overhead
53% Efficiency
so FastMM4 seems not to be aware of the additional memory the program consumes according to Process Explorer.
I'm using version 4.991 of FastMM4, downloaded today from SourceForge. The test program runs in DEBUG mode, with the following defines set:
FullDebugMode
UseCustomFixedSizeMoveRoutines
UseCustomVariableSizeMoveRoutines
NoDebugInfo
ASMVersion
DetectMMOperationsAfterUninstall
RawStackTraces
LogErrorsToFile
LogMemoryLeakDetailToFile
AlwaysAllocateTopDown
SuppressFreeMemErrorsInsideException
EnableMemoryLeakReporting
HideExpectedLeaksRegisteredByPointer
RequireDebuggerPresenceForLeakReporting
EnableMMX
ForceMMX
EnableBackwardCompatibleMMSharing
UseOutputDebugString
Questions:
Is there any known problem with those functions? Am I not using them properly, are they not intended to be called multiple times in one debugging session? Is there a way to get that memory released again?
Short version:
I have tracked this down to be a version mismatch of the support library FastMM_FullDebugMode.dll.
An older version of the library works with the newer version compiled into the executable. There seems to be no check that versions do match. However, modules don't really work together at run-time.
Long version:
The project originally uses the older version 4.97 of FastMM4, which I have checked in here together with the support library (file version 1.44.0.4, product version 1.42).
While trying to find the bug in the program I have upgraded FastMM4 to version 4.991. I also remember to have copied the new support library (file version 1.61.0.6, product version 1.60) to the build directory. However, some time later I must have deleted it from the directory, or I copied it into the wrong directory to begin with, because two hours ago I checked the modules loaded by the application and found that the app had picked up the old version of the support library from another directory, as it was not in the build directory.
Since copying it there and restarting the app the problem seems to be gone. Memory usage doesn't increase when LogAllocatedBlocksToFile() is called.
Maybe this helps someone, so I answer this instead of deleting the question.
On with debugging...
How can I build and compile my own Lua files on Windows? And make them executable.
I am reading Beginning Lua programming, and I have Windows 7 and MacOS Lion both installed. I am having the hard time to follow the instructions. They do not work for me.
On MacOS I open the terminal and put these in:
export LUA_DIR=/usr/local/lib/lua/5.1
mkdir -p /usr/local/lib/lua/5.1 (it tells me, mkdir: illegal option) and I can not follow from here
SET LUA_DIR=”c:\program files\lua\5.1”
As for Windows I do this according to the book.
This what I see in my shell c:\Users\bd>
mkdir "c:\program files\utility" and it tells me access is denied
I have tried to right click on this folder and check off read only, but it does not work.
Any clues would be appreciated, this part has been really confusing for me.
To package your Lua files into an executable on Windows you have several options. There is srlua, there is wxLuaFreeze from wxLua (available as a binary for Windows), and there are more options in this SO answer.
Essentially, the main two options are: (1) append your Lua code to a precompiled exe file, such that it will be loaded and executed when that exe file is run, and (2) convert your Lua code into real executable by compiling it to bytecode, then to C, and then to your target platform.
As to your MacOS issue, mkdir -p means that mkdir is asked to create intermediate directories (for example, you asked to create /a/b/c, it will also create /a/b if those don't exist). As you don't say which version of MacOS you run, it's difficult to provide more detailed answer.
For now the standard distribution of Lua does not compile a script to native executable code; it execute your scripts by first compiling it to bytecode, then by interpreting the bytecode with a reasonnably fast static interpret (this also means that it is easily portable across native or virtual systems, and very resistant to attacks (that could be targetting bugs in the native compiler itself).
Also Lua still does not feature a runtime JIT compiler like Java and .Net: Lua still does not features a VM to produce a safe sandbox.
There exists Lua packages that convert your bytecode (or directly a source script) to a C source that can be used to convert a Lua library into native mode via the same C compiler used to compile the Lua engine itself (this is how the builtin libraries are produced, though they are slightly optimized manually in some time-critical parts).
However it is possible to compile Lua to a javascript source, and run it with fast performance using Javascript, because today's Javascript interprets do have good performance with their implemented VM featuring a JIT compiler for their own bytecodes.
It is also possible by converting it the Lua bytecode to a .Net or Java source that can then be executed directly from Lua (for that you need a version of Lua that has been ported to .Net or Java or Javascript, something that is not so complicate than developing in C/C++ directly a VM with a JIT compiler (a moderately complex part is the bytecode verifier, but the really complex part is the memory manager its garbage collector and its sandbox so that your Lua script will be fully isolated from the Lua engine itself for itw own memory, but the most complex part if the runtime optimizer and collection of profiling statistics: this has been done in the modern VMs for Java, .Net, Javascript, PHP/Zend, Python, Perl...).
I dont know which other language VM would offer the best performance to port Lua and implement on it a compiler to their own bytecode running at near native speed in their VM. But my own small experience with programs (in a much simpler language) self-generating a bytecode that they can run themselves, has always shown me Java winning in performance over .Net and Javascript. This is most probably because Java features an profiling-based dynamic code optimizer
(On the opposite the .Net optimizer runs only once during program installation, using some profiling data collected during the installation of the .Net VM itself, or at first instanciation of the script, without really knowing any profiling data collected during execution of the compiled program itself, and based on some cheked assumptions about the platform capabilities).
I also don't if would be faster in PHP, Python or Perl; the comparison with newer Javascript engines was never attempted though. Porting/compiling a Lua program to Javascript is relatively easy because it implements closures relatively easy for the resolution of linkages. Then the generated Javascript will compile to native code with the excellent Javascript's JIT compilers we have today (and never cease to improve in performance, so much that I've seen various appliactions running now faster in Javascript than before when they were written in C++ or plain C; as well the memory footprint has largely been reduced, we no longer have memory leaks, and even if there's a garbage collector, today's Javascript VM have a very efficient one, which is even better than the GC implemented in the native Lua).
But Lua remains useful as it is easy to secure and sandbox and offers various security benefits (but there are security issues in Lua as well for some kinds of applications, where Javascript offers some solutions, notably for side-channel attacks based on variation of time of execution; but these side-channel attacks are very hard to solve and can affect any system, any program, any programming language, and this starts becoming a critical issue because they are now more esily exploitable; the reason of that comes from hardware optimizations that we depend more and more today when we want to maximize the performances). And with Lua you may be more immune to these problems that a sandboxing sofware environment cannot solve alone.
Probably later we'll see a true VM implementation of Lua with a JIT and self-generating code and the possibility to instanciate new sandboxed VMs to run their self-generated code. It will take more time to generate an EXE file for distribution; notably because it generally requires adding also an installer and a distribution manager.
So for now we could imagine distributing Lua applications compiled to the bytecode of another JIT-capable VM: this generated bytecode would be faster than the Lua bytecode, and would then be extremely complex to reverse-engineer to the semantics of Lua because it would require two separate reverse engineering first from the bytecode of the other VM to the bytecode of Lua, both bytecodes loosing some easiy inferable rules and options tested and foll, and then again to sme Lua source
For the OSX terminal issue:
This command should work
export LUA_DIR=/usr/local/lib/lua/5.1
This command will probably give you permission problems:
mkdir -p /usr/local/lib/lua/5.1
You may try this to solve that. You will be prompted for your password:
sudo mkdir -p /usr/local/lib/lua/5.1
This command has nothing to do with OSX and will not work. This is a windows command:
SET LUA_DIR=”c:\program files\lua\5.1”
You have a permissions problem with Windows- try creating your cmd or PowerShell in Administrator mode. C:\Program Files is a protected directory that a regular user account doesn't have permission to write to.
As for the OS X issue, check out the mkdir OS X manual page to make sure you have the command correct.
So, if I understood your question correctly, you are trying to build Lua on Windows.
This is of course possible, but not easy for beginners. I would highly recommend you to use a binary distribution, which is much easier to install, unless you have special requirements.
Here are several Windows distributions :
Lua Binaries (Lua 5.1 and 5.2)
LuaForWindows (Lua 5.1)
LuaDist (Lua 5.2)
I develop with Delphi 2010 in a VM, the app runs well inside the VM in any ways within the ide, alone with or without Eurekalog using mainly DevExpress grids and AidAim SQLMemtables.
When the same App is ran on the host (ie directly on windows) the eurekalog finds memory leaks on the SQLMemtable dataset component and sometimes but often enough to make it impossible to release it crashes on a simple Dataset.refresh. It is very hard to debug because in the debug environemnt it works.
Anyone would have any idea what is going on, I am at loss of ideas here
Thank you for your help
Regards
Philippe Watel
Your external dependencies (dlls) differ on both machines? Also consider using Delphi's remote debugger and log more details to find out more details about your problem.
Users have been reporting problems/crashes/bugs that I can't reproduce on my machine. I'm finding these problems difficult to fix.
I've started using EurekaLog (fantastic!) and SmartInspect. Both these tools have helped greatly but I'm still finding it difficult to catch some problems.
I've just purchased Debugging by David Agans (and waiting for it to arrive).
Are there any other tools or techniques specific to Delphi that will help with catching these hard to find remote problems? The kinds of problems I'm finding difficult to track down are those that don't raise exceptions or have a clear cause. EurekaLog catches exceptions and SmartInspect is pretty good once I have a theory to check. But in some cases it is a seemingly random crash and there are several thousand lines of code that may may be at fault. How to narrow down to the root cause?
MadExcept is what I use, and it is fabulous. I have also used EurekaLog and find the functionality almost exactly identical, except that I have more experience and time using MadExcept. it's free for non-commercial use, and reasonably priced for commercial use.
Update: MadExcept 4 is now out and even supports 64 bit Delphi XE2 apps, and has memory-leak checking too.
When nothing blows up, I rely on heavy use of trace logging. I have a TraceMessage(integer,string) function which I call throughout all my apps, and when someone has problems I get them to click a menu item that turns up the debug trace level to the most verbose level; it gives me a complete history of everything my application did, and this has helped me even more than madExcept, to solve problems at customer sites. Customers get a crash, and that crash report sent by madexcept contains a log file (created by my app) that is attached automatically. I believe you can do this equally well with madExcept and EurekaLog. If you need a logging system you could download Log4D, or you could write your own, it's pretty simple.
For always-free, try JclDebug, which requires more work to set up, but which has worked fabulously for me, also.
For help with heap problems, learn more about fastMM (full version) debug options.
And you shouldn't forget that Delphi itself supports Remote debugging, if you can reproduce a crash on machines in your office that don't have delphi installed, use remote debug across the office network instead of installing a complete RAD Studio installation on that other machine at your work. You could also use remote debug to connect to a client PC computer across the internet, but I have not tried remote debug across the internet yet, so I can't say whether it works great over the internet or not. I do know that since remote debug doesn't support automatic deploy of the EXE file you built (you have to do that part yourself), remote debug over internet, to a client PC is more work.
You might also find lots of your problems by fixing all your hints and warnings, and then going through with CodeHealer or Pascal Analyzer (PAL) from Peganza. These static analysis tools can help you find real code problems.
If performance and memory usage are your problems, get the full version of AQTime, and use it to profile and watch your system operate. It will help you fix your memory leaks, and understand your app's runtime behaviour and memory usage, not just leaks but bottlenecks for memory and CPU usage. Some of those bottlenecks can also be the source of some odd problems. I have even used AQTime to help me find deadlocks, since it can generate traces of execution, that can help me figure out what code is running, and locate deadlocks. Update: AQTime is not installable on machines other than your main dev machine, without violating the newly modified license terms for AQTime. These terms were never this restrictive in the good old days.
If you gave more exact idea of what your problems are, I'm sure other people could give you some more ideas that are specific, but all of the above are general techniques that have served me well.
One of the best way is to use the Remote Debugger that comes with Delphi, so you can debug directly the application running on the remote machine. THe remote debugger is somewhat buggy in some Delphi releases, and requires to follow the instructions carefully to make it working, but when needed it's a tool to consider. Also check if there are updates available for your version, they could come in a separate installer for deployment on "remote" systems. Otherwise first install the remote debugger, than check if the files installed has newer versions in your local installation, and the copy tehm on the remote machine.
CodeSite has helped me a lot in these situations. Since XE it is bundled with Delphi.
Logging is the key, in this matter.
Take a look at our TSynLog class available in our Open Source SynCommons library.
It does have the JCL Debug / MadExcept features, with some additional (like customer-side profiling, and logging):
logging with a set of levels;
fast, low execution overhead;
can load .map file symbols to be used in logging;
compression of .map into binary .mab (900 KB -> 70 KB);
inclusion of the .map/.mab into the .exe;
reading of an external .map to add unit names and line numbers to a log file without .map available information at execution;
exception logging (Delphi or low-level exceptions) with unit names and line numbers;
optional stack trace with units and line numbers;
methods or procedure recursive tracing, with Enter and auto-Leave using interfaces;
high resolution time stamps, for customer-side profiling of the application execution;
set / enumerates / TList / TPersistent / TObjectList / TContainer / dynamic array JSON serialization;
per-thread or global logging;
multiple log files on the same process;
integrated log archival (in zip or any other format);
Open Source, works from Delphi 5 up to XE.
Is there a tool equivalent to a bounds checker or purify or valgrind for CUDA?
I'm basically looking for something that might tell me if I'm reading or writing outside of allocated memory.
NVIDIA have released CUDA memcheck which does exactly this. It's available in the 3.0 beta toolkit, you'll need to be a registered developer to download it. In addition, NVIDIA have also release Nexus which is their debugger/profiler for Visual Studio 2008 (Vista/7/2008) and includes memory checking (see the features list).
If you compile in emulation mode, you can use Valgrind itself to detect memory access problems in your kernels.