I'm working with an old and complex system that shares memory between dozens (sometimes hundreds) of Win32 processes. The code is mostly very old Pascal that has been ported to Delphi a few years ago.
(Almost) all of the code is in a single DLL, which all of the processes load. At the moment, we have forced a fixed loading address of that DLL. Image base is defined and ASLR is disabled in linker settings. Each process checks the DLL loading addresses at startup and the entire system refuses to work if the DLL cannot be loaded at the exact same address in all of the processes. This is of course a problematic solution. Sometimes customers have all sorts of 3rd party gadgets which affect the address space and prevents our product from having the address it wants for the DLL.
The reason for the fixed DLL loading address is below. I'm wondering if there is a way to work around this problem.
I've been trying to introduce object-oriented programming. The problem is, if I instantiate a Delphi class in the shared memory, that instance now seems to be dependent on the DLL loading address. For example, if another process tries to destroy that object, it will crash, unless the two processes happen to have the same DLL address. Delphi runtime seems to save function addresses in the object instance, assuming they will stay fixed for the lifetime of the object.
One possible solution might be to copy the DLL contents to the shared memory, then do some sort of magic trickery on DLL_PROCESS_ATTACH to make the process run that copy of the code instead of the loaded DLL address. The shared memory we have is always mapped at the same addresses. (Yes, this is also a problem sometimes but very rarely since the shared memory can be mapped at high (above 2 GB) addresses which are easily available.)
Or is there a way to tell Delphi compiler "please do not assume that the addresses of the functions related to this class are fixed"? I'm using Delphi 11.1.
I was able to figure out a a solution that seems to work well, so let me answer my own question.
The issue is that in order for dynamic dispatch to work, the object instance must be 'tagged' with type information. In the case of Delphi in Win32, this tag is in the first 32 bits of the object instance, and it is a memory address into the DLL where the the code of the class in question is.
If you shift this address to match the variable (process-specific) address of the DLL, the dynamically dispatched methods work fine. In order to do this, you need to compare this address to the loading address of the DLL (or any other reference address inside the DLL) and save the offset, when creating the object.
Then, before calling the object's methods in another process, "localize" the object by taking the actual address of the DLL, adding the offset, and writing this sum to the first 32 bits of the object.
Now you can use the object in any process, as long as you localize it first:
Obj.Localize;
Obj.Do_Something;
This can be neatly wrapped in a class.
Offset is simply a private 32-bit UInt32.
constructor Global_Object.Create;
begin
Self.Offset := PUInt32(Self)^ - Self.Reference_Address;
end;
procedure Global_Object.Localize;
begin
PUInt32(Self)^ := Self.Reference_Address + Self.Offset;
end;
destructor Global_Object.Destroy;
begin
inherited Destroy;
end;
function Global_Object.Reference_Address
: Cardinal;
begin
// Anything in the DLL can be used as a reference,
// such as the address of this function.
Result := Cardinal(#Global_Object.Reference_Address);
end;
Related
I need to be able to pass the same set of structures (basically arrays of different records) over two different interfaces
The first (legacy) which is working requires a pointer to a record and the record size
The second, which I am attempting to develop, is type-safe and requires individual fields to be set using Get/Set methods for each field
Existing code uses records (probably around 100 or so) with memory management being handled in a 3rd party DLL (i.e. we pass the record pointer and size to it and it deals with memory management of new records).
My original thought was to bring the memory management into my app and then copy over the data on the API call. This would be easy enough with the old interface, as I just need to be able to access SizeOf() and the pointer to the record structure held in my internal TList. The problem comes when writing the adapter for the new type-safe interface
As these records are reliant on having a known size, there is heavy use of array 0..n of char static arrays, however as soon as I try to access these via 2010-flavour RTTI I get error messages stating 'Insufficient RTTI information available to support this operation'. Standard Delphi strings work, but old short-strings don't. Unfortunately, fixing string lengths is important for the old-style interface to work properly. I've had a look at 3rd party solutions such as SuperObject and the streaming in MorMot, though they can't do anything out of the box which doesn't give me too much hope of a solution not needing significant re-work.
What I want to be able to do is something like the following (don't have access to my Delphi VM at the moment, so not perfect code, but hopefully you get the gist):
type
RTestRec = record
a : array [0..5] of char;
b : integer;
end;
// hopefully this would be handled by generic <T = record> or passing instance as a pointer
procedure PassToAPI(TypeInfo: (old or new RTTI info); instance: TestRec)
var
Field: RTTIField;
begin
for Field in TypeInfo.Fields do
begin
case Field.FieldType of
ftArray: APICallArray(Field.FieldName, Field.Value);
ftInteger: APICallInteger(Field.FieldName, Field.Value.AsInteger);
...
end;
end;
Called as:
var
MyTestRec: RTestRec;
begin
MyTestRec.a := 'TEST';
MyTestRec.b := 5;
PassToAPI(TypeInfo(TestRec), MyTestRec);
end;
Can the lack of RTTI be forced by a Compiler flag or similar (wishful thinking I feel!)
Can a mixture of old-style and new-style RTTI help?
Can I declare the arrays differently to give RTTI but still having the size constraints needed for old-style streaming?
Would moving from Records to Classes help? (I think I'd need to write my own streaming to an ArrayOfByte to handle the old interface)
Could a hacky solution using Attributes help? Maybe storing some of the missing RTTI information there? Feels like a bit of a long-term maintenance issue, though.
What is best method to free up memory when my console application terminates?
I would like to free up any related loaded DLLs and any other resources consumed by my application.
I want to free all things in memory :)
I have searched Google but could not find a way to do this.
I'm not using Units in my application it just console application
program MyAPP;
{$APPTYPE CONSOLE}
uses
Windows, SysUtils;
/// functions
// procedures
begin
end.
begin
try
// Your entire program goes here.
finally
SomeFunction;
end;
end.
When your application terminates all memory allocated by it either directly or by Delphi internal allocator (TObject.Create / InitInstance) or by modules is freed by operating system.
Even memory leaks within the application are not an issue after program was terminated. You can check for memory leaks by using FastMM4 library.
Windows should also free it's objects for which you have obtained handles by initialization calls to specialized libraries like GDIPlus yet these should be freed manually after object instance became obsolete in the local scope of the code using them.
For all in-code deallocations you should use try..finally. So your program.dpr might ideally look like:
program YourProgram;
{$APPTYPE CONSOLE}
uses
MainUnit;
var
main: TProgramMain
begin
main := TProgramMain.Create;
try
main.Execute;
finally
main.Free;
end;
end.
You have not showed what objects you created, but whatever you did to create those objects, the reverse is how you free them, if you wish to.
Note that if your objects are simply consuming memory in the heap, you could just let Windows reclaim all memory used by your process, and the side effects of this (if there are none) might include that your tiny application shuts down faster.
Nevertheless, most conscientious developers choose to fully free every object they create. If you created an object like this:
x := TMyObject.Create;
.. then you free it by calling Free:
x.Free;
If the object is a component, owned by its parent, you don't need to free it.
If the object is an interfaced object (reference counted) you simply let go of your reference like this:
x := nil;
If the type in question is a value-type (Double, Integer) then you don't have to free it.
If the type in question is automatically managed (record, string) you also don't have to free it.
If your object allocates more objects, the correct place to free them is in the destructor of that object.
These are the rules. Just follow them, and you will have no leaks. Your console application terminates in a manner you have not specified. Were you expecting some magic answer to your unspecified question? If it were me, I would ensure my application had an orderly shutdown, and that during normal shutdown, it frees its resources. If it terminates abnormally, then any "try..finally" blocks you have written will not be executed. I recommend you single step through your shutdown code, to see that it executes at all.
If you had posted some examples of your code, then more specific answers would have been possible.
We are trying to figure out if we have memory leaks in our software. So, I have been using various tools and programs to help me find possible memory leaks. One of the software I used was AQTime. As it came with Delphi XE, it was only a demo. So, I was not really able to get any useful information from it. Then, I decided to use free software, MemProof. So far, it has shown me many issues with our software that requires attention. One of which is an error.
As soon as I start my program through MemProof, it lists 2 errors, which is attempting to destroy non-existent object from the unit file, system.pas. So, when I actually put a break point within TObject.Free procedure, it breaks even before my program started all the way. Stepping through the procedure Free in system.pas, I found out that TIconimage is trying to destroy or free itself. In other word, free procedure is not invoked from within my program prior to actually starting up.
Here is the actual Free procedure:
procedure TObject.Free;
begin
if Self <> nil then
Destroy;
end;
After that observation, I removed the breakpoint and let the program run all the way. My programs main window popped up ready for user input. However, I also found out that TObject.Free procedure is invoked non-stop if any part of my program's WINDOW is displayed on the screen. I don't understand that at all. Why is that? Can anyone explain? How is TForm is related to TObject.Free in any shape or form as the procedure is constantly invoked when the TForm is displayed on the screen?
Thanks in advance.
Regarding why TObject.Free executes a lot, every single time an object is destroyed, any object, that method will be called. All classes derive from TObject, it's the common ancestor, so almost any action in a Delphi program involves large numbers of object create/destroy pairs and consequently will hit TObject.Free.
Regarding detection of memory leaks, you have all you need built in to Delphi to solve this. The FastMM memory manager can be run in "report memory leaks" mode and it will give you loads of diagnostics of any memory that you leak.
Consider the following trivial program:
program Leaker;
begin
ReportMemoryLeaksOnShutdown := True;
TObject.Create;
end.
This results in the following output:
You just need to set ReportMemoryLeaksOnShutdown to True somewhere in your app (the start of the .dpr file is as good a place as any).
If you wish to receive more information in the report then you can download the full version of FastMM and configure it to your heart's content.
Then you get output like this:
A memory block has been leaked. The size is: 84
This block was allocated by thread 0x1304, and the stack trace (return addresses) at the time was:
40455E [System][System.#GetMem]
405A2F [System][System.TObject.NewInstance]
40602E [System][System.#ClassCreate]
4474C2 [Classes][Classes.TStringList.Create]
C275A3 [Main.pas][Main][Main.TMainForm.CreateAuxiliaryForms][997]
C84C8A [OrcaFlex.dpr][OrcaFlex][OrcaFlex.OrcaFlex][351]
75E633CA [BaseThreadInitThunk]
77519ED2 [Unknown function at RtlInitializeExceptionChain]
77519EA5 [Unknown function at RtlInitializeExceptionChain]
The block is currently used for an object of class: TStringList
It's truly wonderful. It tells me that the leaking memory was allocated in Main.pas line 997, and that's precisely where I put my intentional leak!
As you know, TApplication has an Icon property which you can f.i. assign in the application settings in project options. This property is reflected with an FIcon field of TApplication which is created in the constructor of the Application object. TIcon has a TIconImage field representing the actual image which gets created in its constructor. When the Application object loads and assigns the icon from the project resource file, this initial 'TIconImage' has to be freed in order to prevent a leak. All this happens even before Application.Initialize is called in the project source, because the Application object is constructed from the initialization section of 'controls.pas'.
Lots of things are happening when an application is launching or running. When launching, the streaming mechanism creates objects (resource streams, readers, class finders, component lists ..) and then frees them. Even a blank VCL form (with no controls on it) when running, creates a list each time it gets activated to find a control to put the focus on, and then frees this list. With complex GUI applications, a variety of graphics objects can be created and freed even if you hover the mouse on something. Or the alignment/arrangement code can create/free objects even if you press your mouse on to something.
To debug leaks you can take the course outlined by David's answer, or when using a 3rd party product concentrate on what it says leaked, not on every object which gets created/freed. :)
TObject.Free will be called whenever ANY instance of a class in Delphi is Free'd.
This includes a whole host of objects that are created and destroyed simply as part of the normal execution of a Delphi program, including in response to events processed automatically by a TForm object in response to the messages generated by the system simply to maintain the window object itself in existence in the Windows own Window Manager.
For example, consider this snipped fragment of code from the TCustomForm WndProc:
WM_MEASUREITEM:
begin
:
Canvas := TControlCanvas.Create;
with Canvas do
try
:
finally
Canvas.Free;
end;
:
end;
The key here being that in response to a WM_MEASUREITEM message, a custom form (and therefore a standard TForm derived class, since this ultimately derives from TCustomForm) creates a temporary TControlCanvas, which it then Free's when it is finished with it.
This may not necessarily be the source of the TObject.Free calls that you are seeing in your particular form's case, it is just an example, but shows how a TForm merely existing can result in other objects being brought into existence and destroyed in response to automatic, system generated messages.
I'm allocating memory for several PWideChar on my main executable file
var
pwcValor: PWideChar;
begin
pwcValor := AllocMem( sizeof(WideChar) * Succ(Length(pValor)));
StringToWideChar(pValor, pwcValor, Succ(Length(pValor)));
pMetodo(pCodigo, pCodigoParametro, pwcValor);
All of these variables are sent over to an external DLL using late binding. I have some questions about this situation to avoid memory leaks.
Where (on my exe or my dll) should I call the FreeMem on these variables?
Do I need to call FreeMem on these variables?
When can I (or should I) call FreeMem on these variables?
If I call them inside the external DLL (which is also mine), I get Access Violations when I try to Unload from memory the DLL library.
Tks
EDIT
Something I forgot to ask. And the other way around? I have so return parameters from my DLL to the EXE, so the PWideChars are allocated on the DLL. So, I would have to free them on the DLL, right? But I'll probably still be using them on the EXE. Must I pre-allocate on the EXE, send to the DLL the pointer, and have it filled in the DLL in these cases? Or just make a copy on the EXE of the returned parameter, so I can free it safely on the DLL?
Ultimately that depends on the design of the DLLs you use. However, I would say that if not documented otherwise it is safe to free the resources as soon as the DLL function returns. I would even suggest that you should do it. Anyway you must do it to avoid memory leaks.
Regarding the last sentence, a DLL and the invoking EXE eventhough they both be Delphi code, they use different memory managers, so you cannot in a DLL free memory allocated in the EXE.
Regarding to freeing:
There are different possible solutions here:
Your exe can allocate buffer, which then would be filled by dll;
Your dll can export one more function, say FreeString. Exe should call it every time it has finished with the string;
You can use simple WideString type. This type of strings use system memory manager which is the same for exe and dll.
Personally I recommend you the last option.
If you have a Delphi string variable and are using D2009 or later so that PChar maps to PWideChar then you can just call your function with pMetodo(pCodigo, pCodigoParametro, PChar(pValor)). Then inside your DLL, you take a copy of the string by declaring a string variable and simply assigning to the string. For example the DLL code would look like this:
procedure pMetodo(pwcValor: PChar);
var
pValor: string;
begin
pValor := pwcValor;
...
end;
The code as you have it is quite unnecessarily complex. Doing it the way I suggest avoids having to use any explicit memory allocation routines. If you want to write C code then why use Delphi!! ;-)
If you wanted to be more explicit then you could write PWideChar rather than PChar.
I'm currently working on porting an existing Delphi 5 application to Delphi 2010.
It's a multithreaded DLL (where the threads are spawned by Outlook) that loads into Outlook. When compiled through Delphi 2010, whenever I close a form I run into an "invalid pointer operation" inside TMonitor.Destroy... the one in system.pas, that is.
As this is an existing and kinda complex application, I have a lot of directions to look into, and the delphi help doesn't even document barely documents this particular TMonitor class to begin with (I traced it to some Allen Bauer posts with additional information) ... so I figured I'd first ask around if anyone had encountered this before or had any suggestions on what could cause this problem.
For the record: I am not using the TMonitor functionality explicitly in my code, we are talking a straight port of Delphi 5 code here.
Edit Callstack at the moment the problem occurs:
System.TMonitor.Destroy
System.TObject.Free
Forms.TCustomForm.CMRelease(???)
Controls.TControl.WndProc(???)
Controls.TWinControl.WndProc((45089, 0, 0, 0, 0, 0, 0, 0, 0, 0))
Forms.TCustomForm.WndProc(???)
Controls.TWinControl.MainWndProc(???)
Classes.StdWndProc(15992630,45089,0,0)
Forms.TApplication.ProcessMessage(???)
The pointer to the System.Monitor instance of each object is stored after all the data fields. If you write too much data to the last field of an object it could happen that you write a bogus value to the address of the monitor, which would most probably lead to a crash when the destructor of the object attempts to destroy the bogus monitor. You could check for this address being nil in the BeforeDestruction method of your forms, for a straight Delphi 5 port there shouldn't be any monitors assigned. Something like
procedure TForm1.BeforeDestruction;
var
MonitorPtr: PPMonitor;
begin
MonitorPtr := PPMonitor(Integer(Self) + InstanceSize - hfFieldSize + hfMonitorOffset);
Assert(MonitorPtr^ = nil);
inherited;
end;
If this is a problem in your original code you should be able to detect it in the Delphi 5 version of your DLL by using the FastMM4 memory manager with all checks activated. OTOH this could also be caused by the size increase of character data in Unicode builds, and in that case it would only manifest in DLL builds using Delphi 2009 or 2010. It would still be a good idea to use the latest FastMM4 with all checks.
Edit:
From your stack trace it looks like the monitor is indeed assigned. To find out why I would use a data breakpoint. I haven't been able to make them work with Delphi 2009, but you can do it easily with WinDbg.
In the OnCreate handler of your form put the following:
var
MonitorPtr: PPMonitor;
begin
MonitorPtr := PPMonitor(Integer(Self) + InstanceSize - hfFieldSize + hfMonitorOffset);
MessageDlg(Format('MonitorPtr: %p', [pointer(MonitorPtr)]), mtInformation,
[mbOK], 0);
DebugBreak;
// ...
Now load WinDbg and open and run the process that calls your DLL. When the form is created a message box will show you the address of the monitor instance. Write down the address, and click OK. The debugger will come up, and you set a breakpoint on write access to that pointer, like so:
ba w4 A32D00
replacing A32D00 with the correct address from the message box. Continue the execution, and the debugger should hit the breakpoint when the monitor gets assigned. Using the various debugger views (modules, threads, stack) you may get important information about the code that writes to that address.
An invalid pointer operation means your program attempted to free a pointer, but there was one of three things wrong with it:
It was allocated by some other memory manager.
It had already been freed once before.
It had never been allocated by anything.
It's unlikely that you'd have multiple memory managers allocating TMonitor records, so I think we can rule out the first possibility.
As for the second possibility, if there's a class in your program that either doesn't have a custom destructor or that doesn't free any memory in its destructor, then the first actual memory deallocation for that object could be in TObject, where it frees the object's monitor. If you have an instance of that class and you attempt to free it twice, that problem could appear in the form of an exception in TMonitor. Look for double-free errors in your program. The debugging options in FastMM can help you with that. Also, when you get that exception, use the call stack to find out how you got to TMonitor's destructor.
If the third possibility is the cause, then you have memory corruption. If you have code that makes assumptions about the size of an object, then that could be the cause. TObject is four bytes larger as of Delphi 2009. Always use the InstanceSize method to get an object's size; don't just add up the size of all its fields or use a magic number.
You say the threads are created by Outlook. Have you set the IsMultithread global variable? Your program normally sets it to True when it creates a thread, but if you're not the one creating threads, it will remain at its default False value, which affects whether the memory manager bothers to protects its global data structures during allocation and deallocation. Set it to True in your DPR file's main program block.
After a lot of digging it turns out I was doing a nice (read: horrifying, but it has been properly doing its job in our delphi 5 apps for ages)
PClass(TForm)^ := TMyOwnClass
somewhere deep down in the bowels of our application framework. Apparently Delphi 2010 has some class initialization to initialize the "monitor field" that now didn't happen, causing the RTL to try and "free the syncobject" upon form destruction because getFieldAddress returned a non-nil value. Ugh.
The reason why we were doing this hack in the first place was because I wanted to automatically change the createParams on all form instances, to achieve an iconless resizable form. I will open up a new question on how to do this without rtl-breaking hacks (and for now will simply add a nice shiny icon to the forms).
I will mark Mghie's suggestion as the answer, because it has provided me (and anyone reading this thread) with a very large amount of insight. Thanks everyone for contributing!
There are two TMonitor in Delphi:
System.TMonitor; which is a record, and is used for thread synchronization.
Forms.TMonitor; which is a class representing an attached monitor (display device).
System.TMonitor is added to Delphi since Delphi 2009; so if you are porting a code from Delphi 5, what your code was using was Forms.TMonitor, not System.TMonitor.
I think the class name is referenced without unit name in your code, and that is making the confusion.