This is the function found in Delphi Rio in System.Net.HttpClient
THTTPClientHelper = class helper for THTTPClient
....
procedure THTTPClientHelper.SetExt(const Value);
var
{$IFDEF AUTOREFCOUNT}
LRelease: Boolean;
{$ENDIF}
LExt: THTTPClientExt;
begin
if FHTTPClientList = nil then
Exit;
TMonitor.Enter(FHTTPClientList);
try
{$IFDEF AUTOREFCOUNT}
LRelease := not FHTTPClientList.ContainsKey(Self);
{$ENDIF}
LExt := THTTPClientExt(Value);
FHTTPClientList.AddOrSetValue(Self, LExt);
{$IFDEF AUTOREFCOUNT}
if LRelease then __ObjRelease;
{$ENDIF}
finally
TMonitor.Exit(FHTTPClientList);
end;
end;
What the guy try to do with LRelease here?
{$IFDEF AUTOREFCOUNT}
LRelease := not FHTTPClientList.ContainsKey(Self);
{$ENDIF}
LExt := THTTPClientExt(Value);
FHTTPClientList.AddOrSetValue(Self, LExt);
{$IFDEF AUTOREFCOUNT}
if LRelease then __ObjRelease;
{$ENDIF}
So if FHTTPClientList doesn't contain the THTTPClient add it in the FHTTPClientList and then reduce it's refcount by one. Why reduce it's refcount by one?? the THTTPClient is still alive and used why breaking it's refcount? Their is a bug here, maybe the guy make a typo, but i don't understand what he want to do originally ...
for info this this how items are removing from the dictionary :
procedure THTTPClientHelper.RemoveExt;
begin
if FHTTPClientList = nil then
Exit;
TMonitor.Enter(FHTTPClientList);
try
FHTTPClientList.Remove(Self);
finally
TMonitor.Exit(FHTTPClientList);
end;
end;
Purpose of the manual reference counting for ARC compiler in above code is simulating dictionary with weak references. Delphi generic collections are backed with generic arrays that will hold strong reference to any object added to the collection on ARC compiler.
There is several ways to achieve weak references - using pointers, using wrappers around object where object is declared as weak and manual reference counting in appropriate places.
With pointers you lose type safety, wrappers require significantly more code, so I guess the author of above code opted for manual reference counting. Nothing wrong with that part.
However, as you noticed, there is something fishy in that code - while SetExt routine is written properly RemoveExt has a bug resulting in a crash later on.
Let's go through the code in context on ARC compiler (I will omit compiler directives and unrelated code for brevity):
Since adding object into collection (array) increases reference count, to achieve weak reference we have to decrease reference count of the added object instance - that way the instance's reference count will remain the same after it is stored in collection. Next, when we remove object from such collection, we have to restore reference count balance and increase reference count. Also we have to make sure that object will be removed from such collection before it is destroyed - good place to do that is destructor.
Adding to collection:
LRelease := not FHTTPClientList.ContainsKey(Self);
FHTTPClientList.AddOrSetValue(Self, LExt);
if LRelease then __ObjRelease;
We add the object to the collection and then after collection holds strong reference to our object we can release it's reference count. If object is already inside collection that means it's reference count was already decreased and we must not decrease it again - that is the purpose of LRelease flag.
Removing from collection:
if FHTTPClientList.ContainsKey(Self) then
begin
__ObjAddRef;
FHTTPClientList.Remove(Self);
end;
If object is in collection we have to restore the balance and increase the reference count before removing object from collection. This is the part that is missing from RemoveExt method.
Making sure that object is not in the list upon destruction:
destructor THTTPClient.Destroy;
begin
RemoveExt;
inherited;
end;
Note: In order for such faked weak collection to work properly items must be added and removed only through above methods that take care of balancing reference count. Using any other original collection methods like Clear will result in broken reference count.
Bug or not?
In System.Net.HttpClient code broken RemoveExt method is called only in destructor, also FHTTPClientList is private variable and it is not altered in any other way. On the first glimpse, that code works properly, but actually contains rather subtle bug.
To unravel the real bug we need to cover possible usage scenarios, starting with few established facts:
Only methods that alter content and by that reference count of items in FHTTPClientList dictionary are SetExt and RemoveExt methods
SetExt method is correct
Broken RemoveExt method that does not call __ObjAddRef is called only in THTTPClient destructor and this is where this subtle bug originates.
When destructor is called upon any particular object instance that means object instance has reached it's lifetime, and any subsequent reference counting triggers (during destructor execution) have no influence on the code correctness.
That is ensured by applying objDestroyingFlag on FRefCount variable changing its value and any further count increasing/decreasing can no longer result in special value 0 that starts destruction process - so object is safe and will not get destroyed twice.
In above code when THTTPClient destructor is called that means last strong reference to object instance has gone out of scope or was set to nil and at that moment the only remaining live reference that can trigger reference counting mechanism is the one in FHTTPClientList. That reference has been cleared by RemoveExt method (broken or not) at that point as previously said it does not matter. And everything works fine.
But, author of the code has forgotten one tiny weeny thingy - DisposeOf method that triggers the destructor, but at that point object instance has not reached its reference counting lifetime. In other words - if destructor is called by DisposeOf, any subsequent reference counting triggers must be balanced because there are still live references to the object that will trigger reference counting mechanism after the destructor chain calls are completed. If we break the counting at that point result will be catastrophic.
Since THTTPClient is not TComponent descendant that requires DisposeOf it is easy to make the oversight and forget that someone, somewhere could call DipsoseOf on such variable anyway - for instance if you make owned list of THTTPClient instances clearing such list will call DisposeOf on them and happily break their reference count because RemoveExt method is ultimately broken.
Conclusion: Yes, it is a BUG.
Related
I know of these three techniques:
Manually
uses
System.Classes;
procedure DoSomething;
var
sl: TStringList;
begin
sl := TStringList.Create;
try
finally
sl.Free; // Invoking destructor
end;
end;
Reference counting / interface (TInterfacedObject)
Fastest example from the top of my head in the standard library:
uses
Xml.XMLDoc, Xml.XMLIntf;
procedure DoSomething;
var
xmldoc: IXMLDocument;
begin
xmldoc := TXMLDocument.Create(nil) as IXMLDocument;
end; // No reference to xmldoc anymore, freed automatically
Ownership
Like almost the whole VCLibrary or this one:
uses
System.Generics.Collections;
procedure DoSomething;
var
ol: TObjectList<TObject>;
i: Integer;
o: TObject;
begin
ol := TObjectList<TObject>.Create(true); // the list takes ownership of the objects
try
for i := 0 to 9 do begin
o := TObject.Create;
ol.Add(o);
end;
finally
ol.Free; // does not only free the list but all objects in the list, too
end;
end;
Are there more?
When it comes to memory management models for managing object instances in Delphi, there are two: manual memory management and automatic reference counting. All object instances will be released either manually or through reference counting mechanism.
But when it comes to actual coding patterns, there is number of ways we can write the code in order to trigger the release of an object instance and it is almost impossible to list and categorize them all.
The best way to illustrate the complexity involved is by asking additional question:
What do you consider as a memory management technique?
For instance, manually releasing object instance requires invoking the destructor. But there are several commonly used ways to do so: by calling Free, Destroy, or FreeAndNil. But, Free and FreeAndNil eventually will call Destroy. So the question is, should we consider that those different methods of invoking the destructor are the same technique or different techniques? What about other custom written methods that will trigger destruction of an object instance?
When it comes to releasing reference counted object instance, your example has shown indirect way of release - just letting reference go out of scope. But there is an additional way to release such object instance, and that is by explicitly assigning nil to such reference.
procedure DoSomething;
var
xmldoc: IXMLDocument;
begin
xmldoc := TXMLDocument.Create(nil) as IXMLDocument;
...
xmldoc := nil;
...
end;
Again, the question is whether we consider those two different examples as the same or different?
When it comes to ownership, it is just a way to delegate releasing an object instance to some other entity. At the end, in case of manually managed object instances some code at some point will directly invoke destructor on such object. While this is clearly a different coding pattern than directly invoking destructor on object reference without going through additional layers of indirection, at the end the instance will be released manually.
We can also transfer ownership of reference counted object instances. If you have a collection that holds interface references, then this can also be considered as ownership transfer, as release of those instances will depend on the release of the collection itself, even though involved code will not directly call destructor, but will rely on automatic reference counting to do so.
The next question that arises is: What about fields? Your first example shows construction and destruction of local object instance. If we have an object field in a class and manually call Free to such field in its destructor, should we consider that as a manual technique or ownership transfer, because actual release of that inner object instance will depend on the release of its outer, owning object.
There is additional aspect to reference counting. While compiler automatically inserts reference counting code (calls to _AddRef and _Release methods) in appropriate places, the _Release method itself will have to directly call the destructor to actually free the instance. In a way this is just another example of ownership transfer, with some help of the compiler.
From one perspective, we can say that those three techniques you have mentioned are the (two) three basic techniques to release an object instance. But one the other hand, there is an infinite number of them.
Question
In the code below a new object of type TStringList is created and passed to a procedure which is using that object. By passing the object to the method ProcToFillStringList a new object reference is created by coping the reference. My questions regarding this code are:
What happens to the object reference stored in the parameter SList
after the method returns? Does it remove the reference to the object
from the stack?
What does the Free() method actually do internally? Does it remove all references to the object from the stack or does it remove
the object itself? Which references are removed?
Do object references (not the object itself) get removed from stack automatically when a method returns?
Would it be better to pass the reference byref?
Code
var
SL: TStringList; // first object reference
begin
SL := TStringList.Create; // creating object
try
ProcToFillStringList(SL);
finally
SL.Free; // -> what gets 'freed' here? the object? the references? both?
end;
end;
procedure ProcToFillStringList(const SList: TStrings); // second object reference
SList.Add('x'); // not calling Free -> does the reference get removed?
end;
Here is code of Free method on newer versions of Delphi:
procedure TObject.Free;
begin
// under ARC, this method isn't actually called since the compiler translates
// the call to be a mere nil assignment to the instance variable, which then calls _InstClear
{$IFNDEF AUTOREFCOUNT}
if Self <> nil then
Destroy;
{$ENDIF}
end;
There are two different cases. When compiled to environment with automatic reference counting (that is iOS), Free doesn't work at all, objects are freed only when the last reference to them is removed (but as said in comments to code above, compiler changes your SL.Free to SL:=nil, so if it was the last reference to object, it will be freed and SL is really set to nil.
But in all other platforms objects are not reference counted. When calling Free, object memory is freed, but your variable is not set automatically to nil (not saying about another variables pointing to the same object), that's just impossible with syntax like this. Any method of object can't change variable it's called from. That's why you write SL := TStringList.Create instead of SL.Create. In first case you get new memory address where object is created and assign SL to it. In second SL is not initialized and can point anywhere, so there is no way to create object exactly there.
So, to answer your questions:
Object reference in local procedure is removed when it goes beyond the scope. But if you use const or var argument, it is not created in the first place. Actually, you're using the same reference SL here.
In iOS Free does nothing, object will be destroyed automatically when SL variable goes beyond the scope. In other platforms, Free destroys object and doesn't affect other references at all.
Yes, they do.
Use that modifier which describes your situation best. Const will tell compiler and people working with your code (including yourself) that argument won't be changed in procedure, compiler may pass it by value (for objects less than pointer) or by reference, but no matter what it chooses, refcount will never be increased, so from this point of view, you can think that you use exactly the same object, like it was passed by reference.
Using Var (by reference) you can accidentally change the variable you passed to procedure and this makes your intentions unclear, so use it only when you really want to change this variable and Const otherwise.
In the documentation of embarcadero is written
System::TObject::Free automatically calls the destructor if the object reference is not nil
It means in your case that the object SL is cleared at the point you called SL.Free. A object inherited from TObject does not know how many references are alive to that instance. Only the pointer to the address of the instance of SL is passed to the function call ProcToFillStringList. The instance is not informed about the new reference.
If you want to handle reference counting have a look at TInterfacedObject and the 3 Methods
QueryInterface
_AddRef
_Release
a new object reference is created by coping the reference
New reference const SList is just non-changeable pointer to the object. It will be removed from the stack if it lives there (in this case parameter is passed through register).
Free doesn't clear any references. It just destructs an object, frees it's memory. There is 'FreeAndNil' routine that frees object and makes one reference nil. Other references still exist.
My question is about debugging memory leaks which seem to be a nightmare.
In my app there is a simple class derived from TObject. All objects of that class are stored in a collection/list of of the class derived from TObjectList:
type
TOffer = class(TObject)
Item: string;
Price: string;
Id: string;
end;
TOffers = class(TObjectList<TOffer>)
protected
procedure SetOffer(I: Integer; AOffer: TOffer);
function GetOffer(I: Integer): TOffer;
public
property Offers[I: Integer]: TOffer read GetOffer write SetOffer
end;
The usage scenario:The crawler downloads the offers, parses them and saves to objects collection. This approach seems to be quite convenient as I can refer to the objects later (fill grids/lists, write them to file, etc.)
The problem is the proper disposal of the objects to avoid memory leaks. The app allocates ~4Mb memory on start but after processing ~12k offers it devours 32Mb. The leaks caused by not properly disposed objects/variables after the process finishes.
ReportMemoryLeaksOnShutdown shows horrible digits, but the crucial is -- I have no idea where to look and how to properly debug the damn thing.
Another example is the variable var MyString: string which also needs a proper disposal!! It was sorta insight for me :) I thought each procedure/function automatically manages garbage collection of the out-of-scope variables.
The list of offers is created by a function:
function GetOffersList: TOffers;
begin
Result := TOffers.Create;
while not rs.EOF do
begin
Offer := TOffer.Create;
try
// here come collected offer attributes as variables of type string:
Order.Item := CollectedOfferItem;
Order.Price := CollectedOfferPrice;
Order.Id := CollectedOfferId;
Result.Add(Offer);
finally
Offer := nil;
end;
end;
end;
Then I address those offers directly as a collection. The key thing is that I want this app to run 24/7, so the correct resource disposal is a must.
How to properly dispose object(s) of the above types?
Shall I consider the other techniques to manage object/object lists?
How to properly dispose variables of type string?
Can you please advise the good reading on fighting memory leaks in Delphi?
Thank you.
By default, when you create an object, you become its owner. So long as you are the owner, you are responsible for freeing it. Here are some of the common patterns:
1. Local variable
For an object that is created in a method and only referred to locally, you use the try/finally pattern:
Obj := TMyClass.Create;
try
... use Obj
finally
Obj.Free;
end;
2. Object owned by another object
Commonly created in the constructor and destroyed in the destructor. Here you have a member field of the owning object that holds the reference to the owned object. All you need to do is call Free on all owned objects in the owning class destructor.
3. Owned TComponent
If a TComponent or a derived class is created with an Owner, then that owner destroys the component. You do not need to.
4. TObjectList or similar with OwnsObjects set to True
You show this pattern in your question. You create a TObjectList<T> and by default OwnsObjects is True. This means that when you add a member to the container, the container assumes ownership. From that point on the container assume responsibility for destroying its members and you do not have to. However, somebody still has to destroy the container.
5. Reference counted interfaced objects
Common examples are objects derived from TInterfacedObject. The interface reference counting manages lifetime. You don't need to destroy the object.
6. Function that creates and returns a new instance
This is towards the more tricky end of the spectrum. Thankfully it's a rather rarer pattern. The idea is that the function returns a newly instantiated and initialized object to the caller, who then assumes ownership. But while the function is still executing it is the owner and must defend against exceptions. Typically the code goes like this:
function CreateNewObject(...): TMyClass;
begin
Result := TMyClass.Create;
try
Result.Initialize(...);
except
Result.Free;
raise;
end;
end;
This has to be an exception handler with a call to Free and a re-raise because the code is not in a position to use a finally. The caller will do that:
Obj := CreateNewObject(...);
try
....
finally
Obj.Free;
end;
Looking at the code in the question, that appears to be using both items 4 and 6 from my list. However, do note that your implementation of GetOffersList is not exception safe. But there's no indication that is the problem. It seems plausible that the code that calls GetOffersList is failing to destroy up the container.
Why are you leaking strings? Well, strings are managed objects. They are referenced counted and you need to take no explicit action to destroy them. However, if they are contained in other classes, instances of which are leaked, the contained strings are also leaked. So concentrate on fixing the leaks of objects, and you'll take care of the string leaks.
For what it is worth, TOffer feels more like a value type than a reference type to me. It has no method and contains three simple scalar values. Why not make it a record and use TList<TOffer>?
So, how do you proceed? The FastMM leak report is what you need. You'll want the full FastMM rather than the cut down Embarcadero version. It will identify the allocations that were not matched with deallocations. Deal with them one by one.
In parallel with this, study good quality code. Good open source Delphi libraries will demonstrate all the patterns above, and many more. Learn from them.
String is auto-managed by the compiler, you do not need to free it manually (except in rare corner cases that do not apply to this situation). TObjectList has an OwnsObjects property that you can set to True so the list will free the objects automatically for you. Its constructor has an optional AOwnsObjects parameter to initialize the OwnsObjects property.
If I implement an interface on a form such as TMyForm = class(TForm, IMyInterface), will the object free itself when there are no more interface references to it? It seems not to, although I couldn't work out how TForm is reference counted (if at all). I'm concerned about the form getting freed when an interface reference goes out of scope, but this does not seem to happen.
I guess there are two parts to the question, firstly whether a form might get unexpectedly freed (the real question), and secondly how forms are reference counted.
TForm derives from TComponent, which implements _AddRef() and _Release() to disable reference counting on itself. As such, any interface implemented by any TComponent descendant class, like TForm, will not free its implementing TComponent object by default when the interface is released.
However, if an IVCLComObject interface is assigned to the TComponent.VCLCOMObject property, then TComponent will delegate reference counting to that object, so that object can be freed if its reference count falls to 0 (TComponent does not increment the reference count of its IVCLCOMObject reference).
This is valid for all descendants of TComponent, unless they implement any reference counting of their own by overriding _AddRef() and _Release() manually.
IF you implement an interface in a form, it can be worth adding your own
_Release as follows:
function _Release: Integer; stdcall;
function TMyInterfacedForm._Release: Integer;
begin
result:=-1;
end;
I found that it was possible to free a form with an interface, and then have the _Release called as the local variable went out of scope. This could produce an access violation, as the Tcomponent implementation checks to see if FVCLComObject is set
if FVCLComObject = nil then
Result := -1 // -1 indicates no reference counting is taking place
else
Result := IVCLComObject(FVCLComObject)._Release;
Since at this stage, FVCLComobject is undefined, an annoying A/V exception can occur.
By hardwiring the -1, it can be called safely without trigger exceptions.
i have a class based on TInterfacedObject. i add it to TTreeNode's Data property.
TFacilityTreeItem=class(TInterfacedObject)
private
m_guidItem:TGUID;
m_SomeOtherNode:TTreeNode;
public
end;
i create many instances of this object & had assumed that because they're reference counted, i shouldn't need to Free them. that'd be handy.
however, when checking this, i turned on ReportMemoryLeaksOnShutdown and found they're not being freed after all.
these objects are being created in a frame that's placed on the main form. in the main form's FormClose, i clear the tree nodes such that every object should be freed.
what's happening?
thank you for your help!
TInterfacedObject itself is not reference counted, only interfaces are. You can implement interfaces using TInterfacedObject which basically saves you the effort of implementing the reference counting methods yourself. Unfortunately it still will not work in your case: The compiler does not know that you are assigning interfaces to the TTreeNode.Data property since it is not declared as an interface but as a pointer. So all kinds of weird things will happen:
MyInt := TFacilityTreeItem.Create; // ref count = 1
// Node.Data := MyInt; // won't compile
Node.Data := pointer(MyInt); // no interface assignment, ref count stays 1
...
end; // ref count reaches 0, your object gets freed
As soon as you try to access your object through the .Data property, you will get an access violation.
So, don't bother with interfaces in this case, you could get it to work, but it will be much more effort than it is worth.
You should declare the Field/Variable as Interface
IFacilityTreeItem = IInterface
end;
TFacilityTreeItem=class(TInterfacedObject, IFacilityTreeItem)
private
m_guidItem:TGUID;
m_SomeOtherNode:TTreeNode;
end;
var
Item: IFacilityTreeItem; // Variable as Interface
begin
Item:= TFacilityTreeItem.Create;
...
end;
To access your fields, you should declare properties in IFacilityTreeItem Interface, with Getters and Setters.
As dummzeuch said, you can get this to work with interfaces, but it takes some more code since the Data property of a TTreeNode is a pointer. For anyone wondering how to do that, this link has an example of how to do it for TListItem (it's pretty much the same for TTreeNode). You may also find it useful to read the section about interfaces and subsequent section about reference counting on that page.