I'd like to understand what happens when the size of a dynamic array is increased.
My understanding so far:
Existing array elements will remain unchanged.
New array elements are initialised to 0
All array elements are contiguous in memory.
When the array size is increased, will the extra memory be tacked onto the existing memory block, or will the existing elements be copied to a entirely new memory block?
Does changing the size of a dynamic array have consequences for pointers referencing existing array elements?
Thanks,
[edit] Incorrect assumption struck out. (New array elements are initialised to 0)
Existing array elements will remain unchanged: yes
New array elements are initialized to 0: yes (see update) no, unless it is an array of compiler managed types such as string, an other array or a variant
All array elements are contiguous in memory: yes
When the array size is increased, the array will be copied. From the doc:
...memory for a dynamic array is reallocated when you assign a value to the array or pass it to the SetLength procedure.
So yes, increasing the size of a dynamic array does have consequences for pointers referencing existing array elements.
If you want to keep references to existing elements, use their index in the array (0-based).
Update
Comments by Rob and David prompted me to check the initialization of dynamic arrays in Delphi5 (as I have that readily available anyway). First using some code to create various types of dynamic arrays and inspecting them in the debugger. They were all properly initialized, but that could still have been a result prior initialization of the memory location where they were allocated. So checked the RTL. It turns out D5 already has the FillChar statement in the DynArraySetLength method that Rob pointed to:
// Set the new memory to all zero bits
FillChar((PChar(p) + elSize * oldLength)^, elSize * (newLength - oldLength), 0);
In practice Embarcadero will always zero initialise new elements simply because to do otherwise would break so much code.
In fact it's a shame that they don't officially guarantee the zero allocation because it is so useful. The point is that often at the call site when writing SetLength you don't know whether you are growing or shrinking the array. But the implementation of SetLength does know – clearly it has to. So it really makes sense to have a well-defined action on any new elements.
What's more, if they want people to be able to switch easily between managed and native worlds then zero allocation is desirable since that's what fits with the managed code.
Related
I alread know, that a dynamic array is automatically deallocated/freed after use.
Does the same applies for resizing, especially decreasing? The manual and most help sites only cover increasing the array size.
test: array of TLabel;
SetLength(test, 10);
// fill array here
SetLength(test, 2); // <=== are entries 3-10 are automatically destroyed?
are entries 3-10 are automatically destroyed?
No, they are not automatically destroyed because those entries are dynamically allocated (and are not managed types). Only the pointers that refer to those items are released. It is your responsibility to destroy the items if necessary, because the compiler has no way to guarantee you wouldn't still use them from another reference (or have already destroyed them).
I must also point out that technically items "3-10" is wrong. Dynamic array are zero based. So the references for entries 2 to 9 are the ones released.
I alread know, that a dynamic array is automatically deallocated/freed after use
In addition, your question indicates you don't properly understand this. It seems you believed that when your array goes out of scope the labels referenced would be automatically destroyed. This is incorrect!
No matter where how or why some/all dynamic array entries are released Delphi won't automatically destroy objects types or any dynamically allocated pointer memory. Delphi only automatically releases memory for primitives (Integer, TDateTime, Double short strings), records and managed types1 (interfaces, long strings, other dynamic arrays).
1 Of course this is via reference counting. I.e. reference is reduced by 1; and the underlying object/string/array is released if and only if refCount is reduced to zero.
As whosrdaddy pointed out, if you want automatic destruction of contained objects, then you need to use a container that implements an ownership concept. TObjectList is an example. Although it doesn't work exactly like a dynamic array, it's behaviour is similar enough that it can usually be used as a replacement very easily.
Is there ever a reason where I would do something like:
{foobar, NumberOfElementsInFoobarTuple, ...}
I would imagine the internal datastructure for tuple_size knows it's size already, like the binary type. With binaries it's still beneficial to keep track of byte size as a variable instead of calling byte_size(). It doesn't make sense but that's how it is. I'm creating this tuple through list_to_tuple so it may have varying size.
A tuple is one fixed size block compared to a list which is a linked list of list cells each of which has a head and a tail. So a tuple explicitly contains its size and tuple_size(Tuple) just directly returns this value.
Considere I have a CFMutableArray object created with the following function call:
CFMutableArrayRef marray = CFArrayCreateMutable(kCFAllocatorDefault, 1, &kCFTypeArrayCallBacks);
According to the CFMutableArray Documentation, the second argument of CFArrayCreateMutable, which is called capacity, is "the maximum number of values that can be contained by the new array. The array starts empty and can grow to this number of values (and it can have less).
Pass 0 to specify that the maximum capacity is not limited. The value must not be negative."
However, if I append more than one value to my new array, it keeps growing. I mean, if the new array already has one value and I append a new one with CFArrayAppendValue(marray, newValue), this value is stored and the array count goes to 2, exceeding its capacity.
So, why this happens? Did I misunderstand the documentation?
Interesting. I don't think you're mis-understanding the documentation. I will point out that CFMutableArrayRef is "Toll Free Bridged" on iOS with NSMutableArray and therefore interchangeable. On iOS [NSMutableArray arrayWithCapacity] is NOT so limited and capacity is basically "guidance" and NOT a hard upper limit.
It might be worth filing a bug report on that, probably the docs are just wrong.
UPDATE: Just goes to show ya... always, always, always follow the maxim of the best docs are the source. I CMD-clicked on CFArrayCreateMutable to look at the comment in the source .h file.... guess I was right, because that says 'capacity' is a HINT that the implementation may ignore, as it apparently does in this case.
#function CFArrayCreateMutable
⋮
#param capacity A hint about the number of values that will be held by the CFArray.
Pass 0 for no hint. The implementation may ignore this hint,** or may use it to
optimize various operations. An array's actual capacity is only limited by
address space and available memory constraints). If this parameter is negative,
the behavior is undefined.
Don't forget, header comments are written by developers, whilst "docs" are written by some tech writer that doesn't have the same depth of knowledge.
From docs:
CFArrayAppendValue
Adds a value to an array giving it the new largest index.
Parameters
theArray
The array to which value is to be added. If theArray is a limited-capacity array and it is full before this operation, the behavior is undefined.
So, I think you should only make sure not to add a value to a full limited-capacity array, or the behavior will be undefined!
Or, you can pass 0 for the capacity parameter to specify that the maximum capacity is not limited.
Update:
As #Cliff Ribaudo pointed out, it seems there is a contradiction between the official documentation and code documentation:
#param capacity A hint about the number of values that will be held
by the CFArray. Pass 0 for no hint. The implementation may
ignore this hint, or may use it to optimize various
operations. An array's actual capacity is only limited by
address space and available memory constraints).
So we can assume the online documentation is outdated and the code documentation is possibly the right.
In Delphi, you can get the size of a value type with the sizeof() compiler magic function, but calling sizeof() on a reference type will give you the size of a pointer, not of the value it's pointing to.
For objects, you can get the memory size with the InstanceSize method, but what about for dynamic arrays? Due to padding, length(MyArray) * sizeof(element) may not be accurate. So, is there any accurate way to get the memory size of a dynamic array?
In fact, length(MyArray) * sizeof(element) will be accurate for the array content excluding any internal dynamic array or string.
If you want the whole array used memory, including nested reference types content size, you can use our TDynArray wrapper. It is able to serialize into binary any dynamic array, including reference counted members (like dynamic arrays or strings). You have SaveTo / SaveToStream methods for this purpose, and you are able to get the actual size of all content.
Take a look at this blog article, which presents this wrapper. It is open source, and works from Delphi 5 up to XE4, in both Win32 and Win64 platform.
Between elements of a dynamic array there is no padding, Length(MyArray)*SizeOf(Element) should be accurate.
TDictionary<TKey,TValue> uses an internal array that is doubled if it is full:
newCap := Length(FItems) * 2;
if newCap = 0 then
newCap := 4;
Rehash(newCap);
This performs well with medium number of items, but if one gets to the upper limit it is very unfortunate, because it might throw an EOutOfMemory exception even if there is almost half of the memory still available.
Is there any way to influence this behaviour? How do other collection classes deal with this scenario?
You need to understand how a Dictionary works. A dictionary contains a list of "hash buckets" where the items you insert are placed. That's a finite number, so once you fill it up you need to allocate more buckets, there's no way around it. Since the assignment of objects-to-buckets is based on the result of a hash function, you can't simply add buckets to the end of the array and put stuff in there, you need to re-allocate the whole list of blocks, re-hash everything and put it in the (new) corresponding buckets.
Given this behavior, the only way to make the dictionary not re-allocate once full is to make sure it never gets full. If you know the number of items you'll insert in the dictionary pass it as a parameter to the constructor and you'll be done, no more dictionary reallocations.
If you can't do that (you don't know the number of items you'll have in the dictionary) you'll need to reconsider what made you select the TDictionary in the first place and select a data structure that offers better compromise for your particular algorithm. For example you could use binary search trees, as they do the balancing by rotating information in existing nodes, no need for re-allocations ever.