If we look at the following code (playground link):
use std::cell::RefCell;
use std::rc::Rc;
struct Person {
name: String,
parent: Option<Rc<RefCell<Person>>>,
child: Option<Rc<RefCell<Person>>>,
}
impl Drop for Person {
fn drop(&mut self) {
println!("dropping {}", &self.name);
}
}
pub fn main() {
let anakin = Rc::new(RefCell::new(Person {
parent: None,
child: None,
name: "anakin".to_owned(),
}));
let luke = Rc::new(RefCell::new(Person {
parent: None,
child: None,
name: "luke".to_owned(),
}));
luke.borrow_mut().parent = Some(anakin.clone());
anakin.borrow_mut().child = Some(luke.clone());
println!("anakin {}", Rc::strong_count(&anakin));
println!("luke {}", Rc::strong_count(&luke));
}
When the code runs, the message from the drop implementation is not printed, because this code is supposed to leak memory.
When the code finishes the count for both luke and anakin is supposed to end up at 1 rather than 0 (which is when the managed heap data would be cleaned up).
Why does the count not end up being 0? I would have thought the following sequence would happen:
we start with two data locations in heap, one pointed to by luke and anakin.child, the other pointed to by anakin and luke.parent. The object luke owns luke.parent. The object anakin owns anakin.child
luke goes out of scope, which means its members which it owns are also dropped. So luke and luke.parent drop. Both are Rcs, so the ref count for both the memory locations therefore goes down to 1
anakin is dropped, causing the Rc object anakin and anakin.child to drop, which causes ref count for both memory locations to go down to 0.
This is where the heap data should be cleaned up? Are the members of an object not dropped when it is dropped?
When I remove or comment the lines connecting luke and anakin via borrow_mut, the drop happens as expected and the messages are correctly printed.
luke goes out of scope, which means its members which it owns are also dropped. So luke and luke.parent drop. Both are Rcs, so the ref count for both the memory locations therefore goes down to 1
This is the step where you are misunderstanding.
Are the members of an object not dropped when it is dropped?
Exactly. The count that is decreased is the one directly associated with the data that the Rc points. When luke goes out of scope, the Rc goes out of scope, and the count of things referencing luke's RefCell decreases, the code knows nothing about luke.parent.
In your code, after
luke.borrow_mut().parent = Some(anakin.clone());
anakin.borrow_mut().child = Some(luke.clone());
there are 4 Rc objects (2 on stack, 2 on heap), and 2 RefCell objects on the heap with counts associated with them. Each RefCell has a count of 2 as you've seen, because there are 2 Rcs referencing each RefCell.
When anakin drops, the count for it's RefCell decreases, so you then have the luke RefCell on the heap with a count of 2 and the anakin RefCell with a count of 1.
When luke drops, that then decreases the count of that RefCell, so each one now has a count of 1. You end up with no Rc values on the stack that reference the RefCell, but each RefCell references the other RefCell, so there is no way for Rust to know that they are safe to be dropped.
I can't quite tell from your question, but this is absolutely an expected limitation of Rc type when used with RefCell because it allows for the introduction of cycles in the ownership of objects.
Your code is nearly the minimum reproducible example of an Rc cycle: What is a minimal example of an Rc dependency cycle?
Related
I'm a little baffled about the inner work of the sequence expression in F#.
Normally if we make a sequential file reader with seq with no intentional caching of data
seq {
let mutable current = file.Read()
while current <> -1 do
yield current
}
We will end up with some weird behavior if we try to do some re-iterate or backtracking, My Idea of this was, since Read() is a function calling some mutable value we can't expect the output to be correct if we re-iterate. But then this behaves nicely even on boundary reading?
let Read path =
seq {
use fp = System.IO.File.OpenRead path
let buf = [| for _ in 0 .. 1024 -> 0uy |]
let mutable pos = 1
let mutable current = 0
while pos <> 0 do
if current = 0 then
pos <- fp.Read(buf, 0, 1024)
if pos > 0 && current < pos then
yield buf.[current]
current <- (current + 1) % 1024
}
let content = Read "some path"
We clearly use the same buffer to enhance performance, but assuming that we read the 1025 byte, it will trigger an update to the buffer, if we then try to read any byte with position < 1025 after we still get the correct output. How can that be and what are the difference?
Your question is a bit unclear, so I'll try to guess.
When you create a seq { }, you're essentially creating a state machine which will run only as far as it needs to. When you request the very first element from it, it'll start at the top and run until your first yield instruction. Then, when you request another value, it'll run from that point until the next yield, and so on.
Keep in mind that a seq { } produces an IEnumerable<'T>, which is like a "plan of execution". Each time you start to iterate the sequence (for example by calling Seq.head), a call to GetEnumerator is made behind the scenes, which causes a new IEnumerator<'T> to be created. It is the IEnumerator which does the actual providing of values. You can think of it in more classical terms as having an array over which you can iterate (an iterable or enumerable) and many pointers over that array, each of which are at different points in the array (many iterators or enumerators).
In your first code, file is most likely external to the seq block. This means that the file you are reading from is baked into the plan of execution; no matter how many times you start to iterate the sequence, you'll always be reading from the same file. This is obviously going to cause unpredictable behaviour.
However, in your second code, the file is opened as part of the seq block's definition. This means that you'll get a new file handle each time you iterate the sequence or, essentially, a new file handle per enumerator. The reason this code works is that you can't reverse an enumerator or iterate over it multiple times, not with a single thread at least.
(Now, if you were to manually get an enumerator and advance it over multiple threads, you'd probably run into problems very quickly. But that is a different topic.)
I have a large fixed-size array of variable-sized arrays of u32. Most of the second dimension arrays will be empty (i.e. the first array will be sparsely populated). I think Vec is the most suitable type for both dimensions (Vec<Vec<u32>>). Because my first array might be quite large, I want to find the most space-efficient way to represent this.
I see two options:
I could use a Vec<Option<Vec<u32>>>. I'm guessing that as Option is a tagged union, this would result each cell being sizeof(Vec<u32>) rounded up to the next word boundary for the tag.
I could directly use Vec::with_capacity(0) for all cells. Does an empty Vec allocate zero heap until it's used?
Which is the most space-efficient method?
Actually, both Vec<Vec<T>> and Vec<Option<Vec<T>>> have the same space efficiency.
A Vec contains a pointer that will never be null, so the compiler is smart enough to recognize that in the case of Option<Vec<T>>, it can represent None by putting 0 in the pointer field. What is the overhead of Rust's Option type? contains more information.
What about the backing storage the pointer points to? A Vec doesn't allocate (same link as the first) when you create it with Vec::new or Vec::with_capacity(0); in that case, it uses a special, non-null "empty pointer". Vec only allocates space on the heap when you push something or otherwise force it to allocate. Therefore, the space used both for the Vec itself and for its backing storage are the same.
Vec<Vec<T>> is a decent starting point. Each entry costs 3 pointers, even if it is empty, and for filled entries there can be additional per-allocation overhead. But depending on which trade-offs you're willing to make, there might be a better solution.
Vec<Box<[T]>> This reduces the size of an entry from 3 pointers to 2 pointers. The downside is that changing the number of elements in a box is both inconvenient (convert to and from Vec<T>) and more expensive (reallocation).
HashMap<usize, Vec<T>> This saves a lot of memory if the outer collection is sufficiently sparse. The downsides are higher access cost (hashing, scanning) and a higher per element memory overhead.
If the collection is only filled once and you never resize the inner collections you could use a split data structure:
This not only reduces the per-entry size to 1 pointer, it also eliminates the per-allocation overhead.
struct Nested<T> {
data: Vec<T>,
indices: Vec<usize>,// points after the last element of the i-th slice
}
impl<T> Nested<T> {
fn get_range(&self, i: usize) -> std::ops::Range<usize> {
assert!(i < self.indices.len());
if i > 0 {
self.indices[i-1]..self.indices[i]
} else {
0..self.indices[i]
}
}
pub fn get(&self, i:usize) -> &[T] {
let range = self.get_range(i);
&self.data[range]
}
pub fn get_mut(&mut self, i:usize) -> &mut [T] {
let range = self.get_range(i);
&mut self.data[range]
}
}
For additional memory savings you can reduce the indices to u32 limiting you to 4 billion elements per collection.
The following is base on my guess. Someone please point out the parts that I understand incorrectly.
If I have a class, of which an instance occupies 128 bits, called Class128Bits. And my program runs on a 64 bits computer.
First, I call let pointer = UnsafeMutablePointer<Calss128Bits>.allocate(capacity: 2)
the memory layout should look like this:
000-063 064 bits chaos
064-127 064 bits chaos
128-255 128 bits chaos
256-383 128 bits chaos
If I call pointer.pointee = aClass128Bits, it crashes because the pointers in the first two grids have not been initialized yet. Accessing to what they point to leads to unpredictable results.
But if I call pointer.initialize(to: aClass128Bits, count: 2), the pointers could be initialized like this:
000-063 address to offset 128
064-127 address to offset 256
128-255 a copy of aClass128Bits
256-383 a copy of aClass128Bits
Then any accesses will be safe.
However this cannot explain why UnsafeMutablePointer<Int> does not crash.
Original
The case I am facing:
The pointer to Int works fine, but the one to String crashes.
I know that I need to initialize it like this:
But I can't see the reason why I need to pass "42" twice.
In C, I might do something similar like this:
char *pointer = (char *)malloc(3 * sizeof(char));
memcpy(pointer, "42", 3);
free(pointer)
If allocate equals malloc, free equals deallocate, memcpy equals pointee{ set },
then what do initialize and deinitialize actually do?
And why does my code crash?
let pointer0 = UnsafeMutablePointer<String>.allocate(capacity: 1)
let pointer1 = UnsafeMutablePointer<Int>.allocate(capacity: 1)
let check the size of both
MemoryLayout.size(ofValue: pointer0) // 8
MemoryLayout.size(ofValue: pointer1) // 8
let check the value of .pointee
pointer0.pointee // CRASH!!!
while
pointer1.pointee // some random value
Why? The answer is as simple, as it can be. We allocated 8 bytes, independently from "associated" Type. Now is clear, that 8 bytes in memory are not enough to store any String. the underlying memory must be referenced indirectly. But there are some 8 random bytes there ... Loading what is in the memory with address represented by 8 random bytes as a String will most likely crash :-)
Why didn't it crash in the second case? Int value is 8 bytes long and the address can be represented as Int value.
let's try in the Playground
import Foundation
let pointer = UnsafeMutablePointer<CFString>.allocate(capacity: 1)
let us = Unmanaged<CFString>.passRetained("hello" as CFString)
pointer.initialize(to: us.takeRetainedValue())
print(pointer.pointee)
us.release()
// if this playground crash, try to run it again and again ... -)
print(pointer.pointee)
look what it prints to me :-)
hello
(
"<__NSCFOutputStream: 0x7fb0bdebd120>"
)
There is no miracle behind. pointer.pointee is trying to represent what is in the memory, which address is stored in our pointer, as a value of its associated type. It never crashes for Int because every 8 continues bytes somewhere in the memory can be represented as Int.
Swift use ARC, but creating the Unsafe[Mutable]Poiner doesn't allocate any memory for the instance of T, destroying it doesn't deallocate any memory for it.
Typed memory must be initialized before use and deinitialized after use. This is done using initialize and deinitialize methods respectively. Deinitialization is only required for non-trivial types. That said, including deinitialization is a good way to future-proof your code in case you change to something non-trivial
Why doesn't assignment to .pointee with Int value crash?
Initialize store the address of value
Assignment to pointee update the value at stored address
Without initializing It most likely will crash, only the probability is less by modifying only 8 bytes in memory at some random address.
trying this
import Darwin
var k = Int16.max.toIntMax()
typealias MyTupple = (Int32,Int32,Int8, Int16, Int16)
var arr: [MyTupple] = []
repeat {
let p = UnsafeMutablePointer<MyTupple>.allocate(capacity: 1)
if k == 1 {
print(MemoryLayout.size(ofValue: p), MemoryLayout.alignment(ofValue: p),MemoryLayout.stride(ofValue: p))
}
arr.append(p.pointee)
k -= 1
defer {
p.deallocate(capacity: 1)
}
} while k > 0
let s = arr.reduce([:]) { (r, v) -> [String:Int] in
var r = r
let c = r["\(v.0),\(v.1),\(v.2),\(v.3)"] ?? 0
r["\(v.0),\(v.1),\(v.2),\(v.3)"] = c + 1
return r
}
print(s)
I received
8 8 8
["0,0,-95,4104": 6472, "0,0,0,0": 26295]
Program ended with exit code: 0
It doesn't look very random, is it? That explains, why the crash with the typed pointer to Int is very unlikely.
One reason you need initialize(), and the only one as for now maybe, is
for ARC.
You'd better think with local scope variables, when seeing how ARC works:
func test() {
var refVar: RefType = initValue //<-(1)
//...
refVar = newValue //<-(2)
//...
//<-(3) just before exiting the loacl scope
}
For a usual assignment as (2), Swift generates some code like this:
swift_retain(_newValue)
swift_release(_refVar)
_refVar = _newValue
(Assume _refVar and _newValue are unmanaged pseudo vars.)
Retain means incrementing the reference count by 1, and release means decrementing the reference count by 1.
But, think what happens when the initial value assignment as at (1).
If the usual assignment code was generated, the code might crash at this line:
swift_release(_refVar)
because newly allocated region for a var may be filled with garbages, so swift_release(_refVar) cannot be safely executed.
Filling the newly region with zero (null) and release safely ignoring the null could be one solution, but it's sort of redundant and not effective.
So, Swift generates this sort of code for initial value assignment:
(for already retained values, if you know ownership model, owned by you.)
_refVar = _initValue
(for unretained values, meaning you have no ownership yet.)
swift_retain(_initValue)
_refVar = _initValue
This is initialize.
No-releasing the garbage data, and assign an initial value, retaining it if needed.
(The above explanation of "usual assignment" is a little bit simplified, Swift omits swift_retain(_newValue) when not needed.)
When exiting the local scope at (3), Swift just generates this sort of code:
swift_release(_refVar)
So, this is deinitialize.
Of course, you know retaining and releasing are not needed for primitive types like Int, so initialize and deinitialize may be donothing for such types.
And when you define a value type which includes some reference type properties, Swift generates initialize and deinitialize procedures specialized for the type.
The local scope example works for the regions allocated on the stack, and initialize() and deinitialize() of UnsafeMutablePointer works for the regions allocated in the heap.
And Swift is evolving so swift, that you might find another reason for needing initialize() and deinitialize() in the future, you'd better make it a habit to initialize() and deinitialize() all allocated UnsafeMutablePointers of any Pointee types.
From the documentation it is possible to conclude that .initialize() is a method that :
Initializes memory starting at self with the elements of source.
And .deinitialize() is a method that :
De-initializes the count Pointees starting at self, returning their
memory to an uninitialized state.
We should understand that when we are using UnsafeMutablePointer we should manage memory on our own. And methods that are described above help us to do this.
So in your case lets analyze example that you provide:
let pointer = UnsafeMutablePointer<String>.allocate(capacity: 1)
// allocate a memory space
pointer.initialize(to: "42")
// initialise memory
pointer.pointee // "42"
// reveals what is in the pointee location
pointer.pointee = "43"
// change the contents of the memory
pointer.deinitialize()
// return pointer to an unintialized state
pointer.deallocate(1)
// deallocate memory
So your code crashes because you do not initialize memory and try to set value.
Previously in objective-c when we are working with objects we always use [[MyClass alloc] init]].
In this case :
alloc:
allocates a part of memory to hold the object, and returns the
pointer.
init:
sets up the initial parameters of the object and returns it.
So basically .initialize() sets the value to the allocated memory part. When you create an object only with alloc you only set reference to empty memory part in the heap. When you call .initialize() you set value to this memory allocation in the heap.
Nice article about the pointers.
In most managed languages (that is, the ones with a GC), local variables that go out of scope are inaccessible and have a higher GC-priority (hence, they'll be freed first).
Now, C is not a managed language, what happens to variables that go out of scope here?
I created a small test-case in C:
#include <stdio.h>
int main(void){
int *ptr;
{
// New scope
int tmp = 17;
ptr = &tmp; // Just to see if the memory is cleared
}
//printf("tmp = %d", tmp); // Compile-time error (as expected)
printf("ptr = %d\n", *ptr);
return 0;
}
I'm using GCC 4.7.3 to compile and the program above prints 17, why? And when/under what circumstances will the local variables be freed?
The actual behavior of your code sample is determined by two primary factors: 1) the behavior is undefined by the language, 2) an optimizing compiler will generate machine code that does not physically match your C code.
For example, despite the fact that the behavior is undefined, GCC can (and will) easily optimize your code to a mere
printf("ptr = %d\n", 17);
which means that the output you see has very little to do with what happens to any variables in your code.
If you want the behavior of your code to better reflect what happens physically, you should declare your pointers volatile. The behavior will still be undefined, but at least it will restrict some optimizations.
Now, as to what happens to local variables when they go out of scope. Nothing physical happens. A typical implementation will allocate enough space in the program stack to store all variables at the deepest level of block nesting in the current function. This space is typically allocated in the stack in one shot at the function startup and released back at the function exit.
That means that the memory formerly occupied by tmp continues to remain reserved in the stack until the function exits. That also means that the same stack space can (and will) be reused by different variables having approximately the same level of "locality depth" in sibling blocks. The space will hold the value of the last variable until some other variable declared in some sibling block variable overrides it. In your example nobody overrides the space formerly occupied by tmp, so you will typically see the value 17 survive intact in that memory.
However, if you do this
int main(void) {
volatile int *ptr;
volatile int *ptrd;
{ // Block
int tmp = 17;
ptr = &tmp; // Just to see if the memory is cleared
}
{ // Sibling block
int d = 5;
ptrd = &d;
}
printf("ptr = %d %d\n", *ptr, *ptrd);
printf("%p %p\n", ptr, ptrd);
}
you will see that the space formerly occupied by tmp has been reused for d and its former value has been overriden. The second printf will typically output the same pointer value for both pointers.
The lifetime of an automatic object ends at the end of the block where it is declared.
Accessing an object outside of its lifetime is undefined behavior in C.
(C99, 6.2.4p2) "If an object is referred to outside of its lifetime, the behavior is undefined. The value of a pointer becomes indeterminate when the object it points to reaches the end of its lifetime."
Local variables are allocated on the stack. They are not "freed" in the sense you think about GC languages, or memory allocated on the heap. They simply go out of scope, and for builtin types the code won't do anything - and for objects, the destructor is called.
Accessing them beyond their scope is Undefined Behaviour. You were just lucky, as no other code has overwritten that memory area...yet.
I want to understand what is stored in the stack and heap in swift. I have a rough estimation:
Everything that you print and the memory address appears not the values, those are stored in the stack, and what is printed out as values, those are on the heap, basically according to value and reference types. Am I completely wrong? And optionally, could you provide a visual representation of the stack/heap?
As #Juul stated Reference types are stored in the Heap and values in the stack.
Here is the explanation:
Stack and Heap
Stack is used for static memory allocation and Heap for dynamic memory allocation, both stored in the computer's RAM .
Variables allocated on the stack are stored directly to the memory, and access to this memory is very fast, and its allocation is determined when the program is compiled. When a function or a method calls another function which in turns calls another function, etc., the execution of all those functions remains suspended until the very last function returns its value. The stack is always reserved in a LIFO order, the most recently reserved block is always the next block to be freed. This makes it really simple to keep track of the stack. Freeing a block from the stack is nothing more than adjusting one pointer.
Variables allocated on the heap have their memory allocated at run time, and accessing this memory is a bit slower, but the heap size is only limited by the size of virtual memory. Elements of the heap have no dependencies with each other and can always be accessed randomly at any time. You can allocate a block at any time and free it at any time. This makes it more complex to keep track of which parts of the heap are allocated or free at any given time.
For Escaping Closure:
An important note to keep in mind is that in cases where a value stored on a stack is captured in a closure, that value will be copied to the heap so that it's still available by the time the closure is executed.
For more reference: http://net-informations.com/faq/net/stack-heap.htm
Classes (reference types) are allocated in the heap, value types (like Struct, String, Int, Bool, etc) live in the stack. See this topic for more detailed answers: Why Choose Struct Over Class?
Stack vs Heap
Stack is a part of thread. It consists of method(function) frames in LIFO order. Method frame contains only local variables. Actually it is method stack trace which you see during debugging or analysing error[About].
Heap another part of memory where ARC[About] come in play. It takes more time to allocate memory here(find appropriate place and allocate it in synchronous way).
Theses concepts are the same as [JVM illustration]
Xcode propose you next variant using Debug Memory Graph
*To see Backtrace use:
Edit Scheme... -> <Action> -> Diagnostics -> Malloc Stack Logging
[Value vs Reference type]
[Class vs Struct]
Usually when we ask question like this (is it stack or is it heap) we care about performance and are motivated by the desire to avoid the excessive cost of heap allocation. Following the general rule saying that "reference types are heap-allocated and the value types are stack-allocated" may lead to suboptimal design decisions and needs further discussion.
One may falsely conclude that passing around structs (value types) is universally faster than passing classes (reference types) because it never requires heap allocation. Turns out this is not always true.
The important counter-example are protocol types where concrete polymorphic types with value-semantics (structs) implement a protocol, like in this toy example:
protocol Vehicle {
var mileage: Double { get }
}
struct CombustionCar: Vehicle {
let mpg: Double
let isDiesel: Bool
let isManual: Bool
var fuelLevel: Double // gallons
var mileage: Double { fuelLevel * mpg }
}
struct ElectricCar: Vehicle {
let mpge: Double
var batteryLevel: Double // kWh
var mileage: Double { batteryLevel * mpge / 33.7 }
}
func printMileage(vehicle: Vehicle) {
print("\(vehicle.mileage)")
}
let datsun: Vehicle = CombustionCar(mpg: 18.19,
isDiesel: false,
isManual: false,
fuelLevel: 12)
let tesla: Vehicle = ElectricCar(mpge: 132,
batteryLevel: 50)
let vehicles: [Vehicle] = [datsun, tesla]
for vehicle in vehicles {
printMileage(vehicle: vehicle)
}
Note that CombustionCar and ElectricCar objects have different sizes, yet we are able to mix them in an array of Vehicle protocol types. This raises the question: don't the array container elements need to be of the same size? How can the compiler compute the offset of the array element if it doesn't always know the element size at the compile time?
It turns out there's quite a lot of logic under the hood. Swift compiler will create what is called an Existential Container. It's a fixed-sized data structure acting as a wrapper around an object. It this container that gets passed to a function call (is pushed onto the stack) instead of the actual struct.
Existential Container is five words long:
| |
|valueBuffer|
| |
| vwt |
| pwt |
The first three words are called the valueBuffer and this is where actual structure gets stored. Well, that's unless the struct size is greater than three words - in such case the compiler will allocate the struct on the heap and store the reference to it in the valueBuffer:
STACK STACK HEAP
| mpge | | reference |-->| mpg |
|batteryLevel| | | | isDiesel |
| | | | | isManual |
| vwt | | vwt | | fuelLevel |
| pwt | | pwt |
So passing a protocol type object like this to a function may actually require a heap allocation. The compiler will do the allocation and copy the values so you still get the value semantics but the cost will vary depending on whether your struct is 3 words long or more. This renders the "value-types on stack, reference-types on heap" not always correct.