I'm using openCV with Go and have this function:
func ConvertMatToSlice(c_Mat *CMatrix) []float32 {
c_output := C.Convert(c_Mat.CType)
h := &reflect.SliceHeader{
Data: uintptr(unsafe.Pointer(c_output.val)),
Len: int(c_output.length),
Cap: int(c_output.length),
}
return *(*[]float32)(unsafe.Pointer(h))
}
When I run go vet ./... it gives me:
opencv_wrapper.go:31:7: unsafeptr: possible misuse of reflect.SliceHeader (govet)
h := &reflect.SliceHeader{
^
Is there any other way to get the slice because I'm sure go-vet has its reasons to complain about this.
See Go Issue #40701. A safer alternative would be the following:
func ConvertMatToSlice(c_Mat *CMatrix) []float32 {
c_output := C.Convert(c_Mat.CType)
result := make([]float32, c_output.length)
sh := (*reflect.SliceHeader)(unsafe.Pointer(&result))
sh.Data = uintptr(unsafe.Pointer(c_output.val))
return result
}
I've made a test program mocking the behaviour of your code without using cgo. So let's place it in a directory and do some escape analysis with go build -gcflags "-m -m".
What we're interested is making sure that the slice (slice header) we return is allocated on the heap, and that so is the pointer which it points to in the Data field. As we can see from the escape analysis, b escapes and so does the slice allocation:
./main.go:10:6: b escapes to heap:
./main.go:10:6: flow: {storage for &Output literal} = &b:
./main.go:10:6: from &b (address-of) at ./main.go:17:38
[...]
./main.go:25:16: make([]float32, c_output.length) escapes to heap:
./main.go:25:16: flow: {heap} = &{storage for make([]float32, c_output.length)}:
./main.go:25:16: from make([]float32, c_output.length) (non-constant size) at ./main.go:25:16
In your case, you'll have to make sure yourself from the C code that the pointer is allocated on the heap.
In general, the rule that can be taken from the discussion on the Go issue I've referenced is to never create SliceHeaders yourself. They might work but they can and will lead to memory corruption, creating difficult and hard-to-debug situations in your code. If you do need to get access to a slice yourself, do so either by using make or by using reflect.MakeSlice which are guaranteed to work and will cooperate with the GC properly, and then make the necessary changes to the slice header.
Related
I have been doing Advent of Code this year, to learn Zig, and I discovered something during Day 5 that really confused me. So: mild spoilers for Day 5 of Advent of Code 2022, I guess?
I decided to implement my solution to Day 5 as an ArrayList of ArrayLists of U8s, which has ended up working well. My full solution file is here (probably terribly un-idiomatic Zig, but we all have to start somewhere).
As part of my solution, I have a function, which I call moveCrates, on a struct which wraps my arraylist of arraylists.
The relevant part of the struct declaration looks as so:
const BunchOfStacks = struct {
stacks: ArrayList(ArrayList(u8)),
...
This function is here, and looks like this:
fn moveCrates(self: *BunchOfStacks, amount: usize, source: usize, dest: usize) !void {
const source_height = self.stacks.items[source - 1].items.len;
const crate_slice = self.stacks.items[source - 1].items[(source_height - amount)..];
try self.stacks.items[dest - 1].appendSlice(crate_slice);
self.stacks.items[source - 1].shrinkRetainingCapacity(source_height - amount);
}
You can note that I refer 3 times to the source list by the very verbose reference self.stacks.items[source - 1]. This is not how I first wrote this function. I first wrote it like below:
fn moveCrates(self: *BunchOfStacks, amount: usize, source: usize, dest: usize) !void {
var source_list: ArrayList(u8) = self.stacks.items[source - 1];
const source_height = source_list.items.len;
const crate_slice = source_list.items[(source_height - amount)..];
try self.stacks.items[dest - 1].appendSlice(crate_slice);
source_list.shrinkRetainingCapacity(source_height - amount);
}
But this second form, where I make a local variable for my convenience, DOES NOT GIVE THE CORRECT RESULTS! It compiles fine, but seems to always point source_list to the same internal ArrayList(u8) (whichever one it first picks) regardless of what the value of source is. This means that the test example produces incorrect output.
This function is called within a loop, like so:
while (instructions.next()) |_| {
// First part is the verb, this is always "move" so skip it
// Get the amount next
const amount: usize = try std.fmt.parseInt(usize, instructions.next().?, 10);
// now skip _from_
_ = instructions.next();
// Now get source
const source: usize = try std.fmt.parseInt(usize, instructions.next().?, 10);
// Now skip _to_
_ = instructions.next();
// Now get dest
const dest: usize = try std.fmt.parseInt(usize, instructions.next().?, 10);
var crates_moved: usize = 0;
while (crates_moved < amount) : (crates_moved += 1) {
try stacks_part1.moveCrates(1, source, dest);
}
try stacks_part2.moveCrates(amount, source, dest);
}
Ultimately, as you can see, I have just avoided making variable assignments in the function and this passes the test (and the puzzle).
I have checked for issues in the zig repo that might be related, and cannot find anything immediately obvious (search used is this). I've looked on StackOverflow, and found this question, which does have some similarity to my issue (pointers seem a bit whack in while loops), but it's not the same.
I've scoured the zig documentation on loops and assignment, on the site, but don't see anything calling out this behaviour specifically.
I'm assuming I've either completely misunderstood something (or missed something that isn't well documented), or this is a bug — the language is under heavy active dev, after all.
I'm expecting that an assignment like I perform should work as expected — being a simple shorthand to avoid having to write out the repeated self.stacks.items[source - 1], so I'm hopeful that this is something that I'm just doing wrong. Zig version is v0.11.0-dev.537+36da3000c
An array list stores items as a slice (a pointer to the first item plus length).
When you do
var source_list: ArrayList(u8) = self.stacks.items[source - 1];
you make a shallow copy of the array list. You might be confused by your knowledge of some higher-level languages where this would've copied a reference to an object of the array list, but in Zig this isn't the case. In Zig everything is a "value object".
When you call shrinkRetainingCapacity it changes the length property of the items slice, but the changes are done to the local copy of the array list. The "real" array list, that is stored in another array list, remain unaffected.
TL;DR You need to use a pointer:
var source_list: *ArrayList(u8) = &self.stacks.items[source - 1];
This fixes the failing test.
Use case: I am converting data from a very old program of mine to a database friendly format. There are parts where I have to do multiple passes over the old data, because in particular the keys have to first exist before I can reference them in relationships. So I thought why not put the incomplete parts in a vector of references during the first pass and return it from the working function, so I can easily use that vector to make the second pass over whatever is still incomplete. I like to avoid pointers when possible so I looked into std::reference_wrapper<T> which seemes like exactly what I need .. except I don't understand it's behavior at all.
I have both vector<OldData> old_data and vector<NewData> new_data as member of my conversion class. The converting member function essentially does:
//...
vector<reference_wrapper<NewData>> incomplete;
for(const auto& old_elem : old_data) {
auto& new_ref = *new_data.insert(new_data.end(), convert(old_elem));
if(is_incomplete(new_ref)) incomplete.push_back(ref(new_ref));
}
return incomplete;
However, incomplete is already broken immediately after the for loop. The program compiles, but crashes and produces gibberish. Now I don't know if I placed ref correctly, but this is only one of many tries where I tried to put it somewhere else, use push_back or emplace_back instead, etc. ..
Something seems to be going out of scope, but what? both new_data and old_data are class members, incomplete also lives outside the loop, and according to the documentation, reference_wrapper is copyable.
Here's a simplified MWE that compiles, crashes, and produces gibberish:
// includes ..
using namespace std;
int main() {
int N = 2; // works correctly for N = 1 without any other changes ... ???
vector<string> strs;
vector<reference_wrapper<string>> refs;
for(int i = 0; i < N; ++i) {
string& sref = ref(strs.emplace_back("a"));
refs.push_back(sref);
}
for (const auto& r : refs) cout << r.get(); // crash & gibberish
}
This is g++ 10.2.0 with -std=c++17 if it means anything. Now I will probably just use pointers and be done, but I would like to understand what is going on here, documentation / search does not seem to help..
The problem here is that you are using vector data structure which might re-allocate memory for the entire vector any time that you add an element, so all previous references on that vector most probably get invalidated, you can resolve your problem by using list instead of vector.
I've got some code that worked fine under Delphi 2007 but breaks under D2010. It involves passing in a string, converting it to a PWideChar (specifically, a WideString pointer, not a UnicodeString pointer), doing some processing, and then calling SysFreeString on it. It works fine until a blank string is passed in, then SysFreeString breaks. It calls a bunch of things that end up raising an Int 3 breakpoint inside NTDLL.DLL. Continuing past this point results in
Project raised exception class
$C0000005 with message 'access
violation at 0x7747206e: read of
address 0x539b8dba'.
Which, if you look closely, is not the standard Access Violation message.
The top of the stack trace when it hits the Int 3 looks like this:
:774e475d ; ntdll.dll
:774afad0 ; ntdll.dll
:774e5de9 ; ntdll.dll
:774a6dff ; ntdll.dll
:76fc1075 ; C:\Windows\system32\ole32.dll
:770e443a ; C:\Windows\system32\oleaut32.dll
:770e3ea3 oleaut32.SysFreeString + 0x4a
Does anyone have any idea what's going on here?
Edit (from the comments):
This isn't a WideString, though. It's
a PWideChar generated by
StringToOleStr, and there are no
double-free errors when a non-blank
string is passed in. Unfortunately, I
can't really post a code sample
because this is a third-party
component that's under copyright. (And
I can't ask them for support because
it's no longer supported. Basically,
the whole thing's one big mess.)
I'm going to try psychic debugging. You've got some kind of heap corruption in your application and SysFreeString is the unfortunate victim (it's hard to tell without OS symbols, you should probably install the MSFT symbol packages for your OS).
Try enabling application verifier (in particular pageheap) for your app and see if it crashes earlier.
It is hard to diagnose without seeing your actual code, however WideString automatically calls SysFreeString() when it goes out of scope. It sounds like your code may be making a second call to SysFreeString() on memory that has already been freed. WideString itself has not changed at all between D2007 and D2010, but other aspects of Delphi's string handling have. Maybe you are not managing the strings correctly. Can you please show your actual code?
A simple test shows that you need to be really careful on what you do in which order.
So: even though you cannot post a small example, can you indicate what you are doing in a bit more detail?
Bad debugging; ignore the things below; see comment.
The SysFreeString() is being called at the end of the the Allocate() call, even though it returns a PWideChar:
program ShowStringToOleStrBehaviourProject;
{$APPTYPE CONSOLE}
uses
SysUtils;
function Allocate(const Value: UnicodeString): PWideChar;
begin
Result := StringToOleStr(Value);
// implicit SysFreeString(WideChars);
end;
procedure Run;
var
WideChars: PWideChar;
begin
WideChars := Allocate('Foo');
Writeln(WideChars);
end;
begin
try
Run();
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
end.
Note the console still outputs 'Foo' because the memory has not been overwritten yet.
--jeroen
It can be different reasons of such kind of errors:
You try to free with SysFreeString a memory which are allocated not with SysAllocString, but for example with CoTaskMemAlloc.
You have heap correct.
Heap corruptions are difficult to localize. The function HeapSetInformation can be very helpful. For example you can use
HeapSetInformation(NULL,HeapEnableTerminationOnCorruption,NULL,0);
Other good way is usage of HeapValidate function. For example you can define a function which verify all heaps pf the process (code in C, which can be easy rewritten in Delphi):
BOOL MyHeapValidate (void)
{
HANDLE hProcessHeaps[1024];
DWORD i;
DWORD dwNumberOfHeaps;
BOOL bSuccess = FALSE;
dwNumberOfHeaps = GetProcessHeaps (sizeof(hProcessHeaps)/sizeof(hProcessHeaps[0]),
hProcessHeaps);
if (dwNumberOfHeaps > sizeof(hProcessHeaps)/sizeof(hProcessHeaps[0])) {
MessageBox(NULL, TEXT("GetProcessHeaps()"),
TEXT("Error in MyHeapValidate()"), MB_OK);
return FALSE;
}
for (i=0; i<dwNumberOfHeaps; i++) {
bSuccess = HeapValidate (hProcessHeaps[i], 0, NULL);
if (!bSuccess)
return bSuccess;
}
return bSuccess;
}
The usage of this function can be like following:
void BadFunction(BSTR bstr)
{
LPOLESTR psz = OLESTR("Test12");
lstrcpy (bstr, psz);
}
int main()
{
LPOLESTR psz = OLESTR("Test");
BSTR bstr = SysAllocString (psz);
// verify that before call of BadFunction() all process heaps are OK!
if (!MyHeapValidate()) {
_tprintf(TEXT("heap is corrupted after the step 1.\n"));
return 1;
}
BadFunction(bstr);
if (!MyHeapValidate()) {
_tprintf(TEXT("heap is corrupted after the step 1.\n"));
return 1;
}
SysFreeString (bstr);
return 0;
}
With respect of inserting MyHeapValidate() in different suspected places you can very quickly local the place of corruption.
+1 for Larry Osterman's answer.
Some Windows memory functions behave slightly different under debugger: if they detect some kind of misuse - they trigger breakpoint to notify debugger. So, basically, your code is doing something wrong.
You can install hooks on SysAllocString/SysFreeString and redirect them to your memory manager (which should be in full debug mode) to collect more info. Or you can just pass these calls through to original functions, installing only a filter, which watches for memory actions.
And you can install debug symbols to get more info too (I'm not sure if Delphi debugger can use it, but Process Explorer - can. You can connect it to your process and see call stack).
If I declare
PSomeStruct = ^TSomeStruct;
TSomeStruct = record
s1 : string;
end;
and I run the following code:
var
p: PSomeStruct;
begin
new(p);
p^.s1:= 'something bla bla bla';
dispose(p);
the FastMM 4 memory manager reports that there was a memory leak (type: string, data dump: "something bla bla bla"). However, if I do set the s1 string to empty before calling dispose it's OK.
The second way I found is to change from record type to class, then instead of new I'm creating the instance, and instead of dispose I'm calling instance.Free(). It works without manually cleaning the strings.
Is there a way to make Delphi automatically clean my strings when I call dispose?
Is FastMM the first unit used in your .dpr? Otherwise it could be finalized too early, reporting false memoryleaks.
And does this simplified codesample also generate the same memoryleak as when you use your JvSimpleXML? When it's not, there is probably more going on then you suspect.
In my opinion: when FastMM reports a memory leak, there is a memoryleak.
You are already doing the correct thing. If FastMM says that string has leaked, then FastMM is wrong, or it's reporting a different string from the one you think it is. The Dispose procedure releases strings from records.
In this particular case, there shouldn't have been any memory allocated for that string anyway. It's a string literal, so I'd expect the compiler to assign that literal; its reference count should be -1 and FastMM never should have seen it.
I tried to make my code as simple as possible,but I failed at it.
This is my code:
class function TWS.WinsockSend(s:integer;buffer:pointer;size:word):boolean;
begin
dwError := Send(s,buffer,size,0);
// Debug
if(dwError = SOCKET_ERROR) then
begin
dwError := WSAGetLastError;
CloseSocket(s);
WSACleanup;
case (dwerror) of
//Case statement
else
LogToFile('Unhandled error: ' + IntToStr(dwError) + ' generated by WSASend');
end;
Exit(false);
end;
// if the size of the bytes sent isn't the expected one.
while(dwError <> size) do
dwError:= dwError + Send(s,Ptr(cardinal(buffer) + dwError),size-dwError,0);
Exit(true);
end;
The error is placed at
dwError:= dwError + Send(s,Ptr(cardinal(buffer) + dwError),size-dwError,0);
Error is "Constant object cannot be passed as var parameter"
I understand I need a variable,but isn't there a way I can do it without adding one more line?
When the compiler complains about the way you're passing a parameter, the first thing you need to know is what the parameter expects. Therefore, you should go look at the declaration of Send. If looking at the declaration doesn't immediately give you an idea of what to fix, then you need to include that declaration with the code you post in your question.
I suspect that this actually has nothing to do with incrementing a pointer. Instead, the compiler is complaining about the third parameter, where you are trying to pass the expression size-dwError. I guess the parameter is declared like this:
var buffersize: Word;
The function plans on providing a new value for that parameter — that's what var means — so the thing you pass to that parameter needs to be something that can receive a value. You can't assign a new value to the result of subtracting two variables.
Take a closer look at where the compiler complained about that line. Didn't it place the cursor somewhere near the third parameter? That's a clue that the problem is there.
Decrement size, and then pass it to the function.
Dec(size, dwError);
Inc(dwError, Send(s, Ptr(cardinal(buffer) + dwError), size, 0));
Why do you care about adding another line? Have you reached your quota for the day? Lines are cheap; don't be afraid to use two to express yourself when one won't do. Likewise for variables. When your code doesn't work, saving a byte or two doesn't matter at all.
At the very least, you should have added more lines in order to track down the source of the problem. When you have one line of code that's performing several independent calculations (such as getting a new pointer value, getting a new size, and calling a function), break the line into several separate pieces. That way, if there's a problem with one of them and the compiler complains, you'll know exactly which one to blame.
Correct, this will not work as written. When your dealing with var parameters, you have to build the parameter BEFORE passing it to the procedure/function. When a Var parameter is passed, the procedure is allowed to modify it. If you attempted to copy two variables together on the call, where would this result go?
The other issue is that dwError is not delcared. A class method does NOT have access to the data elements of the object the class defines. If you drop the class, then you will have access to the data elements, but will require that the class first be created.
You should only be using class methods in places where the input and output are completely contained within the method.
How are you allocating your buffer? Internally is it an array?
Sounds like Send has a format parameter (like send (const something;size:integer)
Workaround is using pchar (entirepointerexpression)[0]