I am trying to organize saving and loading of data changing in size. So the save file needs to store several (unknown and every time different number) of dynamic arrays.
The mistake appears in this MCVE:
procedure TAnnMainF.Button6Click(Sender: TObject);
var
f: file;
ari, aro: array of double;
i, Count: word;
begin
SetLength(aro, random(5) + 1);
for i := 0 to High(aro) do
aro[i] := random(2001) / 2000 - 1;
AssignFile(f, 'c:\delme\1.txt');
ReWrite(f);
Count := Length(aro);
BlockWrite(f, Count, SizeOf(word));
BlockWrite(f, aro[0], SizeOf(double) * Count);
CloseFile(f);
Reset(f);
BlockRead(f, Count, SizeOf(word));
BlockRead(f, ari[0], SizeOf(double) * Count);
CloseFile(f);
end;
This code results in I/O error 998.
I was trying to declare the type TDoubleArray = array of Double; and pass ari as a parameter in BlockRead. I also tried to SetLength(ari, Count) before I call BlockRead without any success.
The Answer to this question did not help me.
The code reads the Count properly but rises an exception at array loading.
What am I doing wrong?
You must set the size of the block in the ReWrite/Reset commands:
ReWrite(f,1);
...
Reset(f,1);
From documentation:
RecSize is an optional expression that can be specified only if F is an untyped file. If F is an untyped file, RecSize specifies the record size to be used in data transfers. If RecSize is omitted, a default record size of 128 bytes is assumed.
This means that reading the data will overflow the allocated buffer, hence the I/O error from the system.
Also read this warning about using ancient file I/O BlockRead/BlockWrite:
Warning: This is an older method that is particularly dangerous to use because of the untyped Buf parameter, leading to potential memory corruption. The record size used by BlockRead and BlockWrite is governed by the optional 2nd parameter to the Reset or Rewrite call that was used to open the file being written. It is preferable to use streams in your applications. For example, a user procedure involving a stream can use both TMemoryStreams and TFileStreams, instead of being limited to using files as with these older routines.
In general the speed difference between BlockRead/Write and streams is insignificant. For larger files, a buffered handler is preferred.
There is an excellent example of a buffered file stream handler from David: Buffered files (for faster disk access)
As #kami/#TomBrunberg noted and what you tried, you must also allocate the length of the ari dynamic array before reading the data.
I'm having a weird issue on using Delphi's TMemoryStream (or TFileStream for that matter). While reading a part of the stream into a byte array. Here's some code as an example.
procedure readfromstream();
var
ms : TMemoryStream;
buffer : array of byte;
recordSize : Integer;
begin
try
begin
ms := TMemeoryStream.Create();
ms.LoadFromFile(<some_path_to_a_binary_file>);
while ms.Position < ms.Size do
begin
buffer := nil;
SetLength(buffer, 4);
ms.ReadBuffer(buffer, 4);
move(buffer[0], recordSize, 4);
SetLength(buffer, recordSize);
ms.Position := ms.Position - 4; // Because I was having issues trying to read the rest of the record into a specific point in the buffer
FillChar(buffer, recordSize, ' ');
ms.ReadBuffer(buffer, recordSize); // Issue line ???
// Create the record from the buffer
end;
finally
begin
ms.Free();
end;
end;
procedure is called as,
// Some stuff happens before it
readfromstream();
// Some stuff happens after it
on debugging, I can see that it reads the stream into the buffer and the record is stored in memory appropriately. The procedure then exits normally and the debugger steps out of the procedure, but I end up straight back into the procedure and it repeats.
By forcing the procedure to exit prematurely I believe the issue involves the ms.ReadBuffer(buffer, recordSize); but I don't see why it would cause the issue.
This procedure is called only once. My test data has only one entry/data.
Any help would be greatly appreciated.
FillChar(buffer, recordSize, ' ');
Here you are overwriting the dynamic array variable, a pointer, rather than writing to the content of the array. That causes a memory corruption. Pretty much anything goes at that point.
The call to FillChar is needless anyway. You are going to read into the entire array anyway. Remove the call to FillChar.
For future reference, to do that call correctly, you write it like this:
FillChar(Pointer(buffer)^, ...);
or
FillChar(buffer[0], ...);
I prefer the former since the latter is subject to range errors when the array length is zero.
And then
ms.ReadBuffer(buffer, recordSize);
makes the exact same mistake, writing to the array variable rather than the array, and thus corrupting memory.
That should be
ms.ReadBuffer(Pointer(buffer)^, recordSize);
or
ms.ReadBuffer(buffer[0], recordSize);
The first 4 lines inside the loop are clumsy. Read directly into the variable:
ms.ReadBuffer(recordSize, SizeOf(recordSize));
I recommend that you perform some sanity checks on the value of recordSize that you read. For instance, any value less than 4 is clearly an error.
There's not a lot of point in moving the stream pointer back and reading again. You can copy recordSize into the first 4 bytes and the array and then read the rest.
Move(recordSize, buffer[0], SizeOf(recordSize));
ms.ReadBuffer(buffer[SizeOf(recordSize)], recordSize - SizeOf(recordSize));
A memory stream also seems wasteful. Why read the entire file into memory? That's going to place stress on your address space for large files. Use a buffered file stream.
Letting the caller allocate the stream would give more flexibility to the caller. They could then read from any type of stream and not be constrained to use a disk file.
Your try/finally block is wrong. You must acquire the resource immediately before the try. As you have it, an exception in the constructor leads to you calling Free on an uninitialized variable.
A better version might be:
procedure ReadFromStream(Stream: TStream);
var
buffer: TArray<byte>;
recordSize: Integer;
begin
while Stream.Position < Stream.Size do
begin
Stream.ReadBuffer(recordSize, SizeOf(recordSize));
if recordSize < SizeOf(recordSize) then
raise ...;
SetLength(buffer, recordSize);
Move(recordSize, buffer[0], SizeOf(recordSize));
if recordSize > SizeOf(recordSize) then
Stream.ReadBuffer(buffer[SizeOf(recordSize)],
recordSize - SizeOf(recordSize));
// process record
end;
end;
Sorry I can't add a comment, being a newb and all :) This reply is based on my understanding of Clayton's code in light of his comment with the recordSize values.
The reason David's code is looping is probably that you are interpreting each four byte "block" is a number. I'll assume your first Stream.Readbuffer is correct and that the first four bytes in the file is a length.
Now, unless I'm mistaken, I expect the recordSize will usually be greater than SizeOf(recordSize), which I think should be 4 (the size of an int). Nevertheless, this line is meaningless here.
The SetLength is correct, given my previous assumption.
Now your Move is where the story hits a snag: you haven't read anything since you read the length! So before the move, you should have:
bytesRead := Stream.Readbuffer(Pointer(buffer)^, recordSize);
Now you can check for EOF:
if bytesRead <> recordSize then
raise...;
...and move the buffer somewhere (if you wish):
Move(buffer[0], dest[0], recordSize);
And you are positioned to read the next recordSize value.
Repeat until EOF.
I have a binary file (2.5 MB) and I want to find position of this sequence of bytes: CD 09 D9 F5. Then I want to write some data after this position and also overwrite old data (4 KB) with zeros.
Here is how I do it now but it is a bit slow.
ProcessFile(dataToWrite: string);
var
fileContent: string;
f: file of char;
c: char;
n, i, startIndex, endIndex: integer;
begin
AssignFile(f, 'file.bin');
reset(f);
n := FileSize(f);
while n > 0 do
begin
Read(f, c);
fileContent := fileContent + c;
dec(n);
end;
CloseFile(f);
startindex := Pos(Char($CD)+Char($09)+Char($D9)+Char($F5), fileContent) + 4;
endIndex := startIndex + 4088;
Seek(f, startIndex);
for i := 1 to length(dataToWrite) do
Write(f, dataToWrite[i]);
c := #0;
while (i < endIndex) do
begin
Write(f, c); inc(i);
end;
CloseFile(f);
end;
See this answer: Fast read/write from file in delphi
Some options are:
memory mapped files
TFileStream
blockread
To search the file buffer, see Best way to find position in the Stream where given byte sequence starts - one answer mentions the Boyer-Moore algorithm for fast detection of a byte sequence.
Your code to read the entire file into a string is very wasteful. Pascal I/O uses buffering so I don't think it's the byte by byte aspect particularly. Although one big read would be better. The main problem will be the string concatenation and the extreme heap allocation demand required to concatenate the string, one character at a time.
I'd do it like this:
function LoadFileIntoString(const FileName: string): string;
var
Stream: TFileStream;
begin
Stream := TFileStream.Create(FileName, fmOpenRead);
try
SetLength(Result, Stream.Size);//one single heap allocation
Stream.ReadBuffer(Pointer(Result)^, Length(Result));
finally
Stream.Free;
end;
end;
That alone should make a big difference. When it comes to writing the file, a similar use of strings will be much faster. I've not attempted to decipher the writing part of your code. Writing the new data, and the block of zeros again should be batched up to as few separate writes as possible.
If ever you find that you need to read or write very small blocks to a file, then I offer you my buffered file streams: Buffered files (for faster disk access).
The code could be optimised further to read only a portion of the file, and search until you find the target. You may be able to avoid reading the entire file that way. However, I suspect that these changes will make enough of a difference.
I am using a MemoryStream to construct an array of bytes i need to send to a server.I have thre questions:
1) Is there a better way to construct an array of bytes than this ?
2) Why this pice of code writes bogus in my memory stream ?
var
serial : word;
MS : TMemoryStream;
const
somebytes : array [0..1] of byte = ($72,$72);
...
begin
MS := TMemoryStream.Create();
try
MS.Write(somebytes[0],2);
serial := $3E6C;
MS.Write(serial,2);
finally
MS.Free;
end;
Using the debugger i see that in the stream is added the value $6F32 instead of $3E6C.
3) If i call
MS.Position := 2;
and then i access PByte(MS.Memory)^ why do i get the first byte in the stream instead of the third?
Is there a better way to construct an array of bytes than this?
That's a perfectly reasonable way to do it, in my view.
I see that in the stream is added the value $6F32 instead of $3E6C.
Check again. The correct values are in fact added. But beware of the traps of little endian data types. The 4 bytes added to your stream, in order, are: $72, $72, $6C, $3E.
Why do I get the first byte in the stream instead of the third?
Because the Memory property always refers to the beginning of the stream. It does not take account of the stream's current position.
I do not know how to use any API that is not in the RTL. I have been using SetFilePointer and GetFileSize to read a Physical Disk into a buffer and dump it to a file, something like this in a loop does the job for flash memory cards under 2GB:
SetFilePointer(PD,0,nil,FILE_BEGIN);
SetLength(Buffer,512);
ReadFile(PD,Buffer[0],512,BytesReturned,nil);
However GetFileSize has a limit at 2GB and so does SetFilePointer. I have absolutley no idea how to delcare an external API, I have looked at the RTL and googled for many examples and have found no correct answer.
I tried this
function GetFileSizeEx(hFile: THandle; lpFileSizeHigh: Pointer): DWORD;
external 'kernel32';
and as suggested this
function GetFileSizeEx(hFile: THandle; var FileSize: Int64): DWORD;
stdcall; external 'kernel32';
But the function returns a 0 even though I am using a valid disk handle which I have confirmed and dumped data from using the older API's.
I am using SetFilePointer to jump every 512 bytes and ReadFile to write into a buffer, in reverse I can use it to set when I am using WriteFile to write Initial Program Loader Code or something else to the disk. I need to be able to set the file pointer beyond 2gb well beyond.
Can someone help me make the external declarations and a call to both GetFileSizeEx and SetFilePointerEx that work so I can modify my older code to work with say 4 to 32gb flash cards.
I suggest that you take a look at this Primoz Gabrijelcic blog article and his GpHugeFile unit which should give you enough pointers to get the file size.
Edit 1 This looks rather a daft answer now in light of the edit to the question.
Edit 2 Now that this answer has been accepted, following a long threads of comments to jachguate's answer, I feel it incumbent to summarise what has been learnt.
GetFileSize and
SetFilePointer have no 2GB
limitation, they can be used on files
of essentially arbitrary size.
GetFileSizeEx and
SetFilePointerEx are much
easier to use because they work
directly with 64 bit quantities and
have far simpler error condition
signals.
The OP did not in fact need to
calculate the size of his disk. Since
the OP was reading the entire
contents of the disk the size was not
needed. All that was required was to
read the contents sequentially until
there was nothing left.
In fact
GetFileSize/GetFileSizeEx
do not support handles to devices
(e.g. a physical disk or volume) as
was requested by the OP. What's more,
SetFilePointer/SetFilePointerEx
cannot seek to the end of such device
handles.
In order to obtain the size of a
disk, volume or partition, one should
pass the the
IOCTL_DISK_GET_LENGTH_INFO
control code to
DeviceIoControl.
Finally, should you need to use GetFileSizeEx and SetFilePointerEx then they can be declared as follows:
function GetFileSizeEx(hFile: THandle; var lpFileSize: Int64): BOOL;
stdcall; external 'kernel32.dll';
function SetFilePointerEx(hFile: THandle; liDistanceToMove: Int64;
lpNewFilePointer: PInt64; dwMoveMethod: DWORD): BOOL;
stdcall; external 'kernel32.dll';
One easy way to obtain these API imports them is through the excellent JEDI API Library.
The GetFileSizeEx routine expects a pointer to a LARGE_INTEGER data type, and documentation says:
If your compiler has built-in support for 64-bit integers, use the QuadPart member to store the 64-bit integer
Lucky you, Delphi has built-in support for 64 bit integers, so use it:
var
DriveSize: LongWord;
begin
GetFilePointerSizeEx(PD, #DriveSize);
end;
SetFilePointerEx, on the other hand, expects parameters for liDistanceToMove, lpNewFilePointer, both 64 bit integers. My understanding is it wants signed integers, but you have the UInt64 data type for Unsingned 64 bit integers if I'm missunderstanding the documentation.
Alternative coding
Suicide, first of all your approach is wrong, and because of your wrong approach you ran into some hairy problems with the way Windows handles Disk drives opened as files. In pseudo code your approach seems to be:
Size = GetFileSize;
for i=0 to (Size / 512) do
begin
Seek(i * 512);
ReadBlock;
WriteBlockToFile;
end;
That's functionally correct, but there's a simpler way to do the same without actually getting the SizeOfDisk and without seeking. When reading something from a file (or a stream), the "pointer" is automatically moved with the ammount of data you just read, so you can skip the "seek". All the functions used to read data from a file return the amount of data that was actually read: you can use that to know when you reached the end of the file without knowing the size of the file to start with!
Here's an idea of how you can read an physical disk to a file, without knowing much about the disk device, using Delphi's TFileStream:
var DiskStream, DestinationStream:TFileStream;
Buff:array[0..512-1] of Byte;
BuffRead:Integer;
begin
// Open the disk for reading
DiskStream := TFileStream.Create('\\.\PhysicalDrive0', fmOpenRead);
try
// Create the file
DestinationStream := TFileStream.Create('D:\Something.IMG', fmCreate);
try
// Read & write in a loop; This is where all the work's done:
BuffRead := DiskStream.Read(Buff, SizeOf(Buff));
while BuffRead > 0 do
begin
DestinationStream.Write(Buff, BuffRead);
BuffRead := DiskStream.Read(Buff, SizeOf(Buff));
end;
finally DestinationStream.Free;
end;
finally DiskStream.Free;
end;
end;
You can obviously do something similar the other way around, reading from a file and writing to disk. Before writing that code I actually attempted doing it your way (getting the file size, etc), and immediately ran into problems! Apparently Windows doesn't know the exact size of the "file", not unless you read from it.
Problems with disks opened as files
For all my testing I used this simple code as the base:
var F: TFileStream;
begin
F := TFileStream.Create('\\.\PhysicalDrive0', fmOpenRead);
try
// Test code goes here...
finally F.Free;
end;
end;
The first (obvious) thing to try was:
ShowMessage(IntToStr(DiskStream.Size));
That fails. In the TFileStream implementation that depends on calling FileSeek, and FileSeek can't handle files larger then 2Gb. So I gave GetFileSize a try, using this code:
var RetSize, UpperWord:DWORD;
RetSize := GetFileSize(F.Handle, #UpperWord);
ShowMessage(IntToStr(UpperWord) + ' / ' + IntToStr(RetSize));
That also fails, even those it should be perfectly capable of returning file size as an 64 bit number! Next I tried using the SetFilePointer API, because that's also supposed to handle 64bit numbers. I thought I'd simply seek to the end of the file and look at the result, using this code:
var RetPos, UpperWord:DWORD;
UpperWord := 0;
RetPos := SetFilePos(F.Handle, 0, #UpperWord, FILE_END);
ShowMessage(IntToStr(UpperWord) + ' / ' + IntToStr(RetPos));
This code also fails! And now I'm thinking, why did the first code work? Apparently reading block-by-block works just fine and Windows knows exactly when to stop reading!! So I thought maybe there's a problem with the implementation of the 64 bit file handling routines, let's try seeking to end of the file in small increments; When we get an error seeking we know we reached the end we'll stop:
var PrevUpWord, PrevPos: DWORD;
UpWord, Pos: DWORD;
UpWord := 0;
Pos := SetFilePointer(F.Handle, 1024, #UpWord, FILE_CURRENT); // Advance the pointer 512 bytes from it's current position
while (UpWord <> PrevUpWord) or (Pos <> PrevPos) do
begin
PrevUpWord := UpWord;
PrevPos := Pos;
UpWord := 0;
Pos := SetFilePointer(F.Handle, 1024, #UpWord, FILE_CURRENT);
end;
When trying this code I had a surprise: It doesn't stop at the of the file, it just goes on and on, for ever. It never fails. To be perfectly honest I'm not sure it's supposed to ever fail... It's probably not supposed to fail. Anyway, doing a READ in that loop fails when we're past the end of file so we can use a VERY hacky mixed approach to handle this situation.
Ready-made routines that work around the problem
Here's the ready-made routine that gets the size of the physical disk opened as a file, even when GetFileSize fails, and SetFilePointer with FILE_END fails. Pass it an opened TFileStream and it will return the size as an Int64:
function Hacky_GetStreamSize(F: TFileStream): Int64;
var Step:DWORD;
StartPos: Int64;
StartPos_DWORD: packed array [0..1] of DWORD absolute StartPos;
KnownGoodPosition: Int64;
KGP_DWORD: packed array [0..1] of DWORD absolute KnownGoodPosition;
Dummy:DWORD;
Block:array[0..512-1] of Byte;
begin
// Get starting pointer position
StartPos := 0;
StartPos_DWORD[0] := SetFilePointer(F.Handle, 0, #StartPos_DWORD[1], FILE_CURRENT);
try
// Move file pointer to the first byte
SetFilePointer(F.Handle, 0, nil, FILE_BEGIN);
// Init
KnownGoodPosition := 0;
Step := 1024 * 1024 * 1024; // Initial step will be 1Gb
while Step > 512 do
begin
// Try to move
Dummy := 0;
SetFilePointer(F.Handle, Step, #Dummy, FILE_CURRENT);
// Test: Try to read!
if F.Read(Block, 512) = 512 then
begin
// Ok! Save the last known good position
KGP_DWORD[1] := 0;
KGP_DWORD[0] := SetFilePointer(F.Handle, 0, #KGP_DWORD[1], FILE_CURRENT);
end
else
begin
// Read failed! Move back to the last known good position and make Step smaller
SetFilePointer(F.Handle, KGP_DWORD[0], #KGP_DWORD[1], FILE_BEGIN);
Step := Step div 4; // it's optimal to devide by 4
end;
end;
// From here on we'll use 512 byte steps until we can't read any more
SetFilePointer(F.Handle, KGP_DWORD[0], #KGP_DWORD[1], FILE_BEGIN);
while F.Read(Block, 512) = 512 do
KnownGoodPosition := KnownGoodPosition + 512;
// Done!
Result := KnownGoodPosition;
finally
// Move file pointer back to starting position
SetFilePointer(F.Handle, StartPos_DWORD[0], #StartPos_DWORD[1], FILE_BEGIN);
end;
end;
To be complete, here are two routines that may be used to set and get the file pointer using Int64 for positioning:
function Hacky_SetStreamPos(F: TFileStream; Pos: Int64):Int64;
var aPos:Int64;
DWA:packed array[0..1] of DWORD absolute aPos;
const INVALID_SET_FILE_POINTER = $FFFFFFFF;
begin
aPos := Pos;
DWA[0] := SetFilePointer(F.Handle, DWA[0], #DWA[1], FILE_BEGIN);
if (DWA[0] = INVALID_SET_FILE_POINTER) and (GetLastError <> NO_ERROR) then
RaiseLastOSError;
Result := aPos;
end;
function Hacky_GetStreamPos(F: TFileStream): Int64;
var Pos:Int64;
DWA:packed array[0..1] of DWORD absolute Pos;
begin
Pos := 0;
DWA[0] := SetFilePointer(F.Handle, 0, #DWA[1], FILE_CURRENT);
Result := Pos;
end;
Last notes
The 3 routines I'm providing take as a parameter an TFileStream, because that's what I use for file reading and writing. They obviously only use TFileStream.Handle, so the parameter can simply be replaced with an file handle: the functionality would stay the same.
I know this thread is old, but...
One small suggestion - if you use the Windows DeviceIoControl(...) function you can get Drive Geometry and/or Partition Information, and use them to get the total size/length of the opened drive or partition. No more messing around with incrementally seeking to the end of the device.
Those IOCTLs can also be used to give you the correct volume sector size, and you could use that instead of defaulting to 512 everywhere.
Very very useful. But I got a problem for disks greater then 4 GB.
I solved replacing:
// Ok! Save the last known good position
KGP_DWORD[1] := 0;
KGP_DWORD[0] := SetFilePointer(F.Handle, 0, #KGP_DWORD[1], FILE_CURRENT);
with the following:
// Ok! Save the last known good position
KnownGoodPosition := KnownGoodPosition + Step;
Many thanks again...
And many thanks also to James R. Twine. I followed the advice of using IOCTL_DISK_GET_DRIVE_GEOMETRY_EX and got disk dimension with no problem and no strange workaround.
Here is the code:
TDISK_GEOMETRY = record
Cylinders : Int64; //LargeInteger
MediaType : DWORD; //MEDIA_TYPE
TracksPerCylinder: DWORD ;
SectorsPerTrack: DWORD ;
BytesPerSector : DWORD ;
end;
TDISK_GEOMETRY_EX = record
Geometry: TDISK_GEOMETRY ;
DiskSize: Int64; //LARGE_INTEGER ;
Data : array[1..1000] of byte; // unknown length
end;
function get_disk_size(handle: thandle): int64;
var
BytesReturned: DWORD;
DISK_GEOMETRY_EX : TDISK_GEOMETRY_EX;
begin
result := 0;
if DeviceIOControl(handle,IOCTL_DISK_GET_DRIVE_GEOMETRY_EX,
nil,0,#DISK_GEOMETRY_EX, sizeof(TDISK_GEOMETRY_EX),BytesReturned,nil)
then result := DISK_GEOMETRY_EX.DiskSize;
end;