I am experiencing a strange error which I have narrowed down to a small piece of code (see below). I am reading a binary file into a TMemoryStream (variable ms1), through a TFileStream object (variable fs). Then I want to copy the binary data to another TMemoryStream object (ms2). This is what gives me the "stream read error" exception. Strange thing is, that if I don't load up the ms1 object with the file contents, things work fine, i.e. ms2.CopyFrom does not give me an exception..
Any help is greatly appreciated....
procedure TForm5.BitBtn1Click(Sender: TObject);
var
ms1: TMemoryStream;
fs: TFileStream;
ms2 : TMemoryStream;
FilePath: string;
begin
FilePath := 'C:\weekcpdf_tech6.bin';
ms1 := TMemoryStream.Create;
fs := nil;
try
ms1 := TMemoryStream.Create;
fs := TFileStream.Create(FilePath, fmOpenRead);
ms1.CopyFrom(fs, fs.Size);
ms2 := TMemoryStream.Create;
ms2.CopyFrom(ms1, ms1.Size);
finally
FreeAndNil(fs);
FreeAndNil(ms1);
end;
end;
First, you have a memory leak, since you create two memory stream objects that you assign to ms1. Remove the second one.
Second, after ms1.CopyFrom(fs, fs.Size); you must set ms1.Position := 0 so that you copy from the start of ms1 -- and not from the end of it.
This is actually fairly well documented:
CopyFrom copies Count bytes from the stream specified by Source into the stream. It then moves the current position by Count bytes and returns the number of bytes copied.
Hence, after ms1.CopyFrom(fs, fs.Size); you are at the end of ms1. Further,
If Count is greater than or less than 0, CopyFrom reads from the current position in Source.
Therefore, ms2.CopyFrom(ms1, ms1.Size); will read from the current position (= the end!) of ms1. Hence, you will likely try to read a lot of bytes that do not exist. And what happens then?
Because the CopyFrom method uses ReadBuffer and WriteBuffer to do the effective copying, if the Count is greater than the SourceStream size, ReadBuffer throws an exception stating that a stream read error has occured [sic!].
Always read the docs! :)
(Although I must admit that the last quote isn't quite perfect in this situation.)
Related
I have the following code for a series of file drawdowns using IDHttp.Get, the contents of the files
procedure Tform1.GetData;
{***************************}
var
fs2 : tfilestream;
s : char;
begin
Sleep(1000);
idhttp1.HandleRedirects := TRUE;
fsjson2 := tfilestream.Create((GstrPath+GstrRep+'-'+GstrHome+'.json'),fmcreate);;
idhttp1.IOHandler := idssl;
IdSSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2];
IdSSL.SSLOptions.Mode := sslmUnassigned;
try
idhttp1.Get(GstrURL,fs2);
except
on E: Exception do
begin
rememo1.Lines.Add('Seems to be an issue, trying again...');
Sleep(500);
idhttp1.Get(GstrURL,fs2);
end;
end;
I would like there to be a method of either trimming the first character although I don't think this is possible, or replacing the first character (which comes with the info by default) with a blank character. I think it's a little out of my skillset at the moment to do, and so would appreciate any help someone can give.
Thanks
Ant
There are several possible solutions:
You are receiving data into a stream. If you want to trim the first character, you could, at the place you handle the stream, skip reading the first character and throw it away.
If you have no control on where the receiving stream is handled, then you may simply create a new stream and loop reading all characters from the receiving stream and write them into the destination stream. Then throw away the received stream and keep the one you created.
If the data received (currently the stream) is not too big, you could receive it into a string instead of a stream, then you can trim/delete/insert anything with simple string manipulation, and finally write the modified string back to a stream for later use.
i am working on a little byte patching program but i encountered an error.
copying the file before modifying fails with no error, (no copied output is seen) but the file patches successfully.
Here is the Patch Code
procedure DoMyPatch();
var
i: integer;
FileName: string;
input: TFileStream;
FileByteArray, ExtractedByteArray: array of Byte;
begin
FileName := 'Cute1.res';
try
input := TFileStream.Create(FileName, fmOpenReadWrite);
except
begin
ShowMessage('Error Opening file');
Exit;
end
end;
input.Position := 0;
SetLength(FileByteArray, input.size);
input.Read(FileByteArray[0], Length(FileByteArray));
for i := 0 to Length(FileByteArray) do
begin
SetLength(ExtractedByteArray, Length(OriginalByte));
ExtractedByteArray := Copy(FileByteArray, i, Length(OriginalByte));
// function that compares my array of bytes
if CompareByteArrays(ExtractedByteArray, OriginalByte) = True then
begin
// Begin Patching
CopyFile(PChar(FileName), PChar(ChangeFileExt(FileName, '.BAK')),
true); =======>>> fails at this point, no copied output is seen.
input.Seek(i, SoFromBeginning);
input.Write(BytetoWrite[0], Length(BytetoWrite)); =====>>> patches successfully
input.Free;
ShowMessage('Patch Success');
Exit;
end;
end;
if Assigned(input) then
begin
input.Free;
end;
ShowMessage('Patch Failed');
end;
sidenote : it copies fine if i close the filestream before attempting copy.
by the way, i have tested it on Delphi 7 and XE7.
Thanks
You cannot copy the file because you locked it exclusively when you opened it for the file stream, which is why CopyFile fails.
You should close the file before attempting to call CopyFile. Which would require you to reopen the file to patch it. Or perhaps open the file with a different sharing mode.
Some other comments:
The exception handling is badly implemented. Don't catch exceptions here. Let them float up to the high level.
Lifetime management is fluffed. You can easily leak as it stands. You need to learn about try/finally.
You overrun buffers. Valid indices for a dynamic array are 0 to Length(arr)-1 inclusive. Or use low() and high().
You don't check the value returned by CopyFile. Wrap it with a call to Win32Check.
The Copy function returns a new array. So you make a spurious call to SetLength. To copy the entire array use the one parameter overload of Copy.
Showing messages in this function is probably a mistake. Better to let the caller provide user feedback.
There are loads of other oddities in the code and I've run out of energy to point them all out. I think I got the main ones.
Earlier today I opened a question here asking if my method to scan files in computer was correct. As solution, I received a few tips, and the one of the solutions I thought: "this need to be solved urgent!", was saying about memory overflow, once I was reading the files entirely in memory. So I started trying to find a way to read the files piece by piece, and I got something (wrong/bogus), that I need some help to figure out how to do this correctly.
The method is simple like this for now:
procedure ScanFile(FileName: string);
const
MAX_SIZE = 100*1024*1024;
var
i, aux, ReadLimit: integer;
MyFile: TFileStream;
Target: AnsiString;
PlainText: String;
Buff: array of byte;
TotalSize: Int64;
begin
if (POS('.exe', FileName) = 0) and (POS('.dll', FileName) = 0) and
(POS('.sys', FileName) = 0) then //yeah I know it's not the best way...
begin
try
MyFile:= TFileStream.Create(FileName, fmOpenRead);
except on E: EFOpenError do
MyFile:= NIL;
end;
if MyFile <> NIL then
try
TotalSize:= MyFile.Size;
while TotalSize > 0 do begin
ReadLimit:= Min(TotalSize, MAX_SIZE);
SetLength(Buff, ReadLimit);
MyFile.ReadBuffer(Buff[0], ReadLimit);
PlainText:= RemoveNulls(Buff); //this is to transform the array of bytes in string, I posted the code below too...
for i:= 1 to Length(PlainText) do
begin //Begin the search..
end;
dec(TotalSize, ReadLimit);
end;
finally
MyFile.Free;
end;
end;
Code for RemoveNulls is:
function RemoveNulls(const Buff: array of byte): String;
var
i: integer;
begin
for i:= 0 to Length(Buff) do
begin
if Buff[i] <> 0 then
Result:= Result + Chr(Ord(Buff[i]));
end;
end;
Ok, the problems I got with this code so far was:
1- each time the while is repeated, I get more memory consumed, when I was expecting to get only MAX 100MB as described in the MAX_SIZE variable, right?
2- I created a file with 2 occurrences of what should be filtered, and for some unknown reason I got about 10 repeated occurrences, looks like I'm scanning the file repeatedly.
I appreciate your help guys, and if someone have this kind of code already done, post here please, I don't pretend to re-create the wheel...
I'd say that RemoveNulls is your problem. Suppose that you just read 100MB into a string that you passed to RemoveNulls. You would then allocate a string of length 1. The reallocate to length 2. Then to length 3. Then to length 4. And so on, all the way to length 100*1024*1024.
That process will fragment your memory, as well as being appallingly slow. Heap allocation is to be avoided when performance matters. You've no need for it at all. Read a chunk of the file, and search directly in the buffer that you read.
There are various problems with your code that I can see:
Your file extension check is broken, as I described in your previous question.
You are not handling exceptions correctly, as I described in your previous question.
Your for loop in RemoveNulls has buffer overrun. Loop from low() to high().
It's not possible to comment on the search code since that's not present in the question.
Short form: Is it possible to create a stream from a pointer?
I have a pointer which points to file data I need to read. I used WriteBuffer() to transfer data from the pointer to a TFileStream, which works. But now I want to read it block wise to display a progress bar. So I need to read it step by step.
I have to options:
1) Increment the pointer Buff, and use WriteBuffer() as usual. (WriteBuffer always reads from the beginning, therefore I need to increment the pointer)
2) Create a Stream from the pointer. Is this possible?
Code:
var InputStream : TMemoryStream;
Buff: Pointer;
Header: TOCHeader;
begin
// Here I get the pointer
Size := GetOverlay(Buff);
// Transfer data to memory stream
InputStream.WriteBuffer(Buff^, SizeOf(Header));
InputStream.Seek(0, soFromBeginning);
// Read the header
InputStream.ReadBuffer(Header, SizeOf(Header));
// Increment the pointer. Doesn't work :-(. Message Ordinal type required
Inc(Buff, SizeOf(TOC));
You could create your own TCustomMemoryStream descendant, which calls SetPointer() in the constructor, using the address and size that you receive from GetOverlay(). Consult the Delphi documentation of TCustomMemoryStream for further information.
Edit:
Well, this has nothing to do with your question, but in a comment you write that you only wish to read from the running executable file via a stream. That's easy enough to do:
procedure TForm1.Button1Click(Sender: TObject);
var
Str: TStream;
begin
Str := TFileStream.Create(ParamStr(0), fmOpenRead or fmShareDenyNone);
try
Caption := Format('Size of executable: %u bytes', [Str.Size]);
// start to read the contents of the file ...
finally
Str.Free;
end;
end;
I will leave the original answer, in case somebody really wants to create a (read-only) stream from a pointer to a chunk of memory.
A question comes up my mind when I read your question. You have a pointer to filedata (from disk I presume?) and you are filling a filestream with data from that pointer (although in your question you're filling a memorystream instead). Why not use a TFileStream to open the file you are reading?
var
FileStream: TFileStream;
Header: TOCHeader;
begin
FileStream := TFileStream.Create('c:\fileIWantToRead.txt', fmOpenRead);
FileStream.ReadBuffer(Header, SizeOf(Header));
<...>
end;
If you want to increment your pointer, then first cast it to a pointer of a type, so the compiler knows how to increase your pointer.
Inc(PByte(Buff), SizeOf(TOC)); //sizeof returns size in bytes, so cast pointer to a bytepointer
TResourceStream would do here...
I am loading a file into a array in binary form this seems to take a while
is there a better faster more efficent way to do this.
i am using a similar method for writing back to the file.
procedure openfile(fname:string);
var
myfile: file;
filesizevalue,i:integer;
begin
assignfile(myfile,fname);
filesizevalue:=GetFileSize(fname); //my method
SetLength(dataarray, filesizevalue);
i:=0;
Reset(myFile, 1);
while not Eof(myFile) do
begin
BlockRead(myfile,dataarray[i], 1);
i:=i+1;
end;
CloseFile(myfile);
end;
If your really want to read a binary file fast, let windows worry about buffering ;-) by using Memory Mapped Files. Using this you can simple map a file to a memory location an read like it's an array.
Your function would become:
procedure openfile(fname:string);
var
InputFile: TMappedFile;
begin
InputFile := TMappedFile.Create;
try
InputFile.MapFile(fname);
SetLength(dataarray, InputFile.Size);
Move(PByteArray(InputFile.Content)[0], Result[0], InputFile.Size);
finally
InputFile.Free;
end;
end;
But I would suggest not using the global variable dataarray, but either pass it as a var in the parameter, or use a function which returns the resulting array.
procedure ReadBytesFromFile(const AFileName : String; var ADestination : TByteArray);
var
InputFile : TMappedFile;
begin
InputFile := TMappedFile.Create;
try
InputFile.MapFile(AFileName);
SetLength(ADestination, InputFile.Size);
Move(PByteArray(InputFile.Content)[0], ADestination[0], InputFile.Size);
finally
InputFile.Free;
end;
end;
The TMappedFile is from my article Fast reading of files using Memory Mapping, this article also contains an example of how to use it for more "advanced" binary files.
You generally shouldn't read files byte for byte. Use BlockRead with a larger value (512 or 1024 often are best) and use its return value to find out how many bytes were read.
If the size isn't too large (and your use of SetLength seems to support this), you can also use one BlockRead call reading the complete file at once. So, modifying your approach, this would be:
AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);
Perhaps you could also change the procedure to a boolean function named OpenAndReadFile and return false if the file couldn't be opened or read.
It depends on the file format. If it consists of several identical records, you can decide to create a file of that record type.
For example:
type
TMyRecord = record
fieldA: integer;
..
end;
TMyFile = file of TMyRecord;
const
cBufLen = 100 * sizeof(TMyRecord);
var
file: TMyFile;
i : Integer;
begin
AssignFile(file, filename);
Reset(file);
i := 0;
try
while not Eof(file) do begin
BlockRead(file, dataarray[i], cBufLen);
Inc(i, cBufLen);
end;
finally
CloseFile(file);
end;
end;
If it's a long enough file that reading it this way takes a noticeable amount of time, I'd use a stream instead. The block read will be a lot faster, and there's no loops to worry about. Something like this:
procedure openfile(fname:string);
var
myfile: TFileStream;
filesizevalue:integer;
begin
filesizevalue:=GetFileSize(fname); //my method
SetLength(dataarray, filesizevalue);
myFile := TFileStream.Create(fname);
try
myFile.seek(0, soFromBeginning);
myFile.ReadBuffer(dataarray[0], filesizevalue);
finally
myFile.free;
end;
end;
It appears from your code that your record size is 1 byte long. If not, then change the read line to:
myFile.ReadBuffer(dataarray[0], filesizevalue * SIZE);
or something similar.
Look for a buffered TStream descendant. It will make your code a lot faster as the disk read is done fast, but you can loop through the buffer easily. There are various about, or you can write your own.
If you're feeling very bitheaded, you can bypass Win32 altogether and call the NT Native API function ZwOpenFile() which in my informal testing does shave a tiny bit off. Otherwise, I'd use Davy's Memory Mapped File solution above.