Short form: Is it possible to create a stream from a pointer?
I have a pointer which points to file data I need to read. I used WriteBuffer() to transfer data from the pointer to a TFileStream, which works. But now I want to read it block wise to display a progress bar. So I need to read it step by step.
I have to options:
1) Increment the pointer Buff, and use WriteBuffer() as usual. (WriteBuffer always reads from the beginning, therefore I need to increment the pointer)
2) Create a Stream from the pointer. Is this possible?
Code:
var InputStream : TMemoryStream;
Buff: Pointer;
Header: TOCHeader;
begin
// Here I get the pointer
Size := GetOverlay(Buff);
// Transfer data to memory stream
InputStream.WriteBuffer(Buff^, SizeOf(Header));
InputStream.Seek(0, soFromBeginning);
// Read the header
InputStream.ReadBuffer(Header, SizeOf(Header));
// Increment the pointer. Doesn't work :-(. Message Ordinal type required
Inc(Buff, SizeOf(TOC));
You could create your own TCustomMemoryStream descendant, which calls SetPointer() in the constructor, using the address and size that you receive from GetOverlay(). Consult the Delphi documentation of TCustomMemoryStream for further information.
Edit:
Well, this has nothing to do with your question, but in a comment you write that you only wish to read from the running executable file via a stream. That's easy enough to do:
procedure TForm1.Button1Click(Sender: TObject);
var
Str: TStream;
begin
Str := TFileStream.Create(ParamStr(0), fmOpenRead or fmShareDenyNone);
try
Caption := Format('Size of executable: %u bytes', [Str.Size]);
// start to read the contents of the file ...
finally
Str.Free;
end;
end;
I will leave the original answer, in case somebody really wants to create a (read-only) stream from a pointer to a chunk of memory.
A question comes up my mind when I read your question. You have a pointer to filedata (from disk I presume?) and you are filling a filestream with data from that pointer (although in your question you're filling a memorystream instead). Why not use a TFileStream to open the file you are reading?
var
FileStream: TFileStream;
Header: TOCHeader;
begin
FileStream := TFileStream.Create('c:\fileIWantToRead.txt', fmOpenRead);
FileStream.ReadBuffer(Header, SizeOf(Header));
<...>
end;
If you want to increment your pointer, then first cast it to a pointer of a type, so the compiler knows how to increase your pointer.
Inc(PByte(Buff), SizeOf(TOC)); //sizeof returns size in bytes, so cast pointer to a bytepointer
TResourceStream would do here...
Related
I am experiencing a strange error which I have narrowed down to a small piece of code (see below). I am reading a binary file into a TMemoryStream (variable ms1), through a TFileStream object (variable fs). Then I want to copy the binary data to another TMemoryStream object (ms2). This is what gives me the "stream read error" exception. Strange thing is, that if I don't load up the ms1 object with the file contents, things work fine, i.e. ms2.CopyFrom does not give me an exception..
Any help is greatly appreciated....
procedure TForm5.BitBtn1Click(Sender: TObject);
var
ms1: TMemoryStream;
fs: TFileStream;
ms2 : TMemoryStream;
FilePath: string;
begin
FilePath := 'C:\weekcpdf_tech6.bin';
ms1 := TMemoryStream.Create;
fs := nil;
try
ms1 := TMemoryStream.Create;
fs := TFileStream.Create(FilePath, fmOpenRead);
ms1.CopyFrom(fs, fs.Size);
ms2 := TMemoryStream.Create;
ms2.CopyFrom(ms1, ms1.Size);
finally
FreeAndNil(fs);
FreeAndNil(ms1);
end;
end;
First, you have a memory leak, since you create two memory stream objects that you assign to ms1. Remove the second one.
Second, after ms1.CopyFrom(fs, fs.Size); you must set ms1.Position := 0 so that you copy from the start of ms1 -- and not from the end of it.
This is actually fairly well documented:
CopyFrom copies Count bytes from the stream specified by Source into the stream. It then moves the current position by Count bytes and returns the number of bytes copied.
Hence, after ms1.CopyFrom(fs, fs.Size); you are at the end of ms1. Further,
If Count is greater than or less than 0, CopyFrom reads from the current position in Source.
Therefore, ms2.CopyFrom(ms1, ms1.Size); will read from the current position (= the end!) of ms1. Hence, you will likely try to read a lot of bytes that do not exist. And what happens then?
Because the CopyFrom method uses ReadBuffer and WriteBuffer to do the effective copying, if the Count is greater than the SourceStream size, ReadBuffer throws an exception stating that a stream read error has occured [sic!].
Always read the docs! :)
(Although I must admit that the last quote isn't quite perfect in this situation.)
I am porting my code from 10.1 to 10.2 and this gives me error:
procedure TForm4.FormCreate(Sender: TObject);
const
CFourBytes: array[0..3] of Byte = (1, 2, 3, 4);
var
LStream: TMemoryStream;
LBuffer: array of Byte;
begin
SetLength(LBuffer, 4);
LStream := TMemoryStream.Create;
LStream.Write(#CFourBytes[0], 4); // E2036 Variable required
LStream.Position := 0;
LStream.ReadData(#LBuffer[0], 4);
end;
I had to change offending line to LStream.Write(CFourBytes[0], 4);
What has changed? Have I been doing it wrong for the whole time?
The code in your question did compile in older versions, but it should not have done. The behaviour seen in 10.2 is correct.
What happens in old versions is very strange. The compiler selects this overload in TStream:
function Write(const Buffer: TBytes; Count: Longint): Longint; overload;
That is especially egregious because what has been passed to this method is the address of the static array CFourBytes. Which is categorically not a TBytes object.
Now it just so happens that a TBytes variable is the address of the first element of the array. And nothing in the TBytes override for TMemoryStream.Write refers to Length() of that bogus TBytes object. So your code happens to work as intended. This is very clearly a compiler error that has been fixed.
Your code has always been broken, you have just, up until now, been getting away with it by fortune. You should fix your code. Like this:
LStream := TMemoryStream.Create;
try
LStream.WriteBuffer(CFourBytes, SizeOf(CFourBytes));
SetLength(LBuffer, LStream.Size);
LStream.Position := 0;
LStream.ReadBuffer(LBuffer[0], LStream.Size);
finally
LStream.Free;
end;
Note that I am using WriteBuffer and ReadBuffer instead of Write and Read. These are the preferred methods to use with TStream. The reason being that they perform error checking and raise exceptions in case of errors, unlike Write and Read.
Perhaps nothing has been changed.
TStream.Write/Read methods always used untyped const/var parameter const Buffer (help) and using address of variable is wrong (because method (to be exact - compiler) finds address of variable itself).
Probably you accidentally confuse these methods with Read/WriteData ones that use typed parameter and one of overloaded versions gets Pointer type argument.
Here ReadData implementation dereferences this pointer and uses Read internally (Read in its turn calls Move and the last routine gets address of buffer again :) )
I can copy the memory from the buffer into the safe array as follows
function GetVarArrayFromBuffer(ABuffer : pByte; ASizeInBytes: Cardinal) : OleVariant;
var
LVarArrayPtr: Pointer;
begin
Result := VarArrayCreate([0, ASizeInBytes - 1], varByte);
LVarArrayPtr := VarArrayLock(Result);
try
Move(ABuffer^, LVarArrayPtr^, ASizeInBytes);
finally
VarArrayUnLock(Result);
end;
end;
But, is there a way to directly pass my pointer and size into a varArray type OleVariant without copying memory?
[Edit]
I can see that the array inside the OleVariant is a SAFEARRAY (defined as PVarArray = ^TVarArray), so it seems like there should be a way to do this by populating the values in a TVarArray and setting the VType and VArray values in the OleVariant.
is there a way to directly pass my pointer and size into a varArray type OleVariant without copying memory?
Delphi's OleVariant type is a wrapper for OLE's VARIANT record. The only type of array that OLE supports is SAFEARRAY, and any SAFEARRAY created by a Win32 SafeArrayCreate...() function allocates and owns the data block that it points to. You have to copy your source data into that block.
To bypass that, you would have to skip VarArrayCreate() (which calls SafeArrayCreate()) and allocate the SAFEARRAY yourself using SafeArrayAllocDescriptor/Ex() so it does not allocate a data block. Then you can set the array's pvData field to point at your existing memory block, and enable the FADF_AUTO flag in its fFeatures field to tell SafeArrayDestroy() (which OleVariant calls when it does not need the SAFEARRAY anymore) to not free your memory block.
Try something like this:
uses
..., Ole2, ComObj;
// Delphi's Ole2 unit declares SafeArrayAllocDescriptor()
// but does not declare SafeArrayAllocDescriptorEx()...
function SafeArrayAllocDescriptorEx(vt: TVarType; cDims: Integer; var psaOut: PSafeArray): HResult; stdcall; external 'oleaut32.dll';
function GetVarArrayFromBuffer(ABuffer : pByte; ASizeInBytes: Cardinal) : OleVariant;
var
SA: PSafeArray;
begin
OleCheck(SafeArrayAllocDescriptorEx(VT_UI1, 1, SA));
SA.fFeatures := SA.fFeatures or FADF_AUTO or FADF_FIXEDSIZE;
SA.cbElements := SizeOf(Byte);
SA.pvData := ABuffer;
SA.rgsabound[0].lLbound := 0;
SA.rgsabound[0].cElements := ASizeInBytes;
TVarData(Result).VType := varByte or varArray;
TVarData(Result).VArray := PVarArray(SA);
end;
If you don't actually need to use OLE, such as if you are not passing your array to other people's applications via OLE, then you should use Delphi's Variant type instead. You can write a Custom Variant Type to hold whatever data you want, even a reference to your existing memory block, and then use Variant as needed and let your custom type implementation manage the data as needed.
You may be able to hack your way into having an OleVariant with your array data in it without copying it.
However, a problem you are going to have is when the OleVariant variable falls out of scope.
The RTL is going to call SafeArrayDestroy in oleaut32.dll to destroy the memory associated with the safe array, and that's going to fail because the memory did not come from where Windows expected.
How do I convert contents of a TMemoryStream to a variant? I use Delphi 2010.
TMemoryStream stores contents of a file, it can be PDF or JPG (scanned document).
File is being kept inside MS SQL base.
When I go to editing mode in my program, I extract contents of that file from base into a TMemoryStream.
After editing document's card, I need to post document back to base.
Scanned file could be changed also (or replaced with some other file).
To post record back, I use a stored procedure with a bunch of parameters - one for every field.
I pass parameters to stored procedure as variants.
That's why I need to convert TMemoryStream to a variant.
Assuming you need the Variant to hold an array of bytes, you can use this:
var
MS: TMemoryStream;
V: Variant;
P: Pointer;
begin
...
V := VarArrayCreate([0, MS.Size-1], varByte);
if MS.Size > 0 then
begin
P := VarArrayLock(V);
Move(MS.Memory^, P^, MS.Size);
VarArrayUnlock(V);
end;
...
end;
TMemoryStream doesn't have a convenient way to get direct access to the internal data. It provides a property that gives you a pointer, but not any useful data type. However, if you use TBytesStream, which derives from TMemoryStream, you can get the data from the stream as a variable of type TBytes.
After this, assuming your parameter is a standard parameter object of type TParam, you don't need to use a variant. You can do it like this:
param.AsBlob := MyTBytesVariable;
Or, even simpler than that, you can use the stream directly:
param.AsStream := MyMemoryStream;
(If you do this, make sure that the stream's Position is set to 0 first.)
I am loading a file into a array in binary form this seems to take a while
is there a better faster more efficent way to do this.
i am using a similar method for writing back to the file.
procedure openfile(fname:string);
var
myfile: file;
filesizevalue,i:integer;
begin
assignfile(myfile,fname);
filesizevalue:=GetFileSize(fname); //my method
SetLength(dataarray, filesizevalue);
i:=0;
Reset(myFile, 1);
while not Eof(myFile) do
begin
BlockRead(myfile,dataarray[i], 1);
i:=i+1;
end;
CloseFile(myfile);
end;
If your really want to read a binary file fast, let windows worry about buffering ;-) by using Memory Mapped Files. Using this you can simple map a file to a memory location an read like it's an array.
Your function would become:
procedure openfile(fname:string);
var
InputFile: TMappedFile;
begin
InputFile := TMappedFile.Create;
try
InputFile.MapFile(fname);
SetLength(dataarray, InputFile.Size);
Move(PByteArray(InputFile.Content)[0], Result[0], InputFile.Size);
finally
InputFile.Free;
end;
end;
But I would suggest not using the global variable dataarray, but either pass it as a var in the parameter, or use a function which returns the resulting array.
procedure ReadBytesFromFile(const AFileName : String; var ADestination : TByteArray);
var
InputFile : TMappedFile;
begin
InputFile := TMappedFile.Create;
try
InputFile.MapFile(AFileName);
SetLength(ADestination, InputFile.Size);
Move(PByteArray(InputFile.Content)[0], ADestination[0], InputFile.Size);
finally
InputFile.Free;
end;
end;
The TMappedFile is from my article Fast reading of files using Memory Mapping, this article also contains an example of how to use it for more "advanced" binary files.
You generally shouldn't read files byte for byte. Use BlockRead with a larger value (512 or 1024 often are best) and use its return value to find out how many bytes were read.
If the size isn't too large (and your use of SetLength seems to support this), you can also use one BlockRead call reading the complete file at once. So, modifying your approach, this would be:
AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);
Perhaps you could also change the procedure to a boolean function named OpenAndReadFile and return false if the file couldn't be opened or read.
It depends on the file format. If it consists of several identical records, you can decide to create a file of that record type.
For example:
type
TMyRecord = record
fieldA: integer;
..
end;
TMyFile = file of TMyRecord;
const
cBufLen = 100 * sizeof(TMyRecord);
var
file: TMyFile;
i : Integer;
begin
AssignFile(file, filename);
Reset(file);
i := 0;
try
while not Eof(file) do begin
BlockRead(file, dataarray[i], cBufLen);
Inc(i, cBufLen);
end;
finally
CloseFile(file);
end;
end;
If it's a long enough file that reading it this way takes a noticeable amount of time, I'd use a stream instead. The block read will be a lot faster, and there's no loops to worry about. Something like this:
procedure openfile(fname:string);
var
myfile: TFileStream;
filesizevalue:integer;
begin
filesizevalue:=GetFileSize(fname); //my method
SetLength(dataarray, filesizevalue);
myFile := TFileStream.Create(fname);
try
myFile.seek(0, soFromBeginning);
myFile.ReadBuffer(dataarray[0], filesizevalue);
finally
myFile.free;
end;
end;
It appears from your code that your record size is 1 byte long. If not, then change the read line to:
myFile.ReadBuffer(dataarray[0], filesizevalue * SIZE);
or something similar.
Look for a buffered TStream descendant. It will make your code a lot faster as the disk read is done fast, but you can loop through the buffer easily. There are various about, or you can write your own.
If you're feeling very bitheaded, you can bypass Win32 altogether and call the NT Native API function ZwOpenFile() which in my informal testing does shave a tiny bit off. Otherwise, I'd use Davy's Memory Mapped File solution above.