I was trying to read data from a TBlobField using TADOBlobStream. I wrote the following function
function DecompressBlobFieldCustom(AField:TBlobField):String;
var
BLOBStream:TADOBlobStream;
Size:Integer;
begin
BLOBStream:= TADOBlobStream.Create(AField,bmRead);
Size:= BLOBStream.Size;
BLOBStream.Read(Result,Size);
end;
and i use the function as follows
Data := DecompressBlobFieldCustom(FldImage);
But when i try to do if Length(Data) > 0 then i am getting Access violation at address XXXX error. I couldn't figure out the problem. Please Help.
Result is a string type, but a blob stream operates on byte arrays. It is usually a mistake to try to treat a byte array as though it were a string. Furthermore, you did not allocate a buffer into which to read, which is the actual cause of the error.
Read into a byte array like this:
function ReadBlobField(Field: TBlobField): TBytes;
var
Stream: TStream;
begin
Stream := TADOBlobStream.Create(Field, bmRead);
try
SetLength(Result, Stream.Size);
if Stream.Size>0 then
Stream.ReadBuffer(Result[0], Stream.Size);
finally
Stream.Free;
end;
end;
It is preferable to use the CreateBlobStream method of TDataSet to create blob streams, as discussed in the documentation. So the code would be better like this:
function ReadBlobField(DataSet: TDataSet; Field: TField): TBytes;
var
Stream: TStream;
begin
Stream := DataSet.CreateBlobStream(Field, bmRead);
try
SetLength(Result, Stream.Size);
if Stream.Size>0 then
Stream.ReadBuffer(Result[0], Stream.Size);
finally
Stream.Free;
end;
end;
I've assumed that the data really is a byte array and is not holding text. The fact that it is held in a blob suggests that, as does the mention of decompression and images.
Some other comments:
You must destroy the stream when you are finished with it.
It is generally preferable to use ReadBuffer rather than Read because ReadBuffer raises an exception if the requested number of bytes are not read.
Related
In Delphi 10.4.2, when I use the TWriter.WriteString two extra bytes are saved :
var
FileStream: TFileStream;
Writer: TWriter;
begin
FileStream := TFileStream.Create('stream.txt', fmCreate or fmOpenWrite or fmShareDenyNone);
Writer := TWriter.Create(FileStream, $FF);
try
Writer.WriteString('2');
finally
Writer.Free;
FileStream.free;
end
end;
What are these two bytes? How can I ignore them?
This is by design of TWriter.WriteString. Probably your use case is not the correct one.
The first byte ($06) is the value type (TValueType.vaString for your code). The second byte is the length of the string (1 byte for you).
You can find all that information in the source code provided by Embarcadero in file System.Classes.pas.
You cannot ignore them. Maybe you can use TStream.Write to write your string without extra payload?
Delphi Tokyo - I have a parameter file that I am needing to save (and later load) from disk. The parameters are a series of record objects. There is one HEADER record and then multiple COMMAND records. These are true records (i.e type = records). The HEADER record has String, Boolean, Integer, and TStringList types within it. I save, which appears to work fine, but when I load, whatever is AFTER a TStringList causes a Stream read error. For example...
type tEDP_PROJ = record
Version : Integer;
Name: String;
...
ColList1: TStringList;
ColList2: TStringList;
ReadyToRun : Boolean;
...
end;
When I read ReadyToRun I get a Stream read error. If I move it BEFORE TStringList (on both SAVE and LOAD routines) then ReadyToRun will load properly, but whatever is after the TStringList will cause an error. It is interesting to note that ColList2 loads fine (even though it is NOT the first TStringList).
I am specifying the Encoding method when I save the TStringList.
...
ColList1.SaveToStream(SavingStream, TEncoding.Unicode);
ColList2.SaveToStream(SavingStream, TEncoding.Unicode);
I am using the same encoding when I load from the (file) Stream.
...
ColList1.LoadFromStream(SavingStream, TEncoding.Unicode);
ColList2.LoadFromStream(SavingStream, TEncoding.Unicode);
Note that when I create the StringList, I am just doing the standard create...
ColList1 := TStringList.Create;
When I save and load, I am following the examples Remy gave here...
The TStringList appears to be changing the way that the stream reads non-TStringList types... What do I need to do to fix this?
Why are you using TEncoding.Unicode? TEncoding.UTF8 would have made more sense.
In any case, this is not an encoding issue. What you are attempting to do will simply not work the way you are trying to do it, because TStrings data is variable-length and needs to be handled accordingly. However, TStrings does not save any kind of terminating delimiter or size information to an output stream. When loading in a stream, TStrings.LoadFromStream() simply reads the ENTIRE stream (well, everything between the current Position and the End-Of-Stream, anyway). That is why you are getting streaming errors when trying to read/write any non-TStrings data after any TStrings data.
Just like the earlier code needed to serialize String data and other variable-length data into a flat format to know where one field ends and the next begins, you need to serialize TStrings data as well.
One option is to save a TStrings object to an intermediate TMemoryStream first, then write that stream's Size to your output stream followed by the TMemoryStream's data. When loading back later, first read the Size, then read the specified number of bytes into an intermediate TMemoryStream, and then load that stream into your receiving TStrings object:
procedure WriteInt64ToStream(Stream: TStream; Value: Int64);
begin
Stream.WriteBuffer(Value, Sizeof(Value));
end;
function ReadInt64FromStream(Stream: TStream): Int64;
begin
Stream.ReadBuffer(Result, Sizeof(Result));
end;
procedure WriteStringsToStream(Stream: TStream; Values: TStrings);
var
MS: TMemoryStream;
Size: Int64;
begin
MS := TMemoryStream.Create;
try
Values.SaveToStream(MS, TEncoding.UTF8);
Size := MS.Size;
WriteInt64ToStream(Stream, Size);
if Size > 0 then
begin
MS.Position := 0;
Stream.CopyFrom(MS, Size);
end;
finally
MS.Free;
end;
end;
procedure ReadStringsFromStream(Stream: TStream; Values: TStrings);
var
MS: TMemoryStream;
Size: Int64;
begin
Size := ReadInt64FromStream(Stream);
MS := TMemoryStream.Create;
try
if Size > 0 then
begin
MS.CopyFrom(Stream, Size);
MS.Position := 0;
end;
Values.LoadFromStream(MS, TEncoding.UTF8);
finally
MS.Free;
end;
end;
Another option is to write the number of string elements in the TStrings object to your output stream, and then write the individual strings:
procedure WriteStringsToStream(Stream: TStream; Values: TStrings);
var
Count, I: Integer;
begin
Count := Values.Count;
WriteIntegerToStream(Stream, Count);
for I := 0 to Count-1 do
WriteStringToStream(Stream, Values[I]);
end;
procedure ReadStringsFromStream(Stream: TStream; Values: TStrings);
var
Count, I: Integer;
begin
Count := ReadIntegerFromStream(Stream);
if Count > 0 then
begin
Values.BeginUpdate;
try
for I := 0 to Count-1 do
Values.Add(ReadStringFromStream(Stream));
finally
Values.EndUpdate;
end;
end;
end;
Either way, you can then do this when streaming your individual records:
WriteIntegerToStream(SavingStream, Version);
WriteStringToStream(SavingStream, Name);
...
WriteStringsToStream(SavingStream, ColList1);
WriteStringsToStream(SavingStream, ColList2);
WriteBooleanToStream(SavingStream, ReadyToRun);
Version := ReadIntegerFromStream(SavingStream);
Name := ReadStringFromStream(SavingStream);
...
ReadStringsFromStream(SavingStream, ColList1);
ReadStringsFromStream(SavingStream, ColList2);
ReadyToRun := ReadBooleanFromStream(SavingStream);
I have this code, where I am reading buffers to memory stream and I want to save it to file stream. The problem is that memoryStream.GetBuffer() does not work for my Delphi 7 as it is undeclared identifier.
procedure Dictionary.WriteHeaderObj(filename: String);
var MemStream: TMemoryStream;
i: Integer;
begin
self.fileName := filename;
try
MemStream := TMemoryStream.Create;
try
fsOutput := TFileStream.Create(fileName, fmCreate);
try
MemStream.Write(VAR_META, lengths.VR_META);
for i:=0 to length(buffers)-1 do
MemStream.Write(self.buffers[i].b^,self.buffers[i].l^);
fsOutput.Write(MemStream.GetBuffer(), 0, memoryStream.Position);
finally
MemStream.Free;
end;
finally
fsOutput.Free;
end;
except
on E:Exception do
Writeln(E.Classname, ': ', E.Message);
end;
end;
Your code showed several mistakes common to beginners.
There really is no need for the TMemoryStream. As David said, it hurts performance.
It's also not a good idea to write everything to a single TMemoryStream, because you may ran into "Out of memory" problem.
If the buffer size is really small and using a memory buffer is beneficial, you should do it in two loops - the inner loop writes a certain number of data to the memory stream and the out loop copies the memory stream to the file stream and empty the memory stream.
Unfortunately Capacity (protected) of TMemoryStream is useless because TMemoryStream reallocates memory whenever Size is changed. Using a sufficiently big TBytes avoids memory reallocation but you need to keep track of the length.
Why did you just write the header object to the file? Do you intend to append the body to the file later? Append a file is usually a bad idea.
Some thing trivial: always pass string constants with const.
Some thing trivial: always start a type name with 'T'.
Here is how I would do it:
procedure TDictionary.SaveHeaderToStream(AStream: TStream);
// You want to write to a stream, not a file. This is more reusable.
// You don't want to put the create/free code here,
// this is more flexible as it can also be used by SaveToStream
var
i: Integer;
begin
for i := 0 to length(buffers) - 1 do
AStream.Write(self.buffers[i].b^, self.buffers[i].l^);
end;
procedure TDictionary.SaveHeaderToFile(const Filename: string);
var
fsOutput: TFileStream;
begin
fsOutput := TFileStream.Create(Filename, fmCreate);
try
SaveHeaderToStream(fsOutput); // fsOutput IS TStream
finally
fsOutput.Free;
end;
end;
procedure TDictionary.SaveBodyToStream(AStream: TStream);
begin
// Codes here
end;
procedure TDictionary.SaveToStream(AStream: TStream);
begin
SaveHeaderToStream(AStream);
SaveBodyToStream(AStream);
end;
procedure TDictionary.SaveToFile(const Filename: string);
var
fsOutput: TFileStream;
begin
fsOutput := TFileStream.Create(Filename, fmCreate);
try
SaveToStream(fsOutput);
finally
fsOutput.Free;
end;
end;
okay, so I (VERY) recently started playing with lazaruz/free pascal, and I'm a little stuck with reading files with TMemoryStream and it's streaming kin.
I'm trying to write a simple base64 encoder, that can encode strings of text, or files (like images and WAVs) to then be used in html and javascript.
The following code compiles great but I get EReadError Illegal stream image when trying to load a file. I'll include the working string only procedure for reference:
procedure TForm1.TextStringChange(Sender: TObject);
begin
Memo1.Lines.Text := EncodeStringBase64(TextString.Text);
end;
procedure TForm1.FormCreate(Sender: TObject);
begin
Memo1.Lines.Text := '';
Form1.BorderIcons := [biSystemMenu,biMinimize];
end;
procedure TForm1.BitBtn1Click(Sender: TObject);
begin
if OpenDialog1.Execute then
begin
filename := OpenDialog1.Filename;
stream := TMemoryStream.Create;
try
StrStream := TStringStream.Create(s);
try
stream.LoadFromFile(filename);
stream.Seek(0, soFromBeginning);
ObjectBinaryToText(stream, StrStream);
StrStream.Seek(0, soFromBeginning);
Memo1.Lines.Text := EncodeStringBase64(StrStream.DataString);
finally
StrStream.Free;
end;
finally
stream.Free;
end;
end;
end;
Can anyone help me out?
You get the "illegal stream image" exception because the file you're loading probably isn't a binary DFM file. That's what ObjectBinaryToText is meant to process. It's not for arbitrary data. So get rid of that command.
You can skip the TMemoryStream, too. TStringStream already has a LoadFromFile method, so you can call it directly instead of involving another buffer.
StrStream.LoadFromFile(filename);
But a string isn't really the right data structure to store your file in prior to base64-encoding it. The input to base64 encoding is binary data; the output is text. Using a text data structure as an intermediate format means you may introduce errors into your data because of difficulties in encoding certain data as valid characters. The right interface for your encoding function is this:
function Base64Encode(Data: TStream): string;
You don't need to load the entire file into memory prior to encoding it. Just open the file with a TFileStream and pass it to your encoding function. Read a few bytes from it at a time with the stream's Read method, encode them as base64, and append them to the result string. (If you find that you need them, you can use an intermediate TStringBuilder for collecting the result, and you can add different buffering around the file reads. Don't worry about those right away, though; get your program working correctly first.)
Use it something like this:
procedure TForm1.BitBtn1Click(Sender: TObject);
var
filename: string;
stream: TStream;
begin
if OpenDialog1.Execute then begin
filename := OpenDialog1.Filename;
stream := TFileStream.Create(filename, fmOpenRead);
try
Memo1.Lines.Text := Base64Encode(stream);
finally
stream.Free;
end;
end;
end;
I never heard before about ObjectBinaryToText(), but looks like funky one. Also, what is EncodeStringBase64() function?
At first place, you shouldn't convert binary stream to text to encode it, instead you should directly B64 encode binary data. B64 algorithm is intended to work on array of bytes.
Since Delphi 6, there is EncdDecd.pas unit, which implements B64 encoding methods. I'm not sure if Lazarus/FPC have this, but if they do, your code to B64 encode file should look like this (add EncdDecd to uses list):
procedure TForm1.Button1Click(Sender: TObject);
var
instream : TFileStream;
outstream: TStringStream;
begin
if OpenDialog1.Execute then
begin
instream := TFileStream.Create(OpenDialog1.FileName, fmOpenRead or fmShareDenyNone);
try
outstream := TStringStream.Create;
try
EncodeStream(instream, outstream);
Memo1.Lines.Text := outstream.DataString;
finally
outstream.Free;
end;
finally
instream.Free;
end;
end;
end;
i want to process a text file line by line. In the olden days i loaded the file into a StringList:
slFile := TStringList.Create();
slFile.LoadFromFile(filename);
for i := 0 to slFile.Count-1 do
begin
oneLine := slFile.Strings[i];
//process the line
end;
Problem with that is once the file gets to be a few hundred megabytes, i have to allocate a huge chunk of memory; when really i only need enough memory to hold one line at a time. (Plus, you can't really indicate progress when you the system is locked up loading the file in step 1).
The i tried using the native, and recommended, file I/O routines provided by Delphi:
var
f: TextFile;
begin
Reset(f, filename);
while ReadLn(f, oneLine) do
begin
//process the line
end;
Problem withAssign is that there is no option to read the file without locking (i.e. fmShareDenyNone). The former stringlist example doesn't support no-lock either, unless you change it to LoadFromStream:
slFile := TStringList.Create;
stream := TFileStream.Create(filename, fmOpenRead or fmShareDenyNone);
slFile.LoadFromStream(stream);
stream.Free;
for i := 0 to slFile.Count-1 do
begin
oneLine := slFile.Strings[i];
//process the line
end;
So now even though i've gained no locks being held, i'm back to loading the entire file into memory.
Is there some alternative to Assign/ReadLn, where i can read a file line-by-line, without taking a sharing lock?
i'd rather not get directly into Win32 CreateFile/ReadFile, and having to deal with allocating buffers and detecting CR, LF, CRLF's.
i thought about memory mapped files, but there's the difficulty if the entire file doesn't fit (map) into virtual memory, and having to maps views (pieces) of the file at a time. Starts to get ugly.
i just want Reset with fmShareDenyNone!
With recent Delphi versions, you can use TStreamReader. Construct it with your file stream, and then call its ReadLine method (inherited from TTextReader).
An option for all Delphi versions is to use Peter Below's StreamIO unit, which gives you AssignStream. It works just like AssignFile, but for streams instead of file names. Once you've used that function to associate a stream with a TextFile variable, you can call ReadLn and the other I/O functions on it just like any other file.
You can use this sample code:
TTextStream = class(TObject)
private
FHost: TStream;
FOffset,FSize: Integer;
FBuffer: array[0..1023] of Char;
FEOF: Boolean;
function FillBuffer: Boolean;
protected
property Host: TStream read FHost;
public
constructor Create(AHost: TStream);
destructor Destroy; override;
function ReadLn: string; overload;
function ReadLn(out Data: string): Boolean; overload;
property EOF: Boolean read FEOF;
property HostStream: TStream read FHost;
property Offset: Integer read FOffset write FOffset;
end;
{ TTextStream }
constructor TTextStream.Create(AHost: TStream);
begin
FHost := AHost;
FillBuffer;
end;
destructor TTextStream.Destroy;
begin
FHost.Free;
inherited Destroy;
end;
function TTextStream.FillBuffer: Boolean;
begin
FOffset := 0;
FSize := FHost.Read(FBuffer,SizeOf(FBuffer));
Result := FSize > 0;
FEOF := Result;
end;
function TTextStream.ReadLn(out Data: string): Boolean;
var
Len, Start: Integer;
EOLChar: Char;
begin
Data:='';
Result:=False;
repeat
if FOffset>=FSize then
if not FillBuffer then
Exit; // no more data to read from stream -> exit
Result:=True;
Start:=FOffset;
while (FOffset<FSize) and (not (FBuffer[FOffset] in [#13,#10])) do
Inc(FOffset);
Len:=FOffset-Start;
if Len>0 then begin
SetLength(Data,Length(Data)+Len);
Move(FBuffer[Start],Data[Succ(Length(Data)-Len)],Len);
end else
Data:='';
until FOffset<>FSize; // EOL char found
EOLChar:=FBuffer[FOffset];
Inc(FOffset);
if (FOffset=FSize) then
if not FillBuffer then
Exit;
if FBuffer[FOffset] in ([#13,#10]-[EOLChar]) then begin
Inc(FOffset);
if (FOffset=FSize) then
FillBuffer;
end;
end;
function TTextStream.ReadLn: string;
begin
ReadLn(Result);
end;
Usage:
procedure ReadFileByLine(Filename: string);
var
sLine: string;
tsFile: TTextStream;
begin
tsFile := TTextStream.Create(TFileStream.Create(Filename, fmOpenRead or fmShareDenyWrite));
try
while tsFile.ReadLn(sLine) do
begin
//sLine is your line
end;
finally
tsFile.Free;
end;
end;
If you need support for ansi and Unicode in older Delphis, you can use my GpTextFile or GpTextStream.
As it seems the FileMode variable is not valid for Textfiles, but my tests showed that multiple reading from the file is no problem. You didn't mention it in your question, but if you are not going to write to the textfile while it is read you should be good.
What I do is use a TFileStream but I buffer the input into fairly large blocks (e.g. a few megabytes each) and read and process one block at a time. That way I don't have to load the whole file at once.
It works quite quickly that way, even for large files.
I do have a progress indicator. As I load each block, I increment it by the fraction of the file that has additionally been loaded.
Reading one line at a time, without something to do your buffering, is simply too slow for large files.
I had same problem a few years ago especially the problem of locking the file. What I did was use the low level readfile from the shellapi. I know the question is old since my answer (2 years) but perhaps my contribution could help someone in the future.
const
BUFF_SIZE = $8000;
var
dwread:LongWord;
hFile: THandle;
datafile : array [0..BUFF_SIZE-1] of char;
hFile := createfile(PChar(filename)), GENERIC_READ, FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING, FILE_ATTRIBUTE_READONLY, 0);
SetFilePointer(hFile, 0, nil, FILE_BEGIN);
myEOF := false;
try
Readfile(hFile, datafile, BUFF_SIZE, dwread, nil);
while (dwread > 0) and (not myEOF) do
begin
if dwread = BUFF_SIZE then
begin
apos := LastDelimiter(#10#13, datafile);
if apos = BUFF_SIZE then inc(apos);
SetFilePointer(hFile, aPos-BUFF_SIZE, nil, FILE_CURRENT);
end
else myEOF := true;
Readfile(hFile, datafile, BUFF_SIZE, dwread, nil);
end;
finally
closehandle(hFile);
end;
For me the speed improvement appeared to be significant.
Why not simply read the lines of the file directly from the TFileStream itself one at a time ?
i.e. (in pseudocode):
readline:
while NOT EOF and (readchar <> EOL) do
appendchar to result
while NOT EOF do
begin
s := readline
process s
end;
One problem you may find with this is that iirc TFileStream is not buffered so performance over a large file is going to be sub-optimal. However, there are a number of solutions to the problem of non-buffered streams, including this one, that you may wish to investigate if this approach solves your initial problem.