Delphi 2010 : How to emulate the Delphi XE TStrings.Encoding property? - delphi

Question

Delphi XE added an Encoding property to the TStrings class, which stores the encoding read from the BOM when LoadFromFile() is called.
Delphi 2010 does not have this property.
I would like to emulate it.
I created the below class helper for TStrings.
The helper works, but to get the file's BOM, the only solution I have found is to reload the same file in a FileStream. I would like to avoid this since TStrings.LoadFromFile() already got the BOM.
How can I tell the helper to re-use the BOM that was already found?
unit TestEncodingName_00;
interface
uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, StdCtrls, ExtDlgs;
type
TForm1 = class(TForm)
Memo1: TMemo;
procedure FormCreate(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
TMyStrings = class helper for TStrings // emulate TStrings.Encoding
private
function GetEncodingName(fPath:string):string;
public
property EncodingName[fPath:string]:string read GetEncodingName;
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
function TMyStrings.GetEncodingName(fPath:string):string;
var
fLen : integer;
fBuffer : TBytes;
fEncoding : TEncoding;
fName : string;
fFs : TFileStream;
begin
fFs := TFileStream.Create(fPath, fmOpenRead);
try
SetLength(fBuffer, 4);
flen := fFs.Read(fBuffer[0], 4);
if flen < 4 then
SetLength(fBuffer, flen);
fEncoding := nil;
TEncoding.GetBufferEncoding(fBuffer, fEncoding);
if fEncoding = TEncoding.Unicode then
fName := 'Unicode'
else if fEncoding = TEncoding.UTF8 then
fName := 'UTF8'
else fName := 'Default';
finally
fFs.Free;
end;
result := fName;
end;
procedure TForm1.FormCreate(Sender: TObject);
var
EncName : string;
begin
(* sample UTF8.txt
Ā ā Ă ă
Ρ Σ Τ Υ
ぁ あ ぃ
*)
Memo1.Lines.LoadFromFile('Sample UTF8.txt');
//from here TStrings knows the BOM but I don't know
// how to refer to it...
// so I have to create again a stream of the same file to
// get the BOM. I don't like that.
EncName := Memo1.Lines.EncodingName['Sample UTF8.txt'];
Memo1.Lines.Add(#13#10'Encoding : ' + EncName);
end;
end.

Answer 1

First off, it is LoadFromStream() that discovers the BOM encoding, not LoadFromFile(). LoadFromFile() simply opens the file into a TFileStream and then calls LoadFromStream().
In Delphi (2009 and) 2010, the discovered BOM encoding is not stored anywhere that you can access. That is the very problem that XE solved by adding the new Encoding property. The encoding is only used as a local variable inside of LoadFromStream() when decoding the file data to a UnicodeString prior to parsing, and then it gets discarded when LoadFromStream() exits. There is nothing you can do to change that behavior.
So, the only solution is to load the file manually so you can discover its BOM. Ideally, you would override LoadFromStream() in a descendant class, but you can't make TMemo.Lines use a custom class. And a class helper cannot override virtual methods, either.
However, you can derive a custom class from TStringList to override LoadFromStream(), load the file yourself, and then Assign() the TStringList to TMemo.Lines. For example:
unit TestEncodingName_00;
interface
uses
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, StdCtrls, ExtDlgs;
type
TForm1 = class(TForm)
Memo1: TMemo;
procedure FormCreate(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
TMyStringList = class(TStringList)
private
fEncoding: TEncoding;
public
{ The single-parameter LoadFromStream(Stream: TStream) simply
calls LoadFromStream(Stream: TStream; Encoding: TEncoding) with
the Encoding parameter set to nil, so you only have to override
that version of LoadFromStream()... }
procedure LoadFromStream(Stream: TStream; Encoding: TEncoding); override;
property Encoding: TEncoding read fEncoding;
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
procedure TMyStringList.LoadFromStream(Stream: TStream; Encoding: TEncoding);
var
Size: Integer;
Buffer: TBytes;
begin
{ this is similar to the implementation that LoadFromStream()
uses in XE+, but with some differences:
1. the Encoding property is assigned a bit differently, as XE+
utilizes a TEncoding.Clone() method when the specified Encoding
is not a standard RTL encoding (ie, is a custom user class), but
Clone() does not exist in D2009/2010.
2. XE+ has a TStrings.DefaultEncoding property, which is passed
to TEncoding.GetBufferEncoding() as the default to return if no
BOM is detected, but that property and parameter do not exist in
D2009/2010.
3. TStrings.LoadFromStream() does not check if Size is 0 (file is empty)
before dereferencing the Buffer that is passed to Stream.Read().
That is a runtime crash waiting to happen! }
BeginUpdate;
try
Size := Stream.Size - Stream.Position;
SetLength(Buffer, Size);
if Size > 0 then
Stream.Read(Buffer[0], Size);
Size := TEncoding.GetBufferEncoding(Buffer, Encoding);
fEncoding := Encoding;
SetTextStr(Encoding.GetString(Buffer, Size, Length(Buffer) - Size));
finally
EndUpdate;
end;
end;
procedure TForm1.FormCreate(Sender: TObject);
var
EncName : string;
List: TMyStringList;
begin
List := TMyStringList.Create;
try
List.LoadFromFile('Sample UTF8.txt');
if List.Encoding = TEncoding.Unicode then
EncName := 'Unicode'
else if List.Encoding = TEncoding.UTF8 then
EncName := 'UTF8'
else
EncName := 'Default';
Memo1.Lines.Assign(List);
Memo1.Lines.Add;
Memo1.Lines.Add('Encoding : ' + EncName);
finally
List.Free;
end;
end;
end.