My simple code is:
var
TMPStream : TStringStream;
myencoding: TEncoding;
...
try
myencoding := TEncoding.GetEncoding(CP_UTF8);
TMPStream := TStringStream.Create('', myencoding);
try
(IBQueryTMP.FieldByName('MYTEXT') as TBlobField).SaveToStream(TMPStream);
TMPStream.SaveToFile(ExtractFilePath(Application.ExeName)+'myfile.txt');
except
Showmessage ('Error');
end;
finally
TMPStream.Free;
myencoding.Free;
end;
As I see in the file (for instance, in Notepad++), the codepage is UTF-16 Little Endian, although it was declared UTF-8. The database is UTF-8, too.
What's wrong?
Related
I want to achieve a very very basic task in Delphi: to save a string to disk and load it back. It seems trivial but I had problems doing this TWICE since I upgraded to IOUtils (and one more time before that... this is why I took the 'brilliant' decision to upgrade to IOUtils).
I use something like this:
procedure WriteToFile(CONST FileName: string; CONST uString: string; CONST WriteOp: WriteOperation);
begin
if WriteOp= (woOverwrite)
then IOUtils.TFile.WriteAllText (FileName, uString) //overwrite
else IOUtils.TFile.AppendAllText(FileName, uString); //append
end;
Simple right? What could go wrong? Well, I recently stepped into a (another) bug in IOUtils. So, TFile is buggy. The bug is detailed here.
Anyone has can share an alternative (or simply your thoughts/ideas) that is not based on IOUtils and it is known to work? Well... the code above also worked for a while for me... So, I know if difficult to guaranty that a piece of code (no matter how small) will really work!
Also I would REALLY like to have my WriteToFile procedure to save the string to an ANSI file when it is possible (the uString contains only ANSI chars) and as Unicode otherwise.
Then the ReadAFile function should automagically detect the encoding and correctly read the string back.
The idea is that there are still text editors out there that will wrongly open/interpret an Unicode/UTF file. So, whenever possible, give a good old ANSI text file to the user.
So:
- Overwrite/Append
- Save as ANSI when possible
- Memory efficient (don't eat 4GB of ram when the file to load is 2GB)
- Should work with any text file (up to 2GB, obviously)
- No IOUtils (too buggy to be of use)
Then the ReadAFile function should automagically detect the encoding and correctly read the string back.
This is not possible. There exists files that are well-formed if interpreted as any text encoding. For instance see The Notepad file encoding problem, redux.
This means that your goals are unattainable and that you need to change them.
My advice is to do the following:
Pick a single encoding, UTF-8, and stick to it.
If the file does not exists, create it and write UTF-8 bytes to it.
If the file exists, open it, seek to the end, and append UTF-8 bytes.
A text editor that does not understand UTF-8 is not worth supporting. If you feel inclined, include a UTF-8 BOM when you create the file. Use TEncoding.UTF8.GetBytes and TEncoding.UTF8.GetString to encode and decode.
Just use TStringList, until size of file < ~50-100Mb (it depends on CPU speed):
procedure ReadTextFromFile(const AFileName: string; SL: TStringList);
begin
SL.Clear;
SL.DefaultEncoding:=TEncoding.ANSI; // we know, that old files has this encoding
SL.LoadFromFile(AFileName, nil); // let TStringList detect real encoding.
// if not - it just use DefaultEncoding.
end;
procedure WriteTextToFile(const AFileName: string; const TextToWrite: string);
var
SL: TStringList;
begin
SL:=TStringList.Create;
try
ReadTextFromFile(AFileName, SL); // read all file with encoding detection
SL.Add(TextToWrite);
SL.SaveToFile(AFileName, TEncoding.UTF8); // write file with new encoding.
// DO NOT SET SL.WriteBOM to False!!!
finally
SL.Free;
end;
end;
The Inifiles unit should support unicode. At least according to this answer: How do I read a UTF8 encoded INI file?
Inifiles are quite commonly used to store strings, integers, booleans and even stringlists.
procedure TConfig.ReadValues();
var
appINI: TIniFile;
begin
appINI := TIniFile.Create(ChangeFileExt(Application.ExeName,'.ini'));
try
FMainScreen_Top := appINI.ReadInteger('Options', 'MainScreen_Top', -1);
FMainScreen_Left := appINI.ReadInteger('Options', 'MainScreen_Left', -1);
FUserName := appINI.ReadString('Login', 'UserName', '');
FDevMode := appINI.ReadBool('Globals', 'DevMode', False);
finally
appINI.Free;
end;
end;
procedure TConfig.WriteValues(OnlyWriteAnalyzer: Boolean);
var
appINI: TIniFile;
begin
appINI := TIniFile.Create(ChangeFileExt(Application.ExeName,'.ini'));
try
appINI.WriteInteger('Options', 'MainScreen_Top', FMainScreen_Top);
appINI.WriteInteger('Options', 'MainScreen_Left', FMainScreen_Left);
appINI.WriteString('Login', 'UserName', FUserName);
appINI.WriteBool('Globals', 'DevMode', FDevMode);
finally
appINI.Free;
end;
end;
Also see the embarcadero documentation on inifiles: http://docwiki.embarcadero.com/Libraries/Seattle/en/System.IniFiles.TIniFile
Code based on David's suggestions:
{--------------------------------------------------------------------------------------------------
READ/WRITE UNICODE
--------------------------------------------------------------------------------------------------}
procedure WriteToFile(CONST FileName: string; CONST aString: String; CONST WriteOp: WriteOperation= woOverwrite; WritePreamble: Boolean= FALSE); { Write Unicode strings to a UTF8 file. It can also write a preamble }
VAR
Stream: TFileStream;
Preamble: TBytes;
sUTF8: RawByteString;
aMode: Integer;
begin
ForceDirectories(ExtractFilePath(FileName));
if (WriteOp= woAppend) AND FileExists(FileName)
then aMode := fmOpenReadWrite
else aMode := fmCreate;
Stream := TFileStream.Create(filename, aMode, fmShareDenyWrite); { Allow read during our writes }
TRY
sUTF8 := Utf8Encode(aString); { UTF16 to UTF8 encoding conversion. It will convert UnicodeString to WideString }
if (aMode = fmCreate) AND WritePreamble then
begin
preamble := TEncoding.UTF8.GetPreamble;
Stream.WriteBuffer( PAnsiChar(preamble)^, Length(preamble));
end;
if aMode = fmOpenReadWrite
then Stream.Position:= Stream.Size; { Go to the end }
Stream.WriteBuffer( PAnsiChar(sUTF8)^, Length(sUTF8) );
FINALLY
FreeAndNil(Stream);
END;
end;
procedure WriteToFile (CONST FileName: string; CONST aString: AnsiString; CONST WriteOp: WriteOperation);
begin
WriteToFile(FileName, String(aString), WriteOp, FALSE);
end;
function ReadFile(CONST FileName: string): String; {Tries to autodetermine the file type (ANSI, UTF8, UTF16, etc). Works with UNC paths }
begin
Result:= System.IOUtils.TFile.ReadAllText(FileName);
end;
In Delphi 7, I have a widestring encoded with Base64(That I received from a Web service with WideString result) :
PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8c3RyaW5nPtiq2LPYqjwvc3RyaW5nPg==
when I decoded it, that result is not UTF-8:
<?xml version="1.0"?>
<string>طھط³طھ</string>
But when I decoded it by base64decode.org, result is true :
<?xml version="1.0"?>
<string>تست</string>
I have use EncdDecd unit for DecodeString function.
The problem you have is that you are using DecodeString. That function, in Delphi 7, treats the decoded binary data as being ANSI encoded. And the problem is that your text is UTF-8 encoded.
To continue with the EncdDecd unit you have a couple of options. You can switch to DecodeStream. For instance, this code will produce a UTF-8 encoded text file with your data:
{$APPTYPE CONSOLE}
uses
Classes,
EncdDecd;
const
Data = 'PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8c3RyaW5nPtiq2LPYqjwvc3RyaW5nPg==';
var
Input: TStringStream;
Output: TFileStream;
begin
Input := TStringStream.Create(Data);
try
Output := TFileStream.Create('C:\desktop\out.txt', fmCreate);
try
DecodeStream(Input, Output);
finally
Output.Free;
end;
finally
Input.Free;
end;
end.
Or you could continue with DecodeString, but then immediately decode the UTF-8 text to a WideString. Like this:
{$APPTYPE CONSOLE}
uses
Classes,
EncdDecd;
const
Data = 'PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8c3RyaW5nPtiq2LPYqjwvc3RyaW5nPg==';
var
Utf8: AnsiString;
wstr: WideString;
begin
Utf8 := DecodeString(Data);
wstr := UTF8Decode(Utf8);
end.
If the content of the file can be represented in your application's prevailing ANSI locale then you can convert that WideString to a plain AnsiString.
var
wstr: WideString;
str: string; // alias to AnsiString
....
wstr := ... // as before
str := wstr;
However, I really don't think that using ANSI encoded text is going to lead to a very fruitful programming life. I encourage you to embrace Unicode solutions.
Judging by the content of the decoded data, it is XML. Which is usually handed to an XML parser. Most XML parsers will accept UTF-8 encoded data, so you quite probably can base64 decode to a memory stream using DecodeStream and then hand that stream off to your XML parser. That way you don't need to decode the UTF-8 to text and can let the XML parser deal with that aspect.
As an addendum to David Heffernan's awesome answer, and Remy Lebeau's note on how it's broken on Delphi 7, I would like to add a function that will help any developer stuck on Delphi 7.
Since UTF8Decode() is broken in Delphi 7, I found a function in a forum that solved my problem:
function UTF8ToWideString(const S: AnsiString): WideString;
var
BufSize: Integer;
begin
Result := '';
if Length(S) = 0 then Exit;
BufSize := MultiByteToWideChar(CP_UTF8, 0, PAnsiChar(S), Length(S), nil, 0);
SetLength(result, BufSize);
MultiByteToWideChar(CP_UTF8, 0, PANsiChar(S), Length(S), PWideChar(Result), BufSize);
end;
So now, you can use DecodeString, and then decode the UTF-8 text to a WideString using this function:
begin
Utf8 := DecodeString(Data);
wstr := UTF8ToWideString(Utf8);
end.
okay, so I (VERY) recently started playing with lazaruz/free pascal, and I'm a little stuck with reading files with TMemoryStream and it's streaming kin.
I'm trying to write a simple base64 encoder, that can encode strings of text, or files (like images and WAVs) to then be used in html and javascript.
The following code compiles great but I get EReadError Illegal stream image when trying to load a file. I'll include the working string only procedure for reference:
procedure TForm1.TextStringChange(Sender: TObject);
begin
Memo1.Lines.Text := EncodeStringBase64(TextString.Text);
end;
procedure TForm1.FormCreate(Sender: TObject);
begin
Memo1.Lines.Text := '';
Form1.BorderIcons := [biSystemMenu,biMinimize];
end;
procedure TForm1.BitBtn1Click(Sender: TObject);
begin
if OpenDialog1.Execute then
begin
filename := OpenDialog1.Filename;
stream := TMemoryStream.Create;
try
StrStream := TStringStream.Create(s);
try
stream.LoadFromFile(filename);
stream.Seek(0, soFromBeginning);
ObjectBinaryToText(stream, StrStream);
StrStream.Seek(0, soFromBeginning);
Memo1.Lines.Text := EncodeStringBase64(StrStream.DataString);
finally
StrStream.Free;
end;
finally
stream.Free;
end;
end;
end;
Can anyone help me out?
You get the "illegal stream image" exception because the file you're loading probably isn't a binary DFM file. That's what ObjectBinaryToText is meant to process. It's not for arbitrary data. So get rid of that command.
You can skip the TMemoryStream, too. TStringStream already has a LoadFromFile method, so you can call it directly instead of involving another buffer.
StrStream.LoadFromFile(filename);
But a string isn't really the right data structure to store your file in prior to base64-encoding it. The input to base64 encoding is binary data; the output is text. Using a text data structure as an intermediate format means you may introduce errors into your data because of difficulties in encoding certain data as valid characters. The right interface for your encoding function is this:
function Base64Encode(Data: TStream): string;
You don't need to load the entire file into memory prior to encoding it. Just open the file with a TFileStream and pass it to your encoding function. Read a few bytes from it at a time with the stream's Read method, encode them as base64, and append them to the result string. (If you find that you need them, you can use an intermediate TStringBuilder for collecting the result, and you can add different buffering around the file reads. Don't worry about those right away, though; get your program working correctly first.)
Use it something like this:
procedure TForm1.BitBtn1Click(Sender: TObject);
var
filename: string;
stream: TStream;
begin
if OpenDialog1.Execute then begin
filename := OpenDialog1.Filename;
stream := TFileStream.Create(filename, fmOpenRead);
try
Memo1.Lines.Text := Base64Encode(stream);
finally
stream.Free;
end;
end;
end;
I never heard before about ObjectBinaryToText(), but looks like funky one. Also, what is EncodeStringBase64() function?
At first place, you shouldn't convert binary stream to text to encode it, instead you should directly B64 encode binary data. B64 algorithm is intended to work on array of bytes.
Since Delphi 6, there is EncdDecd.pas unit, which implements B64 encoding methods. I'm not sure if Lazarus/FPC have this, but if they do, your code to B64 encode file should look like this (add EncdDecd to uses list):
procedure TForm1.Button1Click(Sender: TObject);
var
instream : TFileStream;
outstream: TStringStream;
begin
if OpenDialog1.Execute then
begin
instream := TFileStream.Create(OpenDialog1.FileName, fmOpenRead or fmShareDenyNone);
try
outstream := TStringStream.Create;
try
EncodeStream(instream, outstream);
Memo1.Lines.Text := outstream.DataString;
finally
outstream.Free;
end;
finally
instream.Free;
end;
end;
end;
It's possible to convert the XML to UTF-8 encoding in Delphi 6?
Currently that's what I am doing:
Fill TXMLDocument with AnsiString
At the end convert the Data to UTF-8 by using WideStringVariable = AnsiToUtf8(Doc.XML.Text);
Save the value of WideStringVariable to file using TFileStream and Adding BOM for UTF8 at the file beggining.
CODE:
Procedure SaveAsUTF8( const Name:String; Data: TStrings );
const
cUTF8 = $BFBBEF;
var
W_TXT: WideString;
fs: TFileStream;
wBOM: Integer;
begin
if TRIM(Data.Text) <> '' then begin
W_TXT:= AnsiToUTF8(Data.Text);
fs:= Tfilestream.create( Name, fmCreate );
try
wBOM := cUTF8;
fs.WriteBUffer( wBOM, sizeof(wBOM)-1);
fs.WriteBuffer( W_TXT[1], Length(W_TXT)*Sizeof( W_TXT[1] ));
finally
fs.free
end;
end;
end;
If I open the file in Notepad++ or another editor that detects encoding, it shows me UTF-8 with BOM. However, it seems like the text it's not properly encoded.
What is wrong and how can I fix it?
UPDATE: XML Properties:
XMLDoc.Version := '1.0';
XMLDoc.Encoding := 'UTF-8';
XMLDoc.StandAlone := 'yes';
You can save the file using standard SaveToFile method over the TXMLDocument variable: http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/XMLDoc_TXMLDocument_SaveToFile.html
Whether the file would be or not UTF8 you have to check using local tools like aforementioned Notepad++ or Hex Editor or anything else.
If you insist of using intermediate string and file stream, you should use the proper variable. AnsiToUTF8 returns UTF8String type and that is what to be used.
Compiling `WideStringVar := AnsiStringSource' would issue compiler warning and
It is a proper warning. Googling for "Delphi WideString" - or reading Delphi manuals on topic - shows that WideString aka Microsoft OLE BSTR keeps data in UTF-16 format. http://delphi.about.com/od/beginners/l/aa071800a.htm
Thus assignment UTF16 string <= 8-bit source would necessarily convert data and thus dumping WideString data can not be dumping UTF-8 text by the definition of WideString
Procedure SaveAsUTF8( const Name:String; Data: TStrings );
const
cUTF8: array [1..3] of byte = ($EF,$BB,$BF)
var
W_TXT: UTF8String;
fs: TFileStream;
Trimmed: AnsiString;
begin
Trimmed := TRIM(Data.Text);
if Trimmed <> '' then begin
W_TXT:= AnsiToUTF8(Trimmed);
fs:= TFileStream.Create( Name, fmCreate );
try
fs.WriteBuffer( cUTF8[1], sizeof(cUTF8) );
fs.WriteBuffer( W_TXT[1], Length(W_TXT)*Sizeof( W_TXT[1] ));
finally
fs.free
end;
end;
end;
BTW, this code of yours would not create even empty file if the source data was empty. It looks rather suspicious, though it is you to decide whether that is an error or not wrt the rest of your program.
The proper "uploading" of received file or stream to web is yet another issue (to be put as a separate question on Q&A site like SO), related to testing conformance with HTTP. As a foreword, you can readsome hints at WWW server reports error after POST Request by Internet Direct components in Delphi
In order to have the correct encoding inside the document, you should set it by using the Encoding property in your XML Document, like this:
myXMLDocument.Encoding := 'UTF-8';
I hope this helps.
You simply need to call the SaveToFile method of the document:
XMLDoc.SaveToFile(FileName);
Since you specified the encoding already, the component will use that encoding.
This won't include a BOM, but that's generally what you want for an XML file. The content of the file will specify the encoding.
As regards your SaveAsUTF8 method, it is not needed, but it is easy to fix. And that may be instructive to you.
The problem is that you are converting to UTF-16 when you assign to a WideString variable. You should instead put the UTF-8 text into an AnsiString variable. Changing the type of the variable that you named W_TXT to AnsiString is enough.
The function might look like this:
Procedure SaveAsUTF8(const Name: string; Data: TStrings);
const
UTF8BOM: array [0..2] of AnsiChar = #$EF#$BB#$BF;
var
utf8: AnsiString;
fs: TFileStream;
begin
utf8 := AnsiToUTF8(Data.Text);
fs:= Tfilestream.create(Name, fmCreate);
try
fs.WriteBuffer(UTF8BOM, SizeOf(UTF8BOM));
fs.WriteBuffer(Pointer(utf8)^, Length(utf8));
finally
fs.free;
end;
end;
Another solution:
procedure SaveAsUTF8(const Name: string; Data: TStrings);
var
fs: TFileStream;
vStreamWriter: TStreamWriter;
begin
fs := TFileStream.Create(Name, fmCreate);
try
vStreamWriter := TStreamWriter.Create(fs, TEncoding.UTF8);
try
vStreamWriter.Write(Data.Text);
finally
vStreamWriter.Free;
end;
finally
fs.free;
end;
end;
I'm trying to save some lines of text in a codepage different from my system's such as Cyrillic to a TFileStream using Delphi XE. However I can't find any code sample to produce those encoded file ?
I tried using the same code as TStrings.SaveToStream however I'm not sure I implemented it correctly (the WriteBom part for example) and would like to know how it would be done elsewhere. Here is my code:
FEncoding := TEncoding.GetEncoding(1251);
FFilePool := TObjectDictionary<string,TFileStream>.Create([doOwnsValues]);
//...
procedure WriteToFile(const aFile, aText: string);
var
Preamble, Buffer: TBytes;
begin
// Create the file if it doesn't exist
if not FFilePool.ContainsKey(aFile) then
begin
// Create the file
FFilePool.Add(aFile, TFileStream.Create(aFile, fmCreate));
// Write the BOM
Preamble := FEncoding.GetPreamble;
if Length(Preamble) > 0 then
FFilePool[aFile].WriteBuffer(Preamble[0], Length(Preamble));
end;
// Write to the file
Buffer := FEncoding.GetBytes(aText);
FFilePool[aFile].WriteBuffer(Buffer[0], Length(Buffer));
end;
Thanks in advance.
Not sure what example are you looking for; may be the following can help - the example converts unicode strings (SL) to ANSI Cyrillic:
procedure SaveCyrillic(SL: TStrings; Stream: TStream);
var
CyrillicEncoding: TEncoding;
begin
CyrillicEncoding := TEncoding.GetEncoding(1251);
try
SL.SaveToStream(Stream, CyrillicEncoding);
finally
CyrillicEncoding.Free;
end;
end;
If I understand it's pretty simple. Declare an AnsiString with affinity for Cyrillic 1251:
type
// The code page for ANSI-Cyrillic is 1251
CyrillicString = type AnsiString(1251);
Then assign your Unicode string to one of these:
var
UnicodeText: string;
CyrillicText: CyrillicString;
....
CyrillicText := UnicodeText;
You can then write CyrillicText to a stream in the traditional manner:
if Length(CyrillicText)>0 then
Stream.WriteBuffer(CyrillicText[1], Length(CyrillicText));
There should be no BOM for an ANSI encoded text file.