Delphi XE3, Win7 Prof.
I need to write into DBASE 3 (old format) files to export data for a DOS-like application (Clipper?).
Ok, I thought: MS DBASE driver can do this.
But I have problem with hungarian accents.
I tried this connection string:
Driver={Microsoft dBASE Driver (*.dbf)};DriverID=21;Dbq=c:\temp;Extended Properties=dBASE III;charSet=CP 852;Locale Identifier=1038;Character Set=CP 852;CODEPAGE=852
As I saw it cannot write only ANSI files (the DOS app accepts CP852 chars).
I tried to convert the content with AnsiToOEM, but some characters lost on save. In the record I see good content, but the saved file contains wrong accents.
The test text is "árvíztűrő tükörfúrógép".
The "í", "ó", "Ó" is missing from the result.
And I found some strange thing!
If the main form have an opened ADOConnection (the connected property is true in the DFM) then I will read good characters from the DBASE files, and I can write them into the file - the ANSI characters will be converted correctly. "í" is ok, "ó" is ok.
This ADOConnection object could be different than the reader.
If I close this ADOConnection in IDE mode, the opened files won't be converted, so I will see some strange accented chars, and I won't write good text into the file.
It is strange, because if I open this connection on FormCreate by code, the problem will appear...
I can read and write the ADOQuery records if the resource streamer read the ADOConnection's active (True value) "connected" property from the DFM!
I don't know what happened in the background, and how to force this ADO character transformation routine to work, but I wasted more days to find a working DBASE III exporter, and I have found only a buglike thing...
Does anyone know what is this? Why the ADO character encoder/decoder works only if I had a connected ADOConnection in DFM?
Or how I can use ADODB.Connection instead of ADOConnection object to avoid this side effect?
Thanks for every idea!
As I see I need to set the code page to fix the string for ADO.
var
s: string;
aStr1, aStr2: AnsiString;
begin
...
s := 'árvíztûrõ tükörfúrógép';
aStr1 := s;
SetLength(aStr2, Length(aStr1));
AnsiToOemBuff(PAnsiChar(aStr1), PAnsiChar(aStr2), Length(aStr1));
SetCodePage(RawbyteString(aStr2), 852, False); // THIS IS THE SOLUTION
ADOQuery1.FieldBYName('name').AsAnsiString := aStr2;
Otherwise something is converting my AnsiString again in the background.
Related
I have a polling application developed in Delphi 6.
It reads a file, parse the file according to specification, performs validation and uploads into database (SQL Server 2008 Express Edition)
We had to provide support for Operating Systems having Double Byte Character Sets (DBCS) e.g. Japanese OS.
So, we changed the database fields in SQL Server from varchar to nvarchar.
Polling works fine in Operating Systems with DBCS. It also works successfully for non-DBCS Operating systems, if the
System Locale is set to Japanese/Chinese/Korean and Operating system has the respective language pack.
But, if the Locale is set to english then, the database contains junk characters for the double byte characters.
I performed a few tests but failed to identify the solution.
e.g. If I read from a UTF-8 file using a TStringList and save it to another file then, the Unicode data is saved.
But, if I use the contents of the file to run an update query using TADOQuery component then, the junk characters are shown.
The database also contains the junk characters.
PFB the sample code:
var
stlTemp : TStringList;
qry : TADOQuery;
stQuery : string;
begin
stlTemp := TStringList.Create;
qry := TADOQuery.Create(nil);
stlTemp.LoadFromFile('D:\DelphiUnicode\unicode.txt');
//stlTemp.SaveToFile('D:\DelphiUnicode\1.txt'); // This works. Even though
//the stlTemp.Strings[0] contains junk characters if seen in watch
stQuery := 'UPDATE dbo.receivers SET company = ' + QuotedStr(stlTemp.Strings[0]) +
' WHERE receiver_cd = N' + QuotedStr('Receiver');
//company is a nvarchar field in the database
qry.Connection := ADOConnection1;
with qry do
begin
Close;
SQL.Clear;
SQL.Add(stQuery);
ExecSQL;
end;
qry.Free;
stlTemp.Free
end;
The above code works fine in a DBCS Operating system.
I have tried playing with string,widestring and UTF8String. But, this does not work in English OS if the locale is set to English.
Please provide any pointers for this issue.
In non Unicode Delphi version, The basics are that you need to work with WideStrings (Unicode) instead of Strings (Ansi).
Forget about TADOQuery.SQL (TStrings), and work with TADODataSet.CommandText or TADOCommand.CommandText(WideString) or typecast TADOQuery as TADODataSet. e.g:
stlTemp: TWideStringList; // <- Unicode strings - TNT or other Unicode lib
qry: TADOQuery;
stQuery: WideString; // <- Unicode string
TADODataSet(qry).CommandText := stQuery;
RowsAffected := qry.ExecSQL;
You can also use TADOConnection.Execute(stQuery) to execute queries directly.
Be extra careful with Parametrized queries: ADODB.TParameters.ParseSQL is Ansi. If ParamCheck is true (by default) TADOCommand.SetCommandText->AssignCommandText will cause
problems if your Query is Unicode (InitParameters is Ansi).
(note that you can use ADO Command.Parameters directly - using ? chars as placeholder for the parameter instead of Delphi's convention :param_name).
QuotedStr returns Ansi string. You need a Wide version of this function (TNT)
Also, As #Arioch 'The mentioned TNT Unicode Controls suite is your best fried for making Delphi Unicode application.
It has all the controls and classes you need to successfully manage Unicode tasks in your application.
In short, you need to think Wide :)
You did not specified database server, so this investigation remains on our part. You should check how does your database server support Unicode. That means how to specify Unicode charset for the database and the tables/column/indices/collations/etc inside it. You have to ensure that the whole DB is pervasively Unicode-enabled in every its detail, to avoid data loss.
Generally you also should check that your database connection (using database access library of choice) also is unicode-enabled. Generally Microsoft ADO, just like and OLE, should be Unicode-enabled. But still check your database server manual how to specify unicode codepage or charset in the connection string. non-Unicode connection may also result in data loss.
When you tell you read some unicode file - it is ambiguous. What ius unicode file ? Is it UTF-8 ? Or one of four flavours of UTF-16 ? Or UTF-7 ? Or some other Unicode Transportation Format ? Usual windows WideChar roughly corresponds to legacy UCS-2 and is expected be BOM-stripped Intel-Endian flavour of UTF-16. http://msdn.microsoft.com/en-us/library/windows/desktop/ms221069.aspx
If the file is surely that flavour of UTF-16, then you can load it using Delphi TWideStringList or Jedi CodeLibrary TJclWideStringList. Review you code that you never work with your data using string variables - use WideString everywhere to avoid data loss.
Since D6 was one of buggiest releases, i'd prefer to ensure EVERY update to Delphi is installed and then install and use JCL. JCL also provides codepage transition functions, that might be more flexible than plain AnsiStringVar := WideStringVar approach.
For UTF-8 file, it can be loaded by TWideStringList class of JCL (but not TJclWideStringList).
When debugging, load lines of the list to WideString variable and see that their content is preserved.
Don't write queries like that. See http://bobby-tables.com/ Even if you do not expect malicious cracker - you can yourself make errors or meat unexpected data. Use parametrized queries, everywhere, every time! EVER!
See the example of such: http://docs.embarcadero.com/products/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/ADODB_TADOQuery_Parameters.html
Check that every SQL VARCHAR parameter would be ftWideString to contain Unicode, not ftString. Check the same about fields(columns).
Think if legacy technologies can be casted aside since their support would only get harder in time.
7.1. Since Microsoft ADO is deprecated (for exampel newer versions of Microsoft SQL Server would not support it), consider switching to 'live' data access libraries. Like AnyDAC, UniDAC, ZeosDB or some other library. Torry.net may hint you some.
7.2. Since Delphi 6 RTL and VCL is not Unicode-ready, consider migrating your application to TNT Unicode Components, if you'd manage to find their free version or purchase them. Or migrating to newer Delphi releases.
7.3. Since Delphi 6 is very old and long not-supported and since it was one of buggiest Delphi releases, consider migrating to newer Delphi versions or free tools like CodeTyphoon or Lazarus. As a bonus, Lazarus started moving to Unicode in its recent beta builds, and it is possible that by the end of migration to it you would get you application unicode-ready.
7.4 Migration might be excuse and stimulus for re-factoring your application and getting rid of legacy spaghetti.
I want to display the entire content of a TStringList while debugging the application.
Instead I just get pointers. The Flist is showing only the address.
If you are using Delphi 2010 or later, the debugger allows for this using debug visualizers.
For older versions, you can dump the contents of the Text property in the Watch window or using OutputDebugString, but that's difficult to read. You could set up watches for each element of the list, but that's only practical for very short lists.
I would probably use an external logging app like CodeSite or SmartInspect that let you dump the contents of a TStringList in a single call.
Inspect the Text property. It's the concatenated version of the stringlist.
Since i´m using BDS MMVI, i´m using an "ultra clever smart" method for that kind issue, i use it for large xml documents. I start context file editor (very capable free text editor writen in delphi by the way). On the debugger window a just do a FList.SaveToFile('contents.txt'), since context can monitor file modifications i can see whats happening in my xml files.
Sorry for the "clever" joke but it does work for me.
Peace
I use the visualizers now that I have D2010. I used to use a function I called CArray that would return an array of strings. If I added CArray(MyStringList) to the watch window, I would be able to examine the contents of the string list. I used to be employed to write VB6 code and I sort of liked the various 'C' functions for converting to a useful type. CArray for stringlists and CArray for ClientDataset fields were mostly useful for debugging.
function CArray(List: TStrings): TStrArray; Overload;
var i,
iCount: Integer;
begin
iCount := List.Count;
SetLength(Result, iCount);
for i := 0 to Pred(iCount) do Result[i] := List[i];
end;
My two cents:
You can evaluate expression list_instance_variable.SaveToFile('temp_file_name.txt') and then examine content of the file in any editor.
To do that you must use this function anywhere in code and turn off optimization (at least in Delphi 7), otherwise object code of SaveToFile would be removed by the linker.
I'm using HttpGetText with Synapse for Delphi 7 Professional to get the source of a web page - but feel free to recommend any component and code.
The goal is to save some time by 'unifying' non-ASCII characters to a single charset, so I can process it with the same Delphi code.
So I'm looking for something similar to "Select All and Convert To UTF without BOM in Notepad++", if you know what I mean. ANSI instead of UTF8 would also be okay.
Webpages are encoded in 3 charsets: UTF8, "ISO-8859-1=Win 1252=ANSI" and straight up the alley HTML4 without charset spec, ie. htmlencoded Å type characters in the content.
If I need to code a PHP page that does the conversion, that's fine too. Whatever is the least code / time.
When you retreive a webpage, its Content-Type header (or sometimes a <meta> tag inside the HTML itself) tells you which charset is being used for the data. You would decode the data to Unicode using that charset, then you can encode the Unicode to whatever you need for your processing.
I instead did the reverse conversion directly after retrieving the HTML using GpTextStream. Making the documents conform to ISO-8859-1 made them processable using straight up Delphi, which saved quite a bit of code changes. On output all the data was converted to UTF-8 :)
Here's some code. Perhaps not the prettiest solution but it certainly got the job done in less time. Note that this is for the reverse conversion.
procedure UTF8FileTo88591(fileName: string);
const bufsize=1024*1024;
var
fs1,fs2: TFileStream;
ts1,ts2: TGpTextStream;
buf:PChar;
siz:integer;
procedure LG2(ss:string);
begin
//dont log for now.
end;
begin
fs1 := TFileStream.Create(fileName,fmOpenRead);
fs2 := TFileStream.Create(fileName+'_ISO88591.txt',fmCreate);
//compatible enough for my purposes with default 'Windows/Notepad' CP 1252 ANSI and Swe ANSI codepage, Latin1 etc.
//also works for ASCII sources with htmlencoded accent chars, naturally
try
LG2('Files opened OK.');
GetMem(buf,bufsize);
ts1 := TGpTextStream.Create(fs1,tsaccRead,[],CP_UTF8);
ts2 := TGpTextStream.Create(fs2,tsaccWrite,[],ISO_8859_1);
try
siz:=ts1.Read(buf^,bufsize);
LG2(inttostr(siz)+' bytes read.');
if siz>0 then ts2.Write(buf^,siz);
finally
LG2('Bytes read and written OK.');
FreeAndNil(ts1);FreeAndNil(ts2);end;
finally FreeAndNil(fs1);FreeAndNil(fs2);FreeMem(buf);
LG2('Everything freed OK.');
end;
end; // UTF8FileTo88591
I written a program with Delphi 7 which searches *.srt files on a hard drive. This program lists the path and name of these files in a memo. Now I need convert these files from ANSI to UTF-8, but I haven't succeeded.
The Utf8Encode function takes a WideString string as parameter and returns a Utf-8 string.
Sample:
procedure ConvertANSIFileToUTF8File(AInputFileName, AOutputFileName: TFileName);
var
Strings: TStrings;
begin
Strings := TStringList.Create;
try
Strings.LoadFromFile(AInputFileName);
Strings.Text := UTF8Encode(Strings.Text);
Strings.SaveToFile(AOutputFileName);
finally
Strings.Free;
end;
end;
Take a look at GpTextStream which looks like it works with Delphi 7. It has the ability to read/write unicode files in older versions of Delphi (although does work with Delphi 2009) and should help with your conversion.
var
Latin1Encoding: TEncoding;
begin
Latin1Encoding := TEncoding.GetEncoding(28591);
try
MyTStringList.SaveToFile('some file.txt', Latin1Encoding);
finally
Latin1Encoding.Free;
end;
end;
Please read the whole answer before you start coding.
The proper answer to question - and it is not the easy one - basically consist of tree steps:
You have to determine the ANSI code page used on your computer. You can achieve this goal by using the GetACP() function from Windows API. (Important: you have to retrieve the codepage as soon as possible after the file name retrieval, because it can be changed by the user.)
You must convert your ANSI string to Unicode by calling MultiByteToWideChar() Windows API function with the correct CodePage parameter (retrieved in the previous step). After this step you have an UTF-16 string (practically a WideString) containing the file name list.
You have to convert the Unicode string to UTF-8 using UTF8Encode() or the WideCharToMultiByte() Windows API. This function will return an UTF-8 string you needed.
However this solution will return an UTF-8 string containing the input ANSI string, this probably is not the best way to solve your problems, since the file names may already be corrupted when the ANSI functions returned them, so proper file names are not guaranteed.
The proper solution to your problem is ways more complicated:
If you want to be sure that your file name list is exactly clean, you have to make sure it won't get converted to ANSI at all. You can do this by explicitly using the "W" version of the file handling API's. In this case - of course - you can not use TFileStream and other ANSI file handling objects, but the Windows API calls directly.
It is not that hard, but if you already have a complex framework built on e.g. TFileStream it could be a bit of a pain in the #ss. In this case the best solution is to create a TStream descendant that uses the appropriate API's.
I hope my answer helps you or anyone who has to deal with the same problem. (I had to not so long ago.)
I did only this:
procedure TForm1.FormCreate(Sender: TObject);
begin
Strings := TStringList.Create;
end;
procedure TForm1.Button3Click(Sender: TObject);
begin
Strings.Text := UTF8Encode(Memo1.Text);
Strings.SaveToFile('new.txt');
end;
Verified with Notepad++ UTF8 without BOM
Did you mean ASCII?
ASCII is backwards compatible with UTF-8.
http://en.wikipedia.org/wiki/UTF-8
For some reason, lately the *.UDL files on many of my client systems are no longer compatible as they were once saved as ANSI files, which is no longer compatible with the expected UNICODE file format. The end result is an error dialog which states "the file is not a valid compound file".
What is the easiest way to programatically open these files and save as a unicode file? I know I can do this by opening each one in notepad and then saving as the same file but with the "unicode" selected in the encoding section of the save as dialog, but I need to do this in the program to cut down on support calls.
This problem is very easy to duplicate, just create a *.txt file in a directory, rename it to *.UDL, then edit it using the microsoft editor. Then open it in notepad and save as the file as an ANSI encoded file. Try to open the udl from the udl editor and it will tell you its corrupt. then save it (using notepad) as a Unicode encoded file and it will open again properly.
Ok, using delphi 2009, I was able to come up with the following code which appears to work, but is it the proper way of doing this conversion?
var
sl : TStrings;
FileName : string;
begin
FileName := fServerDir+'configuration\hdconfig4.udl';
sl := TStringList.Create;
try
sl.LoadFromFile(FileName, TEncoding.Default);
sl.SaveToFile(FileName, TEncoding.Unicode);
finally
sl.Free;
end;
end;
This is very simple to do with my TGpTextFile unit. I'll put together a short sample and post it here.
It should also be very simple with the new Delphi 2009 - are you maybe using it?
EDIT: This his how you can do it using my stuff in pre-2009 Delphis.
var
strAnsi : TGpTextFile;
strUnicode: TGpTextFile;
begin
strAnsi := TGpTextFile.Create('c:\0\test.udl');
try
strAnsi.Reset; // you can also specify non-default 8-bit codepage here
strUnicode := TGpTextFile.Create('c:\0\test-out.udl');
try
strUnicode.Rewrite([cfUnicode]);
while not strAnsi.Eof do
strUnicode.Writeln(strAnsi.Readln);
finally FreeAndNil(strUnicode); end;
finally FreeAndNil(strAnsi); end;
end;
License: The code fragment above belongs to public domain. Use it anyway you like.