We have built an email service that reads emails (from IMAP) and put them into a table in a database. And it works for German / English text, but now I want to get all other emails, like Cyrillic or Chinese too. But the characters that I receive are something like this: поново теŃтирам or even just ?????.
We are using ADO objects to store data in database and column type for those fields are declared as nvarchar, also parameters of a stored procedure that is used to upload that data are nvarchar too.
I presume that is the problem with Delphi 2007, but still I read on some posts here and there that Delphi 2007 supports Unicode (if Unicode I need). So, does anybody knows how to read email with Cyrillic and store it in mssql table?
To get text from IdMessage I am using IdMessage.Body.Text
Thanks,
Dejan
And INDY version is 10.1.5
I have managed to make it working by using UTF8Decode function.
I have tested it with UTF-8 and ANSI charsets and it works in both cases (of course ANSI will produce ? in the places of Unicode characters).
declare BodyText and subject as WideStrings
after that just use:
BodyText := UTF8Decode(IdMessage.Body.GetText)
Related
I want my asp .net application to store Chinese charachters in WE8MSWIN1252 oracle 12c database, which oracle client should I use without changing db Charachterset.
I'm using windows server, I'v tried to set nls_lang in registry to Chinese charachterset but my app still store charachter like ??? in database.
Short answer:
You cannot store Chinese characters if your database character set is WE8MSWIN1252. Setting NLS_LANG has nothing to do with the ability to store characters in a database.
Long answer:
Usually the National Character Set (check with SELECT * FROM V$NLS_PARAMETERS WHERE PARAMETER = 'NLS_NCHAR_CHARACTERSET';) of a database support full Unicode, nowadays the default should be AL16UTF16. Use NVARCHAR2, NCHAR, NCLOB data type for columns where you like to store Chinese characters.
A workaround could be to convert your string to BASE64 and store these in your database, see Base64 encoding and decoding in oracle
For another ugly and very dangerous workaround see this one: difference between NLS_NCHAR_CHARACTERSET abd NLS_CHARACTERSET for Oracle
I have a private key file encrypted with a password that also contains non-US-ASCII characters (e.g. passwörd or s€cret).
I didn't find a way to use this key file from an indy based server, as Indy seems to use MBCS to convert the unicodestring password to an octet string.
According to https://www.rfc-editor.org/rfc/rfc8018 (end of section 3) UTF-8 is a common encoding rule for password octet string.
According to my investigations Indy (I'm using the version that comes with Delphi 10.2 Tokyo) uses IndyTextEncoding_OSDefault inside PasswordCallback (IdSSLOpenSSL.pas) to convert the (Unicode)String to a PAnsiChar.
IndyTextEncoding_OSDefault() (in IdGlobal.pas) sets GIdOSDefaultEncoding to TIdMBCSEncoding and also returns it. GIdOSDefaultEncoding is not globally available and I also didn't find a method to set it.
Is there a possibility to either change the encoding PasswordCallback uses or already pass the password as byte array/PAnsiChar/RawByteSting to Indy?
There is no option to change the charset that Indy uses for encoding Unicode passwords. You would have to alter Indy's source code and recompile it. In a future version, I'll consider changing it to UTF-8, or make it user-configurable.
Note that IndyTextEncoding_OSDefault is MBCS only on Windows. It is UTF-8 on other platforms.
Otherwise, you would have to call OpenSSL's SSL_CTX_set_default_passwd_cb() function directly to replace the password callback with your own function, then you can do whatever you want with it.
With IdFTP, the server i'm connecting to is not using UTF-8, but ANSI. There's nothing special about my code, i simply set Host, Username, Password and Connect to server. Then i call List method with no parameters. Iterating through DirectoryListing gives me incorrect results for file names. My sample directory name encoded in local code page (CP-1250) is:
aąęsśćńółżźz
I thought i'll be able to "fix" file name field by converting it to AnsiString and setting code page but it seems to be already broken - memory dump of DirectoryListing[I].FileName:
a ? ? s ? ? ? ?? ?? z
6100 FDFF FDFF 7300 FDFF FDFF FDFF 8FDB DFDF 7A00
Manipulating with GIdDefaultAnsiEncoding or IOHandler.DefStringEncoding (after Connect, before List) makes no difference. I don't want to mess in IdFTP or IdGlobal code because i'm using it with other projects that involve Unicode and these works perfectly. Delphi XE2 or XE7.
As you can see FData contains raw file name in a 2 bytes per char string:
Even if i set IOHandler.DefStringEncoding to any TIdTextEncoding that is FIsSingleByte = True, FMaxCharSize = 1. However it looks promising because #$009F is "ź" in CP-1250, but i'm not looking for a per server, temporary solution. I expected Indy to handle this correctly after setting IOHandler.DefStringEncoding and GIdDefaultAnsiEncoding based on server capabilities (UTF-8 or ANSI with specified encoding).
Total Commander connection log:
Your server supports the MLSD command. Total Commander is sending the MLSD command and not the older LIST command. This is good, because MLSD has a standardized format (see RFC 3659), which includes support for embedded charset information. If no charset is explicitly stated, UTF-8 must be used.
You did not show the command/response log for TIdFTP, but the fact that the TIdFTPListItem.Data property is showing MLSD formatted output data means TIdFTP.List() is also using the MLSD command (by calling TIdFTP.ExtListDir() internally). The output shown does not include an explicit charset attribute, so TIdFTP will decode the filename as UTF-8.
However, the raw filename data that is shown in the TIdFTPListItem.Data property is NOT the correct UTF-8 encoded form of the directory name you have shown (even when stored as a raw 8-bit encoded UnicodeString - which is what TIdFTP.ExtListDir() does internally before parsing it). So the problem is either:
your FTP server is not converting the directory name from CP-1250 to UTF-8 correctly in the first place. Considering that Total Commander appears to be able to handle the listing correctly, this is not likely.
TIdFTP is not storing the raw UTF-8 octet data correctly before parsing it. This is more likely.
Hard to say which is actually the case since you did not show the raw listing data that is actually being transmitted. And you did not specify which exact version of Delphi and Indy you are using, either. Assuming the server is transmitting UTF-8 correctly, you might simply be using an older Indy version that does not handle the UTF-8 transmission correctly. AFAIK, the current version available (10.6.2.5270 at the time of this writing) should be able to handle it, as long as you are using Delphi 2009 or later. If you can provide a Wireshark capture of the raw listing data, I can check if there are any logic issues in TIdFTP that need to be fixed or not.
My team was looking for quick solution that i had to provide. My solution is based on this post: http://forums2.atozed.com/viewtopic.php?p=32301#p32301 and this question: Converting UnicodeString to AnsiString
Once FTP listing is finished i do overwrite FileName property via function that extracts file name from Data, and then convert String to RawByteString with correct code page. Fix is applied only if server doesn't support UTF-8. This way i'm able to move around FTP - ChangeDir, Get, Put etc. without problems.
I need your help. Please help me.
I have Delphi 2010
I try to idHTTP.Get a file with Unicode text (Russian) from site into MemoryStream on English version of Windows 7.
Then I load this MemoryStream with Unicode text into, for example, Memo.
If I set the Russian language as "Language for non-Unicode programs" in Control Panel the text appear properly in Memo. But if I set Enlish - I get wrong characters (*$^#~!#).
How can I load Russian text in Unicode with idHTTP.Get from site and show it properly in any Windows (Chinese, English etc.)???
Thank you for help!!!
I suggest updating to a recent Delphi version that is Unicode enabled.
Update: It looks like the memorystream actually contains AnsiText in a specific code page instead of real Unicode text. You can declare an appropriate AnsiString variable with that codepage, load the text into that variable and then load the variable into the memo.
It is hard to tell more without seeing the real data.
TMemo expects Unicode (UTF-16 encoded) text. If you download the text using the version of TIdHTTP.Get() that fills a TStream, then you are downloading the raw (usually Ansi encoded) text, and then are responsible for manually decoding that to Unicode before then assigning that to the TMemo.
Assuming the webserver is specifying a correct charset for the text in the response headers, then use the version of TIdHTTP.Get() that returns a String instead. TIdHTTP will detect the charset and decode the raw data into Unicode for you, eg:
Memo1.Text := IdHTTP1.Get('http://addr_here');
Update to D2009 or higher
Use components that support unicode (I remember TMS offers some components)
Set the character set / code page correctly for the language you are using
I am assigning a string to a custom type I have declared, which I Read/Write using the TTreeViews Node.Data property. I read and write to and from the node, something like this:
Read: RichEdit1.Lines.Text := TMyData(TreeView1.Selected.Data).MyString;
Write: TMyData(TreeView1.Selected.Data).MyString := RichEdit1.Lines.Text;
This works perfect for plain strings, I want to allow Rich Formatted text to be stored in the string, without losing the formatting. I managed to do this by using Streams on the RichEdit, because I am saving my database using the Freeware Zeos Lib (SQL) I get Unknown Token errors (likely from the RTF tags). How can I save without the errors?
UPDATE
I have managed to get it saving correctly without erroring now, using Base64 Encoding/Decoding as suggested by Sylverdrag. This encodes my strings removing the bad characters.
Check out http://delphi.about.com/od/adptips2003/a/bltip1203_5.htm
(My original answer was for C# - misread your question)