I am working on an application in Delphi 2009 which makes heavy use of RTF, edited using TRichEdit and TLMDRichEdit. Users who entered Japanese text in these RTF controls have been submitting intermittent reports about the Japanese text being displayed as gibberish when reloading the content, both on Win XP and Vista, with Eastern Language Support installed.
Typically, English and Japanese is mixed and is mostly displayed without a problem, for example:
Inventory turns partnerships. 在庫回転率の
(my apologies if the Japanese text is broken incorrectly - I do not speak or read the language).
Quite frequently however, only the Japanese portion of the text will be gibberish, for example:
ŒÉñ?“]-¦Œüã‚Ì·•Ê‰?-vˆö‚ðŽû‰v‚ÉŒø‰?“I‚ÉŒ‹‚т‚¯‚é’mŽ¯‚ª‘÷Ý‚·‚é?(マーケットセクター、
見込み客の優 先順位と彼らに販売する知識)
From extensive online searching, it appears that the problem is as a result of the fonts saved as part of the RTF. Fonts present on Japanese language version of Windows is not necessarily the same as a US English version. It is possible to programmatically replace the fonts in the RTF file which yields an almost acceptable result, i.e.
-D‚‚ｽƒIƒyƒŒ[ƒVƒ・“‚ﾆƒƒWƒXƒeƒBƒbƒN‚ﾌƒpƒtƒH[ƒ}ƒ“ƒX‚-˜‰v‚ﾉŒ‹‚ﾑ‚ﾂ‚ｯ‚ﾈ‚｢‚±ﾆ‚ﾍ?A‘‚｢‚ﾉ-ｳ‘ﾊ‚ﾅ‚ ‚驕B‚ｻ‚‚ﾍAl“ｾ‚ｵ‚ｽ・‘P‚ﾌˆﾛ‚ƒƒXƒN‚ﾉ‚ｳ‚‚ｷB
However, there are still quite a few "junk" characters in there which are not correctly recognized as Japanese characters. Looking at the raw RTF you'll see the following:
-D\'82\'82\u65405?\'83I\'83y\'83\'8c[\'83V\'83\u12539?\ldblquote\'82\u65414?
Clearly, the Unicode characters are rendered correctly, but for example the \'82\'82 pair of characters should be something else? My guess is that it actually represents a double byte character of some sort, which was for some mysterious reason encoded as two separate characters rather than a single Unicode character.
Is there a generic, (relatively) foolproof way to take RTF containing Eastern Languages and reliably displaying it again?
For completeness sake, I updated the RTF font table in the following way:
Replaced the font name "?l?r ?o?S?V?b?N;" with "\'82\'6c\'82\'72 \'82\'6f\'83\'53\'83\'56\'83\'62\'83\'4e;"
Updated font names by replacing "\froman\fprq1\fcharset0 " with "\fnil\fprq1\fcharset128 "
Updated font names by replacing "\froman\fprq1\fcharset238 " with "\fnil\fprq1\fcharset128 "
Updated font names by replacing "\froman\fprq1 " with "\fnil\fprq1\fcharset128 "
Replacing font name "?? ?????;" with "\'82\'6c\'82\'72 \'82\'6f\'83\'53\'83\'56\'83\'62\'83\'4e;"
Update: Updating font names alone wont make a difference. The locale seems to be the big problem. I have seen a few site discussing ways around converting the display of Japanese RTF to something most reader would handle, but I haven't found a solution yet, see for example:
here and here.
My guess is that changing font names in the RTF has probably made things worse. If a font specified in the RTF is not a Unicode font, then surely the characters due to be rendered in that font will be encoded as Shift-JIS, not as Unicode. And then so will the other characters in the text. So treating the whole thing as Unicode, or appending Unicode text, will cause the corruption you see. You need to establish whether RTF you import is encoded Shift-JIS or Unicode, and also whether the machine you are running on (and therefore D2009 default input format) is Japanese or not. In Japan, if a text file has no Unicode BOM it would usually be Shift-JIS (but not always).
I was seeing something similar, but not with Japanese fonts. Just special characters like micro (as in microliters) and superscripts. The problem was that even though the RTF string I was sending to the user from an ASP.NET webpage was correct (I could see the encoded RTF stream using Fiddler2), when MS Word actually opened the RTF, it added a bunch of garbage escape codes like what I see in your sample.
What I did was to run the entire RTF text through a conversion routine that swapped all characters over ascii 127 to their special unicode point equivalent. So I would get something like \uc1\u181? (micro) for the special characters. When I did that, Word was able to open the file no problem. Ironically, it re-encoded the \uc1\uxxx? back to their RTF escaped equivalents.
Private Function ConvertRtfToUnicode(ByVal value As String) As String
Dim ch As Char() = value.ToCharArray()
Dim c As Char
Dim sb As New System.Text.StringBuilder()
Dim code As Integer
For i As Integer = 0 To ch.Length - 1
c = ch(i)
code = Microsoft.VisualBasic.AscW(c)
If code <= 127 Then
'Don't need to replace if one of your typical ASCII codes
sb.Append(c)
Else
'MR: Basic idea came from here http://www.eggheadcafe.com/conversation.aspx?messageid=33935981&threadid=33935972
' swaps the character for it's Unicode decimal code point equivalent
sb.Append(String.Format("\uc1\u{0:d}?", code))
End If
Next
Return sb.ToString()
End Function
Not sure if that will help your problem, but it's working for me.
Related
I need to display LST ISO/IEC 8859-13 codepage characters on window. Currently I'm using ShowMessage function for this purpose. Evrything displayed fine when windows locale is from this region, but how to deal when I have for example locale English UK? In this case I have just "?" instead of character. It should be some kind of possibility to show regional characters since MS Word displays them without correct locale. But how to do that?
You have two viable, tractable options:
Upgrade to a Unicode version of Delphi that has built in support for international text, or
Use the TNT Unicode controls that graft that support onto pre-Unicode Delphi by using the COM WideString type which is encoded using Unicode.
Word has no problems doing this because it uses the native Unicode API of Windows. On the other hand Delphi 7 uses the ANSI API that exists solely to provide compatibility with Windows 95/98/ME.
Short version:
you must also set the Font.Charset property if you want to be (more) sure that a particular component will display characters in a given charset.
Long version (sorry: i am prone to be wordy)
Using unicode (and you should switch to an unicode version of delphi, if you haven't done it yet) does not guarantee that the fonts installed on a foreign pc will contain the all the symbols you want to display.
Using unicode, moreover, does nothing to force the operating system to choose a font that actually supports the charset you need: even if there is an installed font able to display cyrillic characters, windows will NOT choose that font just because you are asking him to render a string containing cyrillic unicode code points: it will still be using the default system fonts.
So: there always is the possibility that you will need to ask your customers to install a font supporting the charset your application needs. if this can be a serious issue, you should consider the idea of distributing the required fonts along with your binaries (be careful with font copirights).
In second place: if there are components in your application you are SURE that they will always show russian text, well, in such components you MUST assign Font.Charset = RUSSIAN_CHARSET. This is the way of telling windows "I really need to display cyrillic chars in this component, so choose an appropriate font, regardeless of which side of the planet you are running"
It is a common misconception that che charset property is useless with unicode programs. it is quite the opposite.
Another common error is to assume that the "XYZ" font is identical on all windows installations in the world so, if I can see cyrillic chars with Thamoa on my pc, then I am safe using Thamoa for displaying cyrillic in the rest of the world. it is quite the opposite: a different unicode subset gets installed depending on the computer locale.
and... Since AFAIK ShowMessage() uses the system default font, you can't use this procedure for displaying messages containing "strange" characters: you need to write your own ShowMessage dialog box.
EDIT: here is an example demonstrating what I am saying
just drop a TPaintBox component on a form, name it "pbox", and write this OnPaint event handler:
(remember to save the source in utf-8 format, otherwise the russian symbols will be mangled)
procedure TForm1.pboxPaint(Sender: TObject);
begin
pbox.canvas.Font.Name := 'Fixedsys';
pbox.Canvas.TextOut(0,0,'Это русский');
pbox.canvas.Font.Name := 'Fixedsys';
pbox.canvas.Font.Charset := RUSSIAN_CHARSET;
pbox.Canvas.TextOut(0,20,'Это русский');
end;
On an italian pc (and I guess on any west-european or american pc) the fixedsys font does not normally contain the russian characters symbols: the first TextOut will insist in using the FixedSys font and will write garbage. On my pc i get a sequence of black square boxes, for example.
The second textout is made after having set charset=RUSSIAN_CHARSET, so windows will know that we need the russian symbols and so chooses another font. The second TextOut is not using the FixedSys font I wanted to use, but at least it is readable!
On a russian installation of windows, both TextOut calls will correctly render the russian text using the FixedSys font, since russian installations of windows have a russian version of the fixedsys font. and Windows knows it.
You can install more than one locale on a Windows system. If you are using the matching locale then it is the default locale and you can use a dialog with a text field which uses the correct locale / character set. On your development system, where English UK is installed, add the missing language(s).
Unicode is nicer, but not required to display characters from non-default character sets (computers were able to handle many character sets before Uincode was invented). Even MS Wordpad was able to display characters from different codepages, including multi-byte character sets (Korean, Japanese, Chinese) long before Unicode existed.
ShowMessage can not be used because it sticks to the default locale. But can easily be replaced with a custom dialog-style form.
I am using Clozure Cl on Mac os x 10.9 and Portable allegro serve
I have a file with text has characters like ı ç ş ö (these are some characters Turkish also have) and some Arabic characters. I cannot serve them. when i visit from the browser this kind of characters are not displayed at all, only part of text showed is the ones until the first ı in the text.
In Lisp i use a function composed with a do and read-lines and format (or i have tried print princ prin1 also) reads entire document and when i set the :external-format :utf-8 it shows the read characters properly in Lisp. Problem is in serving them, if i can serve them as i read on Lisp it will be done.
Also If do not set :external-formatat all, in Lisp it is read improperly, as expected, however, this time the browser can show all the text but with wrong characters in place of above described characters.
How to fix that and use external-formats character encodings properly?
See http://www.xach.com/lisp/allegro-cl/2001-3/964.html for an example on how to use :external-format in AllegroServe.
Cheers
Frank
P.S. I also posted an answer to the same question newsgroup comp.lang.lisp .
In our Visual Basic 6.0 program we used function chr(11) appended with some string and is displayed in text box.
In Windows 2003 Server, the value in text box is displayed as "a box(for chr(11)) followed by string"
In windows 7, the value in text box is displayed as "♂(for chr(11) followed by string"
Can anyone advice why it is behaving like this?
Thanks in advance.
It is probably a difference in fonts.
Even when the same "face name" is used the actual installed font can differ in terms of things like which glyphs are suported.
Note that your program isn't using ASCII in any sense of the word, but ANSI. The mapping from Unicode in your program to ANSI for display varies with Locale and Charset settings as well. Charset might also be a factor here.
Chr(11) says "take 11 and treat it as an ANSI character in the current codepage, convert that to Unicode, then return it as a Variant String."
Chr$(11) removes some of that overhead by returning a String, and ChrW$(11) is even cleaner, skipping the laundering through ANSI-to-Unicode conversion as well.
Faster yet is to just used the named constant for this character vbVerticalTab instead.
But none of that impacts display. It's more a question of avoiding unnecessary overhead.
You're relying on something that isn't reliable, i.e. that non-printable characters will always have a glyph. That "box" symbol you see means there is no glyph available for the character.
Even the Character Map applet doesn't display the glyph mapping for values below 33 (&H21).
HI I need to include a french text in vb script with special characters - REJETÉE.
When I add text, it appears as REJETÉE. How do I include the following?
I can use either : É or É. but it shows exceptions.
Visual basic in itself does only support ASCII characters. The 'É' is an ANSI character, thus unsupported.If you are writing to a text file, you may want to print the direct ANSI value of the character. If you now use a text editor that uses ANSI encoding, you will see the 'É' correctly.
Also, maybe the following will work as a debug output, you will have to try thou:
Debug.Print chr$(200)
You should get the 'È' character when writing the value 200. As I said, i don't know if this works in the debug printer, but when writing to text files it will work.
I have some byte streams that may or may not be encoded as 1) extended ASCII, 2) UTF-8, or 3) UTF-16. And they may be in English, French, or Chinese. I would like to write a simple program that allows the user to enter a byte stream and then pick one of the encodings and one of the languages and see what the string would look like when interpreted in that manner. Or simply interpret each string in each of the 9 possible ways and display them all. I would like to avoid having to switch regionalizations repeatedly. I'm using Delphi 2007. Is what I am trying to do even possible?
In Delphi 2009 or later, this would be easier, since it supports Unicode and can do most of this transparently. For older versions, you have to do a bit more manual work.
The first thing you want to do is convert the text to a common codepage; preferably UTF-16, since that's the native codepage on Windows. For that, you use the MultiByteToWideChar function. For UTF-8 to UTF-16, the language doesn't matter; for "extended ASCII", you will need to choose an appropriate source code page (e.g. Windows-1252 for English and French, and GB2312 or Big5 or some other Chinese code page - that depends on what you expect to receive). To store these, you can use a WideString, which stores UTF-16 directly.
Once you have that, you have to draw the text somehow - and that requires you to either get a Unicode-capable control (a label is likely sufficient), or write one, or just call the appropriate Windows API function directly to draw - and that's where it can get a bit messy, because there are several functions for doing that. TextOutW is probably the simplest choice here, but another option would be DrawText. Make sure you explicitly call the W version of these function in order to work with Unicode. (See also the related question How do I draw Unicode text?).
Take note: Due to CJK unification - the encoding of equivalent Chinese Hanzi, Japanese Kanji, and Korean Hanja characters at the same code points in Unicode - you need to pick a font that matches the expected kind of Chinese, traditional or simplified, in order to get expected rendering. To quote a somewhat related post by Michael Kaplan:
What it comes down to is that there are many characters which can have
four different possible looks:
Japanese will default to using MS UI Gothic (fallback to PMingLIU, then SimSun, then Gulim)
Korean will default to using Gulim (fallback to PMingLiu, then MS UI Gothic, then SimSun)
Simplified Chinese will default to using SimSun (fallback to PMingLiu, then MS UI Gothic, then Batang)
Traditional Chinese will default to using PMingLiu (fallback to SimSun, then MS Mincho, then Batang)
Unless you have a specific font you want/need to use, pick the first font in the list for the language variant you want to use, since these are standard fonts (on XP, you will need to enable East Asian Language support before they are available, on Vista and above, they are always included). If you do not do this, then Windows may either not render the characters at all (showing the missing character glyph instead), or it may use an inappropriate fallback (e.g. PMingLiu for Simplified Chinese) - the exact behavior depends on the API function you use to render the text.