I need the Pascal (Delphi or Free Pascal) function to convert UTF-8 text (e.g. in string) to Shift-JIS (also e.g. in string).
For Shift-JIS => UTF-8 I found the code, which is here. Now I need the backward converter.
I want to add this code to Lazarus library LConvEncoding, so it must be without using 'iconv' or Win32 API, it must be pure Pascal. It may be code in another language if it's easy to rewrite it in Pascal.
I found the answer, this code can be made from the Unicode official data:
ftp://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/SHIFTJIS.TXT .
That file tells how to get UTF-16 codes, and next I must convert UTF-16 -> UTF-8, using UTF8Encode().
Related
I'm pretty sure this worked in an earlier version of C++Builder but now I need to figure out the easiest way to get this to work in C++Builder 10.1.
fprintf(out, "%s\n", Edit1->Text);
I have no idea why, but right now it seems to be writing only the first character of the text in Edit1 to the file.
Also if anyone has a link to a good source for best practices for dealing with strings C++Builder 10.1, I would be grateful.
After some more reading it seems the solution is pretty easy.
fwprintf(out, L"%s\n", Edit1->Text);
What I find most odd is, as near as I can tell, the Text property doesn't have a type. How are you supposed to know work with it when it has no type?
I'm pretty sure this worked in an earlier version of C++Builder
Maybe, but only in pre-CB2009 versions, where the Text was an AnsiString (since CB2009, it is now a UnicodeString), and only as a fluke of undefined behavior, since you can't pass an (Ansi|Unicode)String object to fprintf() to begin with, it expects a pointer to a C-style character string instead. The sole data member of an AnsiString is a char*, and the sole data member of a UnicodeString is a wchar_t*/char16_t* on Windows/Posix (respectively).
now I need to figure out the easiest way to get this to work in C++Builder 10.1.
You have a few choices:
cast the Text to an AnsiString and then use its c_str() method to get a char* pointer:
fprintf(out, "%s\n", AnsiString(Edit1->Text).c_str());
use fprintf() with the %Ls placeholder instead:
fprintf(out, "%ls\n", Edit1->Text.c_str());
use fwprintf() instead:
fwprintf(out, L"%s\n", Edit1->Text.c_str());
use a more modern file I/O library, like the standard C++ std::ofstream, or the RTL's TStreaWriter, etc.
I have no idea why, but right now it seems to be writing only the first character of the text in Edit1 to the file.
Because you are telling fprintf() (via %s) to expect an 8bit null-terminated char* string, but you are passing it a 16bit wchar_t* string instead. ASCII characters have 0x00 bytes in them in 16bit strings.
Also if anyone has a link to a good source for best practices for dealing with strings C++Builder 10.1, I would be grateful.
Have you read Embarcadero's Migration and Upgrade documentation yet? It has a section on migrating ANSI code to Unicode.
What the Windows 'hosts' file encoding is? Is it UTF-8? Or ASCII + system codepage? How IDN (international domain names with umlauts etc.) entries should be added and can they be added at all?
It should be ANSI or UTF-8 without BOM. I just dealt with a server that had the hosts file encoding set to UCS-2 Little Endian, and that led to the file being ignored.
There is a wealth of information here:
https://serverfault.com/questions/452268/hosts-file-ignored-how-to-troubleshoot
The simple answer is
ANSI or UTF-8 WITH BOM.
(UTF-8 without BOM is NOT valid).
Details:
As far as I have tried, the encoding of the hosts file on Windows should be
ANSI or UTF-8 with BOM.
I know this question is many years old, but a colleague made the mistake of looking at this post and the ServerFault post, so I decided to add an answer.
1. Simple case only ASCII
Works.
Without any multi-byte characters, This is equivalent to ANSI, also equivalent to UTF-8 without BOM.
2. ANSI (with Japanese ANSI multi-byte characters)
Works.
note: There are Japanese characters but this is valid ANSI encoding in windows.
In Japanese editions of Windows, this code page cp932 is referred to as "ANSI",
https://en.wikipedia.org/wiki/Code_page_932_(Microsoft_Windows)
3. UTF-8 with BOM
Works.
note: BOM 付き means with BOM.
4. UTF-8 without BOM
DOES NOT work.
5. Additional test cases
If you use emoji instead of Japanese, the result will be the same.
Use emoji and save as UTF8 without BOM does not work.
(However, other lines not include emoji may be worked correctly.)
Use emoji and save as UTF8 with BOM can resolve host correctly.
note: If you use Notepad to check it yourself, be sure to put double quotes in the file name when you save it, or Notepad will be create hosts.txt.
Appended:
(Asked in comment)
The hosts file supports inline comments.
Currently I am trying to set application name using
net.rim.blackberry.api.homescreen.HomeScreen.setName("これはある");
but it throws exception: IllegalArgumentException.
Can anyone provide the solution?
I am using Blackberry JDE 5.0.
This is probably a string encoding problem. Try
new String(new String("これはある").getBytes("UTF-16BE"), "UTF-16BE");
It's not pretty but I think that will work.
Here's a link to the Blackberry string spec: http://www.blackberry.com/developers/docs/5.0.0api/java/lang/String.html
By default it's ISO-8859-1 which does not include Japanese characters.
The problem you are facing is how to get a string represented in your source code into your application with the same characters. For latin characters, this is pretty straightforward, as we can just put the characters in quotes, and get a string, like "Hello world"
When you go to non-latin, like Japanese, it gets harder. You can still directly write Japanese in your source code, but you need to make sure your editor and your compiler agree on an encoding so that the characters can be interpreted correctly. The Java-SE compiler takes an argument "-encoding" which allows you to specify the encoding of your java source files.
Unfortunately, rapc, the BlackBerry compiler, does not offer an option to specify encoding, even though it is invoking javac itself. So rapc uses the platform default, which is utf-8 on Linux and OSX and iso-8859-1 on Windows.
The way around this problem is to use a feature of the Java language for parsing strings - unicode escaping. By entering the six character sequence "\u3053" in a string, the java compiler will parse that number as hexidecimal and use the corresponding unicode code point, solving problems with source file encoding.
So "Hello world" and "\u0048\u0065\u006c\u006c\u006f\u0020\u0057\u006f\u0072\u006c\u0064" will result in the same strings appearing in your class files.
Because of this, Svetlin's answer from the comments is the right approach here:
net.rim.blackberry.api.homescreen.HomeScreen.setName("\u3053\u308C\u306F\u3042\u308B");
My program is written in Delphi 7 and I want to avoid a Russian or a Chinese,
Korean try to use my soft because file paths contains Unicode chars and my program can t handle them yet (as long as I do not port my program on a new Delphi version supporting UNICODE).
How do I write a function detecting the "Unicode language" in Delphi 7?
A Delphi 7 program (in its VCL part) can handle Russian, Chinese or Korean characters without any problem.
If the Windows system language is properly set, the charset will match the corresponding encoding, and the file names will be able to have Unicode chars as available in this charset. In fact, default string=AnsiString is converted into Unicode when the VCL calls Windows APIs (all ....A() calls will do the conversion then call the ....W() version).
You can force the default code page (the one which will select the charset to be used) by calling code like this:
if GetThreadLocale<>LCID then // force locale settings if different
if SetThreadLocale(LCID) then
GetFormatSettings; // resets all locale-specific variables
In this case, the TFileName (=AnsiString) in the current system charset will be converted by Windows into the corresponding Unicode characters, and you'll be able to use it in your Delphi 7 application.
What you can't do with the standard VCL AnsiString use it to directly mix charsets, as you can since Delphi 2009, thanks to the new string = UnicodeString default paradigm.
PS:
Since the CharSet only involve #128..#255 chars (i.e. all with bit 7 set), if you use only #0..#127 chars, your string will be consistent whatever the current charset/codepage setting is. If you use only English chars and numbers e.g., your path will always work, whatever the charset/codepage is. But if you use non English chars, the path will only work if the charset/codepage is correctly set, which is the case for a path used by an end-user (using a TOpenDialog at runtime for instance).
We have been reading and writing Sticky Notes/Annotations/Comments to pdfs via an activex control in our application for a number of years. We have recently upgraded to Delphi2009 with Unicode Support. The following is causing problems.
When we call
CAcroPDAnnot.GetContents
The results seem to be rather strange and we lose our Unicode Chars. It is not like saving as an ansi string which would usually result in returning ????? instead we get a string such as
‚És‚“ú‚É•—Ž×‚ð‚Ђ¢‚½‚ç
For a string of Japanese characters.
However if I save the comments in the pdf to a datafile via the menu in the pdf itself it is written to file as something like
0kˆL0Oeå0k˜¨ª0’0r0D0_0‰
The latter can be export and reimported into an acrobat pdf and will recreate the correct unicode characters. However once I call CAcroPDAnnot.GetContents in my code it is coming back as something else.
Is CAcroPDAnnot.GetContents broken?
Is there an encoding scheme I should be aware of?
Is there an alternative I might be able to do?
Thanks
‚És‚“ú‚É•—Ž×‚ð‚Ђ¢‚½‚ç
That's the string:
に行く日に風邪をひいたら
in CP-932 aka Shift-JIS encoding, an awful but lamentably still-popular encoding in Japan.
You're currently interpreting it in as CP-1252 (Windows Western European). If your PDF-reading component won't convert it for you automatically, you'll need to find a way to detect what encoding the document is in and convert it manually.
I don't know what Delphi provides for reading encodings, but have you got the encodings for Shift-JIS installed in Windows, from the Control Panel -> Regional Options -> "Install files for East Asian languages" option? If not, that might explain why it'd be failing to convert automatically, perhaps.
You're not exactly giving us a lot of information to work with.
I take it you're talking about the "Acrobat.CAcroPDAnnot" class' method GetContents here. Which version of Acrobat are you using? Have you perhaps switched versions (or run an update) around the time you started programming with Delphi 2009?
Then: how did you instantiate the object? If using a *_TLB.pas file generated from the DLL, are you certain it still matches it? (Try re-generating it, if uncertain).
Third: how are you calling the method? What type of variable are you assigning the result to?
What might also help, is if you could provide a sample of an annotation (preferably including non-ASCII chars); and for that annotation:
what it should look like (and what it does look like inside Reader)
what it returns when using a pre-2009 version of Delphi*
what it returns when using Delphi 2009*
(* preferably the HEX byte codes of the (ansi/wide)strings; but output from the Ctrl-F7 inspector should do)
Then maybe someone could provide a more meaningful answer.
Ok, one of the main differences between Delphi 2009 and the earlier versions is that the default string type is an unicode string. That means that if you use the same ActiveX component as in previous versions, you are passing unicode strings to ascii strings and that is usually not a good idea.
There are a couple of solutions for this problem:
Try if you can upgrade your activeX component so that it supports full unicode strings.
Use AnsiString and not string to communicate with the activeX component. In this case, you can still use the old interface, but you are still bound to the same limitations.
Use an other control that creates pdf. There is a lot to find, but be prepared to change a big chunk of your software. (Some controls are XML based and use encoding. )