how to load a properties file with non-ascii in ant - ant

Question

Suppose I have a properties file test.properties, which saved using utf-8
testOne=测试
I am using the following ant script to load it and echo it to another file:
<loadproperties srcFile="test.properties" encoding="utf-8"/>
<echo encoding="utf-8" file="text.txt">${testOne}</echo>
When I open the generated text.txt file using "utf-8" encoding I see:
??
What's wrong with my script?

Answer 1

Use "encoding" and "escapeunicode" together. It's work fine.
<loadproperties srcfile="${your.properties.file}" encoding="UTF-8">
<filterchain>
<escapeunicode />
</filterchain>
</loadproperties>

Answer 2

I found a work around, but I still doesn't understand why the org one doesn't work:
<native2ascii src="." dest=".">
<mapper type="glob" from="test.properties" to="testASCII.properties"/>
</native2ascii>
<loadproperties srcFile="testASCII.properties"/>
Then the echo works as expected.
I don't know why the encoding in loadproperties doesn't work.
Can anyone explain?

Answer 3

Try it this way:
<loadproperties srcfile="non_ascii_property.properties">
<filterchain>
<escapeunicode/>
</filterchain>
</loadproperties>
Apparently, InputStreamReader that uses the ISO Latin-1 charset, which kills your non-ascii characters. I ran into the same issue w/Arabic.

Answer 4

What editor were you using and what platform are you on?
Your generated property file might actually be good, but the editor you're using to examine it may be incapable of viewing it. For example, on my Mac, the VIM command line editor can view it (which surprises me), but in Eclipse, it looks like this:
testOne=������
If you're on Unix/Linux/Mac, try using od to dump your generated file, and examine the actual hex code to see what it should be.
For example, I copied your property file, and ran od on a Mac:
$ od -t x1 -t c test.property
0000000 74 65 73 74 4f 6e 65 3d e6 b5 8b e8 af 95 0a
t e s t O n e = 测 ** ** 试 ** ** \n
Here I can see that the code for 测 is 36 b5 8b and 试 is e8 af 95 which is the correct UTF-8 representation for these two characters. (Or, I at least think so. It shows up correctly in the Character Viewer Mac OS X panel).

Answer 5

The right answer is pointed to in this comment by David W.:
how to load a properties file with non-ascii in ant
Java Property Files must be encoded in ISO-8859-1:
http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html
But Unicode escape sequences like \u6d4b can/must be used to encode unicode characters therein.
Tools/ANT-Targets like <native2ascii>, generating ascii-encoded files from natively maintained ones, can help here.

Answer 6

You can write your own task that reads properties from your default java character encoding for your OS (mine is utf-8), instead of converting your property files to unreadable unicode-escaped ASCII files (designed by people who read and write only English). Here's how to do it by copying and modifying Property.java from Ant's source code to your own package (e.g. org.my.ant). I used Ant 1.10.1.
Download Ant's source code in your format of choice from here:
http://ant.apache.org/srcdownload.cgi
Copy src/main/org/apache/tools/ant/taskdefs/Property.java to your own project (such as a new Java project), in org/my/ant/Property.java)
replace:
package org.apache.tools.ant.taskdefs;
with:
package org.my.ant;
Fix any imports needed by the package change. I just needed to add:
import org.apache.tools.ant.taskdefs.Execute;
In the method:
loadProperties(Properties props, InputStream is, boolean isXml)
replace:
props.load(is);
with:
props.load(new InputStreamReader(is));
In your project's resources folder (could be the same as your source folder), add the file org/my/ant/antlib.xml, with the content:
<?xml version="1.0" encoding="UTF-8"?>
<antlib>
<taskdef name="property" classname="org.my.ant.Property"/>
</antlib>
Compile this project (Property.java + antlib.xml).
Put the resulting jar in Ant's classpath, as explained here:
http://ant.apache.org/manual/using.html#external-tasks
Then use it in a build.xml file as follows:
<?xml version="1.0" encoding="UTF-8"?>
<project name="example"
xmlns:my="antlib:org.my.ant"
default="print"
>
<my:property file="greek.properties" prefix="example" />
<target name="print">
<echo message="${example.a}"/>
</target>
</project>
The file greek.properties contains:
a: ΑΒΓ