Stanford NLP dependency parser gives me Chinese characters as question marks - parsing

Question

I'm trying to parse chinese data in the CoNLL format using the Stanford dependency parser edu/stanford/nlp/models/parser/nndep/CTB_CoNLL_params.txt.gz but I seem to have some encoding difficulties.
My input file is in utf-8, already segmented into the different words, a sentence looks like this: 那时 的 坎纳里 鲁夫 ， 有着 西海岸 最大 的 工业化 罐头 工厂 。
The commands I use to run the model are these:
java -mx2200m -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLP \
-language Chinese \
-encoding utf-8 \
-props StanfordCoreNLP-chinese.properties \
-annotators tokenize,ssplit,pos,depparse \
-file ./ChineseCorpus/ChineseTestSegmented.txt \
-outputFormat conll \
It all seems to work fine except for not encoding the chinese character right, this is the output that I get:
1 ?? _ NT _ 2 DEP
2 ? _ DEG _ 4 NMOD
3 ??? _ NR _ 4 NMOD
4 ?? _ NR _ 6 SUB
5 ? _ PU _ 6 P
6 ?? _ VE _ 0 ROOT
7 ??? _ NN _ 12 NMOD
8 ?? _ JJ _ 9 DEP
9 ? _ DEG _ 12 NMOD
10 ??? _ NN _ 12 NMOD
11 ?? _ NN _ 12 NMOD
12 ?? _ NN _ 6 OBJ
13 ? _ PU _ 6 P
According to the Stanford parser faq the standard encoding for Chinese is GB18030 but they also say "However, the parser is able to parse text in any encoding, providing you pass the correct encoding option on the command line", which I did.
I have looked at this question: How to use Stanford LexParser for Chinese text? but their solution using iconv doesn't work for me, I get the error cannot convert and I have been trying several possible combinations of encodings.
Anybody suggestions on what is going wrong?

Answer 1

Try something like:
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLP \
-language Chinese -props StanfordCoreNLP-chinese.properties \
-annotators segment,ssplit,pos,parse -file chinese-in.txt -outputFormat conll
E.g.:
alvas#ubi:~/stanford-corenlp-full-2015-12-09$ cat chinese-in.txt
那时的坎纳里鲁夫，有着西海岸最大的工业化罐头工厂。
alvas#ubi:~/jose-stanford/stanford-corenlp-full-2015-12-09$ \
> java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLP \
> -language Chinese -props StanfordCoreNLP-chinese.properties \
> -annotators segment,ssplit,pos,parse -file chinese-in.txt -outputFormat conll
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Registering annotator segment with class edu.stanford.nlp.pipeline.ChineseSegmenterAnnotator
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator segment
Loading classifier from edu/stanford/nlp/models/segmenter/chinese/ctb.gz ... [main] INFO edu.stanford.nlp.wordseg.ChineseDictionary - Loading Chinese dictionaries from 1 file:
[main] INFO edu.stanford.nlp.wordseg.ChineseDictionary - edu/stanford/nlp/models/segmenter/chinese/dict-chris6.ser.gz
[main] INFO edu.stanford.nlp.wordseg.ChineseDictionary - Done. Unique words in ChineseDictionary is: 423200.
done [14.4 sec].
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/chinese-distsim/chinese-distsim.tagger ... done [1.4 sec].
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator parse
[main] INFO edu.stanford.nlp.parser.common.ParserGrammar - Loading parser from serialized file edu/stanford/nlp/models/lexparser/chineseFactored.ser.gz ...
done [5.2 sec].
Processing file /home/alvas/jose-stanford/stanford-corenlp-full-2015-12-09/chinese-in.txt ... writing to /home/alvas/jose-stanford/stanford-corenlp-full-2015-12-09/chinese-in.txt.conll
Annotating file /home/alvas/jose-stanford/stanford-corenlp-full-2015-12-09/chinese-in.txt
[main] INFO edu.stanford.nlp.wordseg.TagAffixDetector - INFO: TagAffixDetector: useChPos=false | useCTBChar2=true | usePKChar2=false
[main] INFO edu.stanford.nlp.wordseg.TagAffixDetector - INFO: TagAffixDetector: building TagAffixDetector from edu/stanford/nlp/models/segmenter/chinese/dict/character_list and edu/stanford/nlp/models/segmenter/chinese/dict/in.ctb
[main] INFO edu.stanford.nlp.wordseg.CorpusChar - Loading character dictionary file from edu/stanford/nlp/models/segmenter/chinese/dict/character_list
[main] INFO edu.stanford.nlp.wordseg.affDict - Loading affix dictionary from edu/stanford/nlp/models/segmenter/chinese/dict/in.ctb
done.
Annotation pipeline timing information:
ChineseSegmenterAnnotator: 0.2 sec.
WordsToSentencesAnnotator: 0.0 sec.
POSTaggerAnnotator: 0.0 sec.
ParserAnnotator: 0.9 sec.
TOTAL: 1.2 sec. for 13 tokens at 11.0 tokens/sec.
Pipeline setup: 21.1 sec.
Total time for StanfordCoreNLP pipeline: 22.3 sec.
[out]:
http://pastebin.com/raw/Y9J0UBDF