Ruby calling a Python script causes encoding error with German characters - ruby-on-rails

Question

A Ruby on rails application launches a Python script to get German word lemmas. The Python scripts exits with the following error:
File "/PATHTOSCRIPT/script.py", line 15, in <module>
for l in sys.stdin:
File "/PATHTOPYTHON/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
Ruby on Rails:
require 'open3'
#in, #out, stderr = Open3.popen3("/PATHTOSCRIPT/script.py") if ['de'].include? lang
a = "übervölkerung"
#in.write "#{a}\n"
logger.info(#treetagger_out.read.nil?)
logger.info(stderr.read)
Python:
import sys
import os
sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', buffering=1)
for l in sys.stdin:
l = l.strip()
I've detected that in Ruby and Python there are different character counts:
Ruby:
2.2.3 :006 > a="übervölkerung"
=> "übervölkerung"
2.2.3 :007 > print a.bytes
[195, 188, 98, 101, 114, 118, 195, 182, 108, 107, 101, 114, 117, 110, 103] => nil
Python:
>>> a="übervölkerung"
>>> print(list(map(ord, a)))
[252, 98, 101, 114, 118, 246, 108, 107, 101, 114, 117, 110, 103]

Answer 1

The input to your Python script is apparently text encoded with UTF-8.
If you encode your test string "übervölkerung" with UTF-8, then the first byte is C3, which is found in the traceback in the beginning of your post.
This means you need to read STDIN with a text stream that decodes UTF-8, not ASCII.
You already have a line that creates a wrapper around sys.stdin:
sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', buffering=1)
This replaces the default text stream reader (an io.TextIOWrapper instance) with a new one.
But you don't specify the input encoding, so the default encoding is used – which is determined by the environment (based on OS-specific environment variables).
In your case the encoding apparently defaults to ASCII, which is not what you need.
You need UTF-8, so write:
sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', encoding='UTF-8')
(Of course you can leave the buffering=1 parameter there if you think you need it.)
Also, os.fdopen is just a more restricted version of the built-in open function. So you can just use that one, without losing anything:
sys.stdin = open(sys.stdin.fileno(), 'r', encoding='UTF-8')
By the way, the difference in character count you see between Ruby and Python comes from the fact that you are looking at different things.
In the Ruby code, you look at the bytes of the UTF-8 encoded text, while in Python you look at the (Unicode) code points.
In the second case, each number corresponds to a single character, while multiple numbers correspond to a character in the first case.
To see the byte values in Python, do:
>>> a = "übervölkerung"
>>> list(a.encode('utf8'))
[195, 188, 98, 101, 114, 118, 195, 182, 108, 107, 101, 114, 117, 110, 103]
I don't know how to see code points in Ruby, though.