How to know charset of character? - character-encoding

Question

A python script fail when trying to encode a supposed utf-8 string in iso-8859-1:
>>> 'à'.encode('iso-8859-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0300' in position 1: ordinal not in range(256)
How to know wich charset is that character ? When encode it in utf-8:
>>> 'à'.encode('utf-8')
b'a\xcc\x80'
a then \xcc\x80. I can fount \xcc\x80 in http://www.utf8-chartable.de/unicode-utf8-table.pl?start=768&names=-&utf8=string-literal utf8 table.
But it is utf-8 ? If it is utf-8 why 'à'.encode('utf-8') can't encode this string in iso-8859-1 ?

Answer 1

It's a bit unclear where the 'à' character comes from. In fact, it's a combining character sequence and you need to normalize it. Next python script uses unicodedata module to self-explain and solve your questions:
import sys, platform
print (sys.stdout.encoding, platform.python_version())
print ()
import unicodedata
agraveChar='à' # copied from your post
agraveDeco='à' # typed as Alt+0224 (Windows, us keyboard)
# print Unicode names
print ('agraveChar', agraveChar, agraveChar.encode('utf-8'))
for ins in range( 0, len(agraveChar)):
print ( agraveChar[ins], unicodedata.name(agraveChar[ins], '???'))
print ('agraveDeco', agraveDeco, agraveDeco.encode('utf-8'))
for ins in range( 0, len(agraveDeco)):
print ( agraveDeco[ins], unicodedata.name(agraveDeco[ins], '???'))
print ('decomposition(agraveChar)', unicodedata.decomposition(agraveChar))
print ('\nagraveDeco normalized:\n')
print ("NFC to utf-8", unicodedata.normalize("NFC" , agraveDeco).encode('utf-8'))
print ("NFC to latin", unicodedata.normalize("NFC" , agraveDeco).encode('iso-8859-1'))
print ("NFKC to utf-8", unicodedata.normalize("NFKC", agraveDeco).encode('utf-8'))
print ("NFKC to latin", unicodedata.normalize("NFKC", agraveDeco).encode('iso-8859-1'))
Output:
==> D:\test\Python\40422359.py
UTF-8 3.5.1
agraveChar à b'\xc3\xa0'
à LATIN SMALL LETTER A WITH GRAVE
agraveDeco à b'a\xcc\x80'
a LATIN SMALL LETTER A
̀ COMBINING GRAVE ACCENT
decomposition(agraveChar) 0061 0300
agraveDeco normalized:
NFC to utf-8 b'\xc3\xa0'
NFC to latin b'\xe0'
NFKC to utf-8 b'\xc3\xa0'
NFKC to latin b'\xe0'
==>