What is the usual method of encoding documents in foreign alphabets? - character-encoding

Question

What is the usual method of encoding documents in foreign alphabets for the purposes of programs that do terminal to terminal communications? There are two parts to this question: Latin alphabets and non-Latin.
I know that 8859-1 can handle most European languages, so is the usual practice in say Danish to just set your computer to 8859 and you are done? What about French and Polish?
For non-Latin alphabets like Russian, Armenian and Korean obviously you cannot use 8859-1. Do they just write documents in some other code page and have their computer set to that code page or do they use unicode or UTF-8 or do all three? What is the standard practice?
I am only interested in alphabetic systems. I know how the non-alphabetic systems (Chinese/Japanese) work, so no need to explain what they do.
My need here is to understand what kind of support to build into a terminal-based communication system that will be used by people talking to each other in different countries. For example, imagine you are writing an instant messaging system and need it to be interoperable between people in different countries.

Answer 1

For any system set up in this decade, you should expect and demand Unicode (though not necessarily UTF-8) and be done.
Historically, you would see all three of (1) use a legacy codepage or even (gasp) official character set for your locale (much depending on your OS and vendor -- Windows and Mac would traditionally gravitate to their own proprietary code tables, while Linux would use ISO-8859-x where available and applicable); (2) use something "close enough" and just wing it in corner cases (for example, ISO-8859-1 is in principle insufficient for Finnish, but people would just refrain from using the handful of words where it matters, or write them unaccented); and (3) use a local convention such as pr"efix"ed "accents or LaTeX \"acc\"ents or uetterly uenreaedaeble digraphs (these have some base in tradition e.g. in Germany where it is still okay to write "umlaeute" as a variant of "umläute").
It is not really correct to say that ISO-8859-1 "can handle most European languages". It is sufficient for most of the official national languages of Western Europe (especially if you are willing to compromise a bit, like the French grudgingly did) but completely inadequate for the majority of European languages. There is ISO-8859-2 and ISO-8859-3 etc which cater to the needs of groups of other European languages, but in many settings, interoperability with ISO-8859-1 was also desirable, so these were always a bit problematic.
For the specific character sets you ask about, there is ArmSCII for Armenian, a variety of Cyrillic encodings for Russian -- depending on where you look and who you ask, Windows codepage 1251 or KOI-8R would be regarded as the dominant one --, and similarly a variety of Korean standards, though KSC 5601 seems to dominate at least for email (the link has forward pointers to several others).
While Korean is nominally analyzable as a roughly alphabetic writing system, the traditional encoding approach has been to create glyphs for each possible combined syllable, resulting in a large character set which has more in common with Chinese or Japanese than with typical 8-bit alphabetic encodings. I believe the composing jamo characters have only become available for practical use when they were included in Unicode.
For a messaging system in particular, you have two choices, only one of which makes proper sense, really: Design the protocol to tag the encoding of every transmitted character, and implement transcoding in all clients; or just use Unicode everywhere.
The one remaining challenge is to ensure that each client has the necessary fonts to display the glyphs they receive. Things are slowly improving, but this is a complex matter.

Answer 2

There is no out of band information. With the legacy systems you are expected to know the usual encodings and try them by yourself till one works. Which is why anything but unicode is dumb today (and anything but uTF-8 as unicode encoding is fail too). It is not true that no one uses unicode today. utf-8 is the default xml encoding, the default ietf W3C encoding, the default Linux encoding, etc. Building a new multilingual system around anything but utf-8 today is a big mistake.