Eutupon7.dvi

EÖtupon Te˜uqoc No. 7 — >Oktwbrioc 2001 A tutorial on character code issues Jukka K. Korpella Päivänsäteenkuja 4 as. 1FIN-02210 EspooFinlandEmail : [email protected] In computers and in data transmission between them, i.e. in digital data processing and transfer, data is internally presented as octets, as a rule. Anoctet is a small unit of data with a numerical value between 0 and 255, in-clusively. The numerical values are presented in the normal (decimal) notationhere, but notice that other presentations are used too, especially octal (base 8) or hexadecimal (base 16) notation. Octets are often called bytes, but in principle, octet is a more definite concept than byte. Internally, octets consist of eight bits (hence the name, from Latin but we need not go into bit level here. However, you might need to know what the phrase "first bit set" or "signbit set" means, since it is often used. In terms of numerical values of octets,it means that the value is greater than 127. In various contexts, such octetsare sometimes interpreted as negative numbers, and this may cause variousproblems.
Different conventions can be established as regards to how an octet or a sequence of octets presents some data. For instance, four consecutive octetsoften form a unit that presents a real number according to a specific standard.
We are here interested in the presentation of character data (or string data; astring is a sequence of characters) only.
In the simplest case, which is still widely used, one octet corresponds to one character according to some mapping table (encoding). Naturally, this allowsat most 256 different characters being represented. There are several differentencodings, such as the well-known ASCII encoding and the ISO Latin family of encodings. The correct interpretation and processing of character data ofcourse requires knowledge about the encoding used.
Previously the ASCII encoding was usually assumed by default (and it is still very common). Nowadays ISO Latin 1, which can be regarded as an Jukka K. Korpella extension of ASCII, is often the default. The current trend is to avoid giving such a special position to ISO Latin 1 among the variety of encodings.
The following definitions are not universally accepted and used. In fact, one of the greatest causes of confusion around character set issues is that terminol-ogy varies and is sometimes misleading.
character repertoire A set of distinct characters. No specific internal pre- sentation in computers or data transfer is assumed. The repertoire per sedoes not even define an ordering for the characters; ordering for sortingand other purposes is to be specified separately. A character repertoire isusually defined by specifying names of characters and a sample (or ref- erence) presentation of characters in visible form. Notice that a characterrepertoire may contain characters which look the same in some presen-tations but are regarded as logically distinct, such as Latin uppercase A,Cyrillic uppercase A, and Greek uppercase alpha.
character code A mapping, often presented in tabular form, which defines a one-to-one correspondence between characters in a character repertoire and a set of nonnegative integers. That is, it assigns a unique numericalcode, a code position, to each character in the repertoire. In addition tobeing often presented as one or more tables, the code as a whole can beregarded as a single table and the code positions as indexes. As synonymsfor "code position", the following terms are also in use: code number, codevalue, code element, code point, code set value - and just code. Note: Theset of nonnegative integers corresponding to characters need not consistof consecutive numbers; in fact, most character codes have "holes", suchas code positions reserved for control functions or for eventual future use to be defined later.
character encoding A method (algorithm) for presenting characters in dig- ital form by mapping sequences of code numbers of characters into sequences of octets. In the simplest case, each character is mapped to an integer in the range 0 - 255 according to a character code and these areused as such as octets. Naturally, this only works for character reper- toires with at most 256 characters. For larger sets, more complicated encodings are needed. Encodings have names, which can be regis- Character Code Issues Notice that a character code assumes or implicitly defines a character reper- toire. A character encoding could, in principle, be viewed purely as a methodof mapping a sequence of integers to a sequence of octets. However, quite oftenan encoding is specified in terms of a character code (and the implied characterrepertoire). The logical structure is still the following: A character repertoire specifies a collection of characters, such as "a", "!", and "ä".
A character code defines numeric codes for characters in a repertoire. For example, in the ISO 10646 character code the numeric codes for "a", "!", "ä", and "ı" (per mille sign) are 97, 33, 228, and 8240. (Note: Especially theper mille sign, presenting ı as a single character, can be shown incorrectlyon display or on paper. That would be an illustration of the symptoms of theproblems we are discussing.) A character encoding defines how sequences of numeric codes are presented as (i.e., mapped to) sequences of octets. In one possible encoding for ISO 10646, the string a!äı is presented as the following sequence of octets (using two octets for each character): 0, 97, 0, 33, 0, 228, 32, 48.
The phrase character set is used in a variety of meanings. It might denotes just a character repertoire but it may also refer to a character code, and quiteoften a particular character encoding is implied too.
Unfortunately the word charset is used to refer to an encoding, causing much confusion. It is even the official term to be used in several contexts byInternet protocols, in MIME headers.
Quite often the choice of a character repertoire, code, or encoding is pre- sented as the choice of a language. For example, Web browsers typically confusethings quite a lot in this area. A pulldown menu in a program might be labeled"Languages", yet consist of character encoding choices (only). A language set-ting is quite distinct from character issues, although naturally each languagehas its own requirements on character repertoire. Even more seriously, pro-grams and their documentation very often confuse the above-mentioned issueswith the selection of a font.
Jukka K. Korpella Examples of character codes The name ASCII, originally an abbreviation for "American Standard Code for Information Interchange", denotes an old character repertoire, code, and Most character codes currently in use contain ASCII as their subset in some sense. ASCII is the safest character repertoire to be used in data transfer.
However, not even all ASCII characters are "safe"! ASCII has been used and is used so widely that often the word ASCII refers to "text" or "plain text" in general, even if the character code is something else!The words "ASCII file" quite often mean any text file as opposite to a binaryfile.
The definition of ASCII also specifies a set of control codes ("control char- acters") such as linefeed (LF) and escape (ESC). But the character repertoireproper, consisting of the printable characters of ASCII, is the following (wherethe first item is the blank, or space, character) : The appearance of characters varies, of course, especially for some special A formal view on ASCII The character code defined by the ASCII stan- dard is the following: code values are assigned to characters consecutively inthe order in which the characters are listed above (rowwise), starting from 32(assigned to the blank) and ending up with 126 (assigned to the tilde character ). Positions 0 through 31 and 127 are reserved for control codes. They have standardized names and descriptions, but in fact their usage varies a lot.
The character encoding specified by the ASCII standard is very simple, and the most obvious one for any character code where the code numbers do notexceed 255: each code number is presented as an octet with the same value.
Octets 128 - 255 are not used in ASCII (This allows programs to use the first, most significant bit of an octet as a parity bit, for example).
Character Code Issues National variants of ASCII There are several national variants of ASCII.
In such variants, some special characters have been replaced by national letters(and other symbols). There is great variation here, and even within one countryand for one language there might be different variants. The original ASCII istherefore often referred to as US-ASCII ; the formal standard (by ANSI) isANSI X3.4-1986.
The international standard ISO 646 defines a character set similar to US-ASCII but with code positions corresponding to US-ASCII characters@[ ]{ } as "national use positions". It also gives some liberties with characters#$ ‘ . The standard also defines "international reference version (IRV)", whichis (in the 1991 edition of ISO 646) identical to US-ASCII.
Within the framework of ISO 646, and partly otherwise too, several "na- tional variants of ASCII" have been defined, assigning different letters andsymbols to the "national use" positions. Thus, the characters that appear inthose positions - including those in US-ASCII - are somewhat "unsafe" in inter-national data transfer, although this problem is losing significance. The trendis towards using the corresponding codes strictly for US-ASCII meanings; na-tional characters are handled otherwise, giving them their own, unique anduniversal code positions in character codes larger than ASCII. But old soft-ware and devices may still reflect various "national variants of ASCII".
The following table lists ASCII characters which might be replaced by other characters in national variants of ASCII. (That is, the code positions of theseUS-ASCII characters might be occupied by other characters needed for nationaluse.) The lists of characters appearing in national variants are not intended tobe exhaustive, just typical examples.
Almost all of the characters used in the national variants have been incor- porated into ISO Latin 1. Systems that support ISO Latin 1 in principle may still reflect the use of national variants of ASCII in some details; for example,an ASCII character might get printed or displayed according to some nationalvariant. Thus, even "plain ASCII text" is thereby not always portable from onesystem or application to another.
Subsets of ASCII for safety Mainly due to the "national variants" discussed above, some characters are less "safe" than other, i.e. more oftentransferred or interpreted incorrectly.
In addition to the letters of the English alphabet ("A" to "Z", and "a" to "z"), the digits ("0" to "9") and the space (" "), only the followingcharacters can be regarded as really "safe" in data transmission: Jukka K. Korpella official Unicode name National variants left square bracket Ö Ø ç Ñ 1/2 • right square bracket circumflex accent left curly bracket ä æ é à ˚ ¨ right curly bracket Even these characters might eventually be interpreted wrongly by the re- cipient, e.g. by a human reader seeing a glyph for "&" as something else than what it is intended to denote, or by a program interpreting "<" as startingsome special markup, "?" as being a so-called wildcard character, etc.
When you need to name things (e.g. files, variables, data fields, etc.), it is often best to use only the characters listed above, even if a wider characterrepertoire is possible. Naturally you need to take into account any additionalrestrictions imposed by the applicable syntax. For example, the rules of a pro-gramming language might restrict the character repertoire in identifier namesto letters, digits and one or two other characters.
The misnomer "8-bit ASCII" Sometimes the phrase "8-bit ASCII" is used. It follows from the discussion above that in reality ASCII is strictly andunambiguously a 7-bit code in the sense that all code positions are in the range0-127.
It is a misnomer used to refer to various character codes which are ex- tensions of ASCII in the following sense: the character repertoire containsASCII as a subset, the code numbers are in the range 0 - 255, and the codenumbers of ASCII characters equal their ASCII codes.
Another example: ISO Latin 1 alias ISO 8859-1 The ISO 8859-1 standard (which is part of the ISO 8859 family of stan- dards) defines a character repertoire identified as "Latin alphabet No. 1", Character Code Issues commonly called "ISO Latin 1", as well as a character code for it. The reper-toire contains the ASCII repertoire as a subset, and the code numbers for those characters are the same as in ASCII. The standard also specifies an encoding,which is similar to that of ASCII: each code number is presented simply as oneoctet.
In addition to the ASCII characters, ISO Latin 1 contains various accented characters and other letters needed for writing languages of Western Europe,and some special characters. These characters occupy code positions 160 - 255,and they are: The first of the characters above appears as space; it is the so-called no- break space. Naturally, the appearance of characters varies from one font to More examples: the Windows character set(s) In ISO 8859-1, code positions 128 - 159 are explicitly reserved for con- trol purposes; they "correspond to bit combinations that do not represent graphic characters". The so-called Windows character set (WinLatin1, or Windows code page 1252, to be exact) uses some of those positions for printable characters. Thus, the Windows character set is not identical with ISO 8859-1. It is, however, true that the Windows character set is much more similar to ISO 8859-1 than the so-called DOS character sets are. The Win- dows character set is often called "ANSI character set", but this is seriouslymisleading. It has not been approved by ANSI. (Historical background: Mi- crosoft based the design of the set on a draft for an ANSI standard. A glossary by Microsoft explicitly admits this.) Note that programs used on Windows systems may use a DOS character set; for example, if you create a text file using a Windows program and then use thetype command on DOS prompt to see its content, strange things may happen,since the DOS command interprets the data according to a DOS charactercode.
Jukka K. Korpella In the Windows character set, some positions in the range 128 - 159 are assigned to printable characters, such as "smart quotes", em dash, en dash, andtrademark symbol. Thus, the character repertoire is larger than ISO Latin 1.
The use of octets in the range 128 - 159 in any data to be processed by aprogram that expects ISO 8859-1 encoded data is an error which might causejust anything. They might for example get ignored, or be processed in a mannerwhich looks meaningful, or be interpreted as control characters.
The Windows character set exists in different variations, or "code pages" (CP), which generally differ from the corresponding ISO 8859 standard sothat it contains same characters in positions 128 - 159 as code page 1252.
(However, there are some more differences between ISO 8859-7 and WIN-1253(WinGreek)). What we have discussed here is the most usual one, resemblingISO 8859-1. In December 1999, Microsoft finally registered it under the name windows-1252. (The name cp-1252 has been used too, but it isn't offi-cially registered even as an alias name).
The ISO 8859 family There are several character codes which are extensions to ASCII in the same sense as ISO 8859-1 and the Windows character set. ISO 8859-1 itself is just a member of the ISO 8859 family of character codes. Those codes ex-tend the ASCII repertoire in different ways with different special characters (used in different languages and cultures). Just as ISO 8859-1 contains ASCIIcharacters and a collection of characters needed in languages of western (andnorthern) Europe, there is ISO 8859-2 alias ISO Latin 2 constructed similarlyfor languages of central/eastern Europe, etc. The ISO 8859 character codesare isomorphic in the following sense: code positions 0 - 127 contain the samecharacter as in ASCII, positions 128 - 159 are unused (reserved for control characters), and positions 160 - 255 are the varying part, used differently in different members of the ISO 8859 family.
The ISO 8859 character codes are normally presented using the obvious encoding: each code position is presented as one octet. Such encodings haveseveral alternative names in the official registry of character encodings, but the preferred ones are of the form ISO-8859-n.
Although ISO 8859-1 has been a de facto default encoding in many contexts, it has in principle no special role. And in practice, ISO 8859-15 alias ISO Latin 9 (!) will probably replace ISO 8859-1 to a great extent, since it contains the politically important symbol for euro.
Notes: ISO 8859-n is Latin alphabet no. n for n=1,2,3,4, but this corre- spondence is broken for the other Latin alphabets.
Character Code Issues The parts of ISO 8859 Latin alphabet No. 1 "Western", "West European" Latin alphabet No. 2 "Central European", "East European" Latin alphabet No. 3 "South European", "Maltese & Esperanto" Latin alphabet No. 4 "North European" Latin/Cyrillic alphabet (for Slavic languages) Latin/Arabic alphabet (for the Arabic language) Latin/Greek alphabet (for modern Greek) Latin/Hebrew alphabet (for Hebrew and Yiddish) Latin alphabet No. 5 Latin alphabet No. 6 "Nordic" (Sámi, Inuit, Icelandic) Latin/Thai alphabet (for the Thai language; draft (Part 12 has not been defined).
ISO 8859-13 Latin alphabet No. 7 Latin alphabet No. 8 Latin alphabet No. 9 Latin alphabet No. 10 for Romanian and various other languages Other "extensions to ASCII" In addition to the codes discussed above, there are other extensions to ASCII which utilize the code range 0 - 255 ("8-bit ASCII codes"), such as DOS character codes, or "code pages" (CP)In MS DOS systems, different character codes are used; they are called "code pages". The original American code page was CP 437, which has e.g.
some Greek letters, mathematical symbols, and characters which can be usedas elements in simple pseudo-graphics. Later CP 850 became popular, since itcontains letters needed for West European languages - largely the same lettersas ISO 8859-1, but in different code positions. Note that DOS code pages are quite different from Windows character codes, though the latter are sometimes called with names like cp-1252 (= windows-1252)! For further con-fusion, Microsoft now prefers to use the notion "OEM code page" for the DOScharacter set used in a particular country.
Macintosh character codeOn the Macs, the character code is more uniform than on PCs (although there are some national variants). The Mac character repertoire is a mixed combination of ASCII, accented letters, mathematical symbols, and other in-gredients.
Jukka K. Korpella Notice that many of these are very different from ISO 8859-1. They may have different character repertoires, and the same character often has differentcode values in different codes. For example, code position 228 is occupied byä (letter a with dieresis, or umlaut) in ISO 8859-1, by ð (Icelandic letter eth)in HP's Roman-8, by õ (letter o with tilde) in DOS code page 850, and per mille sign (ı)in Macintosh character code.
In general, full conversions between the character codes mentioned above are not possible. For example, the Macintosh character repertoire contains theGreek letter pi, which does not exist in ISO Latin 1 at all. Naturally, a text can be converted (by a simple program which uses a conversion table)from Macintosh character code to ISO 8859-1 if the text contains only thosecharacters which belong to the ISO Latin 1 character repertoire. Text presentedin Windows character code can be used as such as ISO 8859-1 encoded dataif it contains only those characters which belong to the ISO Latin 1 characterrepertoire.
Other "8-bit codes" All the character codes discussed above are "8-bit codes", eight bits are sufficient for presenting the code numbers and in practice the encoding (at least the normal encoding) is the obvious (trivial) one where each code position(thereby, each character) is presented as one octet (byte). This means that thereare 256 code positions, but several positions are reserved for control codes or left unused (unassigned, undefined).
Although currently most "8-bit codes" are extensions to ASCII in the sense described above, this is just a practical matter caused by the widespreaduse of ASCII. It was practical to make the "lower halves" of the character codes the same, for several reasons.
The standards ISO 2022 and ISO 4873 define a general framework for 8-bit codes (and 7-bit codes) and for switching between them. One of the basicideas is that code positions 128 - 159 (decimal) are reserved for use as control codes ("C1 controls"). Note that the Windows character sets do not comply with this principle.
To illustrate that other kinds of 8-bit codes can be defined than extensions to Ascii, we briefly consider the EBCDIC code, defined by IBM and once in widespread use on "mainframes" (and still in use). EBCDIC contains all ASCII characters but in quite different code positions. As an interesting detail, in EBCDIC normal letters A - Z do not all appear in consecutive codepositions. EBCDIC exists in different national variants (cf. to variants of Character Code Issues ISO 10646 (UCS) and Unicode ISO 10646 (officially: ISO/IEC 10646) is an international standard, by ISO and IEC. It defines UCS, Universal Character Set, which is a very large and growing character repertoire, and a character code for it. Currently tens of thousands of characters have been defined, and new amendments are definedfairly often. It contains, among other things, all characters in the characterrepertoires discussed above.
Unicode is a standard, by the Unicode Consortium, which defines a character repertoire and character code intended to be fully compatible withISO 10646, and an encoding for it. ISO 10646 is more general (abstract) innature, whereas Unicode "additional constraints on implementations to ensurethat they treat characters uniformly across platforms and applications", as theysay in the Unicode FAQ. Moreover, Unicode basically corresponds to "BasicMultilingual Plane (BMP)" of ISO 10646 (though there are mechanisms inUnicode to extend beyond BMP); however, other "planes" haven't even been defined yet.
The ISO 10646 and Unicode character repertoire can be regarded as a su- perset of most character repertoires in use. However, the code positions ofcharacters vary from one character code to another.
In practice, people usually talk about Unicode rather than ISO 10646, partly because we prefer names to numbers, partly because Unicode is more explicitabout the meanings of characters, partly because detailed information Unicodeis available on the Web.
Unicode version 1.0 used somewhat different names for some characters than ISO 10646. In Unicode version, 2.0, the names were made the same asin ISO 10646. New versions of Unicode are expected to add new characters mostly. Version 3.0, with a total number of 49,194 characters (38,887 in ver- sion 2.1), was published in February 2000.
The ISO 10646 standard has not been put onto the Web. It is available in printed form from ISO member bodies. But for most practical purposes, the same information is in the Unicode standard.
The "native" Unicode encoding, UCS-2, presents each code number as two consecutive octets m and n so that the number equals 256m + n. This means,to express it in computer jargon, that the code number is presented as a two- byte integer. This is a very obvious and simple encoding. However, it can be inefficient in terms of the number of octets needed. If we have normal Englishtext or other text which contains ISO Latin 1 characters only, the length of the Unicode encoded octet sequence is twice the length of the string in ISO8859-1 encoding.
Jukka K. Korpella It is somewhat debatable whether Unicode defines an encoding or just a character code. However, it refers to code values being presentable as 16-bitintegers, and it seems to imply the corresponding two-octet representation. Inprinciple, Unicode requires that "Unicode values can be stored in native 16-bit machine words" and "does not specify any order of bytes inside a Unicodevalue". Thus, it allows "little-endian" presentation where the least significant byte precedes the most significant byte, if agreed on by higher-level protocols.
ISO 10646 can be, and often is, encoded in other ways, too, such as the following encodings: UTF-8Character codes less than 128 (effectively, the ASCII repertoire) are pre- sented "as such", using one octet for each code (character) All other codesare presented, according to a relatively complicated method, so that one code(character) is presented as a sequence of two to six octets, each of which isin the range 128 - 255. This means that in a sequence of octets, octets in therange 0 - 127 ("bytes with most significant bit set to 0") directly representASCII characters, whereas octets in the range 128 - 255 ("bytes with most significant bit set to 1") are to be interpreted as really encoded presentationsof characters.
UTF-7Each character code is presented as a sequence of one or more octets in the range 0 - 127 ("bytes with most significant bit set to 0", or "seven-bitbytes", hence the name). Most ASCII characters are presented as such, each as one octet, but for obvious reasons some octet values must be reserved foruse as "escape" octets, specifying the octet together with a certain number ofsubsequent octets forms a multi-octet encoded presentation of one character.
IETF Policy on Character Sets and Languages (RFC 2277) clearly favors UTF-8. It requires support to it in Internet protocols (and doesn't even men- tion UTF-7). Note that UTF-8 is efficient, if the data consists dominantly ofASCII characters with just a few "special characters" in addition to them, andreasonably efficient for dominantly ISO Latin 1 text.
The implementation of Unicode support is a long and mostly gradual process. Unicode can be supported by programs on any operating systems,although some systems may allow much easier implementation than others;this mainly depends on whether the system uses Unicode internally so thatsupport to Unicode is "built-in".
Even in circumstances where Unicode is supported in principle, the support usually does not cover all Unicode characters. For example, a font available may cover just some part of Unicode which is practically important in some Character Code Issues area. On the other hand, for data transfer it is essential to know which Unicodecharacters the recipient is able to handle. For such reasons, various subsets of the Unicode character repertoire have been and will be defined. For exam-ple, the Minimum European Subset specified by ENV 1973:1995 is intended to provide a first step towards the implementation of large character sets inEurope. There are also three Multilingual European Subsets (MES-1, MES-2, MES-3, with MES-2 based on the Minimum European Subset).
In addition to international standards, there are company policies which define various subsets of the character repertoire. A practically important oneis Microsoft's "Windows Glyph List 4" (WGL4), or "PanEuropean" char- Unicode characters are often referred to using a notation of the form U+nnnn where nnnn is a four-digit hexadecimal notation of the code value.
For example, U+0020 means the space character (with code value 20 in hexadec-imal, 32 in decimal). Notice that such notations identify a character throughits Unicode code value, without referring to any particular encoding. There areother ways to mention (identify) a character, too.
More about the character concept An "A" (or any other character) is something like a Platonic entity: it is the idea of an "A" and not the "A" itself.
– Michael E. Cohen: Text and Fonts in a Multi-lingual Cross-platform The character concept is very fundamental for the issues discussed here but difficult to define exactly. The more fundamental concepts we use, the harderit is to give good definitions. (How would you define "life"? Or "structure"?)Here we will concentrate on clarifying the character concept by indicating whatit does not imply.
The Unicode standard describes characters as "the smallest components of written language that have semantic value", which is somewhat misleading. Acharacter such as a letter can hardly be described as having a meaning (seman-tic value) in itself. Moreover, a character such as ú (letter u with acute accent),which belongs to Unicode, can often be regarded as consisting of smaller com-ponents: a letter and a diacritic. And in fact the very definition of the characterconcept in Unicode is the following: Jukka K. Korpella abstract character : a unit of information used for the organization, control, or representation of textual data.
Control characters (control codes) The rôle of the so-called control characters in character codes is somewhat obscure. Character codes often contain code positions which are not assigned toany visible character but reserved for control purposes. For example, in com-munication between a terminal and a computer using the ASCII code, the computer could regard octet 3 as a request for terminating the currently run- ning process. Some older character code standards contain explicit descriptionsof such conventions whereas newer standards just reserve some positions forsuch usage, to be defined in separate standards or agreements such as "C0 controls" and "C1 controls", or specifically ISO 6429. And although the definition quoted above suggests that "control characters" might be regardedas characters in the Unicode terminology, perhaps it is more natural to regardthem as control codes.
Control codes can be used for device control such as cursor movement, page eject, or changing colors. Quite often they are used in combination withcodes for graphic characters, so that a device driver is expected to interpret thecombination as a specific command and not display the graphic character(s)contained in it. For example, in the classical VT100 controls, ESC followed by the code corresponding to the letter "A" or something more complicated(depending on mode settings) moves the cursor up. To take a different example,the Emacs editor treats ESC A as a request to move to the beginning of a sentence. Note that the ESC control code is logically distinct from the ESCkey in a keyboard, and many other things than pressing ESC might cause theESC control code to be sent. Also note that phrases like "escape sequences" are often used to refer to things that don't involve ESC at all and operate at aquite different level.
One possible form of device control is changing the way a device interprets the data (octets) that it receives. For example, a control code followed by somedata in a specific format might be interpreted so that any subsequent octets tobe interpreted according to a table identified in some specific way. This is oftencalled "code page switching", and it means that control codes could be usedchange the character encoding. And it is then more logical to consider the control codes and associated data at the level of fundamental interpretation ofdata rather than direct device control. The international standard ISO 2022 defines powerful facilities for using different 8-bit character codes in a document.
Character Code Issues Widely used formatting control codes include carriage return (CR), line- feed (LF), and horizontal tab (HT), which in ASCII occupy code positions 13, 10, and 9. The names (or abbreviations) suggest generic meanings, but theactual meanings are defined partly in each character code definition, partly -and more importantly - by various other conventions "above" the characterlevel. The "formatting" codes might be seen as a special case of device control,in a sense, but more naturally, a CR or a LF or a CR LF pair (to mention themost common conventions) when used in a text file simply indicates a new line.
The HT (TAB) character is often used for real "tabbing" to some predefined writing position. But it is also used e.g. for indicating data boundaries, with-out any particular presentational effect, for example in the widely used "tabseparated values" (TSV) data format.
A glyph - a visual appearance It is important to distinguish the character concept from the glyph concept.
A glyph is a presentation of a particular shape which a character may have whenrendered or displayed. For example, the character Z might be presented as aboldface Z or as an italic Z, and it would still be a presentation of the same character. On the other hand, lower-case z is defined to be a separate character- which in turn may have different glyph presentations.
This is ultimately a matter of definition: a definition of a character reper- toire specifies the "identity" of characters, among other things. One could definea repertoire where uppercase Z and lowercase z are just two glyphs for the samecharacter. On the other hand, one could define that italic Z is a character dif-ferent from normal Z, not just a different glyph for it. In fact, in Unicode forexample there are several characters which could be regarded as typographicvariants of letters only, but for various reasons Unicode defines them as separatecharacters. For example, mathematicians use a variant of letter N to denote theset of natural numbers (0, 1, 2, .), and this variant is defined as being a sep-arate character ("double-struck capital N") in Unicode. There are some morenotes on the identity of characters below.
What's in a name? The names of characters are assigned identifiers rather than definitions.
Typically the names are selected so that they contain only letters A - Z, spaces,and hyphens; often uppercase variant is the reference spelling of a charactername. The same character may have different names in different definitions ofcharacter repertoires. Generally the name is intended to suggest a generic Jukka K. Korpella meaning and scope of use. But the Unicode standard warns (mentioning full stop as an example of a character with varying usage): A character may have a broader range of use than the most literal interpre- tation of its name might indicate; coded representation, name, and represen-tative glyph need to be taken in context when establishing the semantics of acharacter.
When a character repertoire is defined (e.g. in a standard), some particular glyph is often used to describe the appearance of each character, but this shouldbe taken as an example only. The Unicode standard specifically says that great variation is allowed between "representative glyph" appearing in the standardand a glyph used for the corresponding character: Consistency with the representative glyph does not require that the images be identical or even graphically similar; rather, it means that both images aregenerally recognized to be representations of the same character. Representingthe character U+0061 Latin small letter a by the glyph "X" would violate itscharacter identity.
Thus, the definition of a repertoire is not a matter of just listing glyphs, but neither is it a matter of defining exactly the meanings of characters. It'sactually an exception rather than a rule that a character repertoire definitionexplicitly says something about the meaning and use of a character.
Possibly some specific properties (e.g. being classified as a letter or having numeric value in the sense that digits have) are defined, as in the Unicode database, but such properties are rather general in nature.
This vagueness may sound irritating, and it often is. But an essential point to be noted is that quite a lot of information is implied. You are expected to deduce what the character is, using both the character name and its repre-sentative glyph, and perhaps context too, like the grouping of characters underdifferent headings like "currency symbols".
A repertoire of glyphs comprises a font. In a more technical sense, as the implementation of a font, a font is a numbered set of glyphs. The numberscorrespond to code positions of the characters (presented by the glyphs).
Thus, a font in that sense is character code dependent. An expression like Character Code Issues "Unicode font" refers to such issues and does not imply that the font containsglyphs for all Unicode characters.
It is possible that a font which is used for the presentation of some character repertoire does not contain a different glyph for each character. For example,although characters such as Latin uppercase A, Cyrillic uppercase A, and Greekuppercase alpha are regarded as distinct characters (with distinct code values)in Unicode, a particular font might contain just one A which is used to present all of them.
You should never use a character just because it "looks right" or "almost right". Characters with quite different purposes and meanings may well looksimilar, or almost similar, in some fonts at least. Using a character as a surro- gate for another for the sake of apparent similarity may lead to great confusion.
Consider, for example, the so-called sharp s (es-zed), which is used in the Ger-man language. Some people who have noticed such a character in the ISO Latin 1 repertoire have thought "wow, here we have the beta character!".
In many fonts, the sharp s (ß) really looks more or less like the Greek low-ercase beta character (β). But it must not be used as a surrogate for beta.
You wouldn't get very far with it, really; what's the big idea of having betawithout alpha and all the other Greek letters? More seriously, the use of sharps in place of beta would confuse text searches, spelling checkers, indexers, etc.;an automatic converter might well turn sharp s into ss; and some font mightpresent sharp s in a manner which is very different from beta.
Identity of characters: a matter of definition The identity of characters is defined by the definition of a character repertoire. Thus, it is not an absolute concept but relative to the repertoire; some repertoire might contain a character with mixed usage while another de-fines distinct characters for the different uses. For instance, the ASCII reper- toire has a character called hyphen. It is also used as a minus sign (as well as a substitute for a dash, since ASCII contains no dashes). Thus, that ASCIIcharacter is a generic, multipurpose character, and one can say that in ASCIIhyphen and minus are identical. But in Unicode, there are distinct characters named "hyphen" and "minus sign" (as well as different dash characters). Forcompatibility, the old ASCII character is preserved in Unicode, too (in the oldcode position, with the name hyphen-minus).
Similarly, as a matter of definition, Unicode defines characters for micro sign, n-ary product, etc., as distinct from the Greek letters (small mu, capital pi, etc.) they originate from. This is a logical distinction and does notnecessarily imply that different glyphs are used. The distinction is important Jukka K. Korpella e.g. when textual data in digital form is processed by a program (which "sees"the code values, through some encoding, and not the glyphs at all). Notice thatUnicode does not make any distinction e.g. between the greek small letter pi (π), and the mathematical symbol pi denoting the well-known constant 3.14159. (i.e. there is no separate symbol for the latter). For the ohm sign (Ω), there is a specific character (in the Symbols Area), but it is defined as beingcompatibility equivalent to greek capital letter omega (Ω), i.e. there are two separate characters but they are equivalent. On the other hand, it makesa distinction between greek capital letter pi (Π) and the mathematical symbol n-ary product ( ), so that they are not compatibility equivalents.
If you think this doesn't sound quite logical, you are not the only one to think so. But the point is that for symbols resembling Greek letter and used invarious contexts, there are three possibilities in Unicode: — the symbol is regarded as identical to the Greek letter (just as its partic- — the symbol is included as a separate character but only for compatibility and as compatibility equivalent to the Greek letter — the symbol is regarded as a completely separate character.
You need to check the Unicode references for information about each individual symbol. As a rough rule of thumb about symbols looking like Greekletters, mathematical operators (like summation) exist as independent char-acters whereas symbols of quantities and units (like pi and ohm) are eithercompatibility characters or identical to Greek letters.
Failures to display a character In addition to the fact that the appearance of a character may vary, it is quite possible that some program fails to display a character at all.
Perhaps the program cannot interpret a particular way in which the character ispresented. The reason might simply be that some program-specific way had been used to denote the character and a different program is in use now. (Thishappens quite often even if "the same" program is used; for example, InternetExplorer version 4.0 is able to recognize α as denoting the Greek letteralpha (α) but IE 3.0 is not and displays the notation literally.) And naturally itoften occurs that a program does not recognize the basic character encoding of the data, either because it was not properly informed about the encodingaccording to which the data should be interpreted or because it has not beenprogrammed to handle the particular encoding in use.
Character Code Issues But even if a program recognizes some data as denoting a character, it may well be unable to display it since it lacks a glyph for it. Often it will help if the user manually checks the font settings, perhaps manually trying to find a rich enough font. (Advanced programs could be expected to do this automaticallyand even to pick up glyphs from different fonts, but such expectations aremostly unrealistic at present.) But it's quite possible that no such font can befound. As an important detail, the possibility of seeing e.g. Greek characterson some Windows systems depends on whether "internationalization support"has been installed.
A well-design program will in some appropriate way indicate its inability to display a character. For example, a small rectangular box, the size of a charac-ter, could be used to indicate that there is a character which was recognized butcannot be displayed. Some programs use a question mark, but this is risky - howis the reader expected to distinguish such usage from the real "?" character? Linear text vs. mathematical notations Although several character repertoires, most notably that of ISO 10646 and Unicode, contain mathematical and other symbols, the presentation of mathematical formulas is essentially not a character level problem. At the character level, symbols like integration or n-ary summation can be definedand their code positions and encodings defined, and representative glyphs shown, and perhaps some usage notes given. But the construction of real for-mulas, e.g. for a definite integral of a function, is a different thing, no matterwhether one considers formulas abstractly (how the structure of the formulais given) or presentationally (how the formula is displayed on paper or onscreen). To mention just a few approaches to such issues, the TEX system is widely used by mathematicians to produce high-quality presentations of for-mulas, and MathML is an ambitious project for creating a markup language for mathematics so that both structure and presentation can be handled.
In other respects, too, character standards usually deal with plain text only. Other structural or presentational aspects, such as font variation, are tobe handled separately. However, there are characters which would now be con-sidered as differing in font only but for historical reasons regarded as distinct.
Compatibility characters There is a large number of compatibility characters in ISO 10646 and Unicode which are variants of other characters. They were included for com- patibility with other standards so that data presented using some other code Jukka K. Korpella can be converted to ISO 10646 and back without losing information. The Uni-code standard says: Compatibility characters are included in the Unicode Standard only to rep- resent distinctions in other base standards and would not otherwise have beenencoded. However, replacing a compatibility character by its decompositionmay lose round-trip convertibility with a base standard.
There is a large number of compatibility characters in the Compatibility Area but also scattered around the Unicode space. The Unicode database contains, for each character, a field (the sixth one) which specifies whether it isa compatibility character as well as its eventual compatibility decomposition.
Thus, to take a simple example, superscript two (2) is an ISO Latin 1 character with its own code position in that standard. In ISO 10646 way ofthinking, it would have been treated as just a superscript variant of digit two.
But since the character is contained in an important standard, it was includedinto ISO 10646, though only as a "compatibility character". The practical rea-son is that now one can convert from ISO Latin 1 to ISO 10646 and backand get the original data. This does not mean that in the ISO 10646 philoso-phy superscripting (or subscripting, italics, bolding etc.) would be irrelevant;rather, they are to be handled at another level of data presentation, such assome special markup.
The definition of Unicode indicates our sample character, superscript two, as a compatibility character with the compatibility decomposition "<su-per> + 0032 2". Here "<super>" is a semi-formal way of referring to what isconsidered as typographic variation, in this case superscript style, and "00322" shows the hexadecimal code of a character and the character itself.
Some compatibility characters have compatibility decompositions con- sisting of several characters. Due to this property, they can be said to representligatures in the broad sense. For example, latin small ligature fi (U+FB01) hasthe obvious decomposition consisting of letters "f" and "i". It is still a distinctcharacter in Unicode, but in the spirit of Unicode, we should not use it ex- cept for storing and transmitting existing data which contains that character.
Generally, ligature issues should be handled outside the character level, e.g. se-lected automatically by a formatting program or indicated using some suitablemarkup.
Note that the word ligature can be misleading when it appears in a character name. In particular, the old name of the character "æ", latin small letter ae(U+00E6), is latin small ligature ae, but it is not a ligature of "a" and "e" inthe sense described above. It has no compatibility decomposition.
In comp.fonts FAQ, the term ligature is defined as follows: Character Code Issues A ligature occurs where two or more letterforms are written or printed as a unit. Generally, ligatures replace characters that occur next to each other whenthey share common components. Ligatures are a subset of a more general classof figures called "contextual forms." Compositions and decompositions A diacritic mark, i.e. an additional graphic such as an accent or cedilla attached to a character, can be treated in different ways when defining a char-acter repertoire. In the Unicode approach, there are separate characters calledcombining diacritical marks. The general idea is that you can express a vastset of characters with diacritics by representing them so that a base characteris followed by one or more (!) combining (non-spacing) diacritic marks. And aprogram which displays such a construct is expected to do rather clever thingsin formatting, e.g. selecting a particular shape for the diacritic according tothe shape of the base character. This requires Unicode support at implemen- tation level 3. Most programs currently in use are totally incapable of doing anything meaningful with combining diacritic marks. But there is some simplesupport to them in Internet Explorer for example, though you would need afont which contains the combining diacritics (such as Arial Unicode MS); then IE can handle simple combinations reasonably.
Thus, in practical terms, in order to use a character with a diacritic mark, you should try to find it as a precomposed character. A precomposed character,also called composite character or decomposable character, is one that has acode position (and thereby identity) of its own but is in some sense equiva- lent to a sequence of other characters. There are lots of them in Unicode, andthey cover the needs of most (but not all) languages of the world, but not e.g.
the presentation of the International phonetic alphabet by IPA which, in its general form, requires several different diacritic marks. For example, thecharacter latin small letter a with diaeresis (U+00E4, ä) is, by Unicode defi-nition, decomposable to the sequence of the two characters latin small letter a(U+0061) and combining diaeresis (U+0308). This is at present mostly a the-oretic possibility. Generally by decomposing all decomposable characters onecould in many cases simplify the processing of textual data (and the resultingdata might be converted back to a format using precomposed characters).
Jukka K. Korpella Typing characters Just pressing a key? Typing characters on a computer may appear deceptively simple: you press a key labeled "A", and the character "A" appears on the screen. You alsoexpect "A" to be included into a disk file when you save what you are typing,you expect "A" to appear on paper if you print your text, and you expect "A"to be sent if you send your product by E-mail or something like that. And youexpect the recipient to see an "A".
Thus far, you should have learned that the presentation of a character in computer storage or disk or in data transfer may vary a lot. You have probablyrealized that especially if it's not the common "A" but something more special(say, an "A" with an accent), strange things might happen, especially if datais not accompanied with adequate information about its encoding.
But you might still be too confident. You probably expect that on your system at least things are simpler than that. If you use your very own, very per-sonal computer and press the key labeled "A" on its keyboard, then shouldn'tit be evident that in its storage and processor, on its disk, on its screen it'sinvariably "A"? Can't you just ignore its internal character code and characterencoding? Well, probably yes - with "A". I wouldn't be so sure about "Ä", forinstance. (On Windows systems, for example, DOS mode programs differ fromgenuine Windows programs in this respect; they use a DOS character code.) When you press a key on your keyboard, then what actually happens is this. The keyboard sends the code of a character to the processor. The processorthen, in addition to storing the data internally somewhere, normally sends it tothe display device. Now, the keyboard settings and the display settings mightbe different from what you expect. Even if a key is labeled "Ä", it mightsend something else than the code of "Ä" in the character code used in yourcomputer. Similarly, the display device, upon receiving such a code, might beset to display something different. Such mismatches are usually undesirable,but they are definitely possible.
Moreover, there are often keyboard restrictions. If your computer uses in- ternally, say, ISO Latin 1 character repertoire, you probably won't find keys for all 191 characters in it on your keyboard. And for Unicode, it would be quite impossible to have a key for each character! Different keyboards are used,often according to the needs of particular languages. For example, keyboardsused in Sweden often have a key for the ˚ a character but seldom a key for ñ.
Quite often some keys have multiple uses via various "composition" keys.
Character Code Issues Program-specific methods for typing characters Thus, you often need program-specific ways of entering characters from a keyboard, either because there is no key for a character you need or there isbut it does not work (properly). Three important examples of such ways: — On Windows systems, you can (usually - some application programs may override this) produce any character in the Windows character set (naturally, in its Windows encoding) as follows: Press down the Alt key and keep it down. Then type, using the separate numeric keypad (not the numbers above the letter keys!), the four-digit code of the character indecimal. Finally release the Alt key. Notice that the first digit is always 0,since the code values are in the range 32 - 255 (decimal). For instance,to produce the letter "Ä" (which has code 196 in decimal), you wouldpress Alt down, type 0196 and then release Alt. Upon releasing Alt, thecharacter should appear on the screen. In MS Word, the method worksonly if Num Lock is set. This method is often referred to as Alt-0nnn.
(If you omit the leading zero, i.e. use Alt-nnn, the effect is different,since that way you insert the character in code position nnn in the DOScharacter code! For example, Alt-196 would probably insert a graphiccharacter which looks somewhat like a hyphen. There are variations inthe behavior of various Windows programs in this area, and using thoseDOS codes is best avoided).
— In the Emacs editor (which is popular especially on Unix systems), you can produce any ISO Latin 1 character by typing first control-Q, then its code as a three-digit octal number. To produce "Ä", you would thus type control-Q followed by the three digits 304 (and expect the "Ä" characterto appear on screen). This method is often referred to as C-Q-nnn. (Thereare other ways of entering many ISO Latin 1 characters in Emacs, too.) — Programs often process some keyboard key combinations, often in- volving the use of an Alt or Alt Gr key or some other "composition key", by converting them to special characters. In fact, even the well-knownshift key is a composition key: it is used to modify the meaning of an-other key, e.g. by changing a letter to uppercase or turning a digit key toa special character key. Such things are not just "program-specific"; theyalso depend on the program version and settings (and on the keyboard, ofcourse). For example, in order to support the euro sign, various meth- ods have been developed, e.g. by Microsoft so that pressing the "e" keywhile keeping the Alt Gr key pressed down might produce the euro sign -in some encoding! But this may require a special "euro update", and the key combinations vary even when we consider Microsoft products Jukka K. Korpella only. So it would be quite inappropriate to say e.g. "to type the euro, useAltGr+e" as general, unqualified advice.
The last method above could often be called "device dependent" rather than program specific, since the program that performs the conversion mightbe a keyboard driver. In that case, normal programs would have all their input from the keyboard processed that way. This method may also involve the useof auxiliary keys for typing characters with diacritic marks such as "á". Such an auxiliary key is often called dead key, since just pressing ít causes nothing;it works only in combination with some other key. For example, depending onthe keyboard and the driver, you might be able to produce "á" by pressing firsta key labeled with the acute accent ('), then the "a" key. My keyboard has twokeys for such purposes: one with the acute accent and the grave accent (‘) aboveit (meaning I need to use the shift key for it) and one with the dieresis (¨) and the circumflex ( ) above it and the tilde (∼) below or left to it (meaning Ineed to use Alt Gr for it), so I can produce ISO Latin 1 characters with those diacritics. Note that this does not involve any operation on the characters '‘¨∼ - the keyboard does not send those characters at all in such situations. If Itry to enter that way a character outside the ISO Latin 1 repertoire, I get justthe diacritic as a separate character followed by the normal character, e.g. "j". To enter the diacritic itself, such as the tilde (∼), I may need to press the space bar so that the tilde diacritic combines with the blank (producing ∼)instead of a letter (producing e.g. "ã"). Your situation may well be different,in part or entirely. For example, a typical French keyboard has separate keysfor those accented characters which are used in French (e.g. "à") and no keyfor the accents themselves, but there is a key for attaching the circumflex orthe dieresis in the manner outlined above.
"Escape" notations ("meta notations") for characters It is often possible to use various "escape" notations for characters. This rather vague term means notations which are afterwards converted to (or justdisplayed as) characters according to some specific rules by some programs.
They depend on the markup, programming, or other language (in a broad buttechnical meaning for "language", so that data formats can be included buthuman languages are excluded). If different languages have similar conventionsin this respect, a language designer may have picked up a notation from anexisting language, or it might be a coincidence.
The phrase "escape notations" or even "escapes" for short is rather widespread, and it reflects the general idea of escaping from the limitationsof a character repertoire or device or protocol or something else. So it's used Character Code Issues here, although a name like meta notations might be better. It is any case es-sential to distinguish these notations from the use of the ESC (escape) control code in ASCII and other character codes.
— In the PostScript language, characters have names, such as Adieresis for Ä, which can be used to denote them according to certain rules.
— In the RTF data format, the notation 'c4 is used to denote Ä.
— In TEX systems, there are different ways of producing characters, possibly depending on the "packages" used. Examples of ways to produce Ä: "A, symbol{196}, char'0304, capitaldieresis{A}.
— In the HTML language one can use the notation Ä for the char- acter Ä. In the official HTML terminology, such notations are called en- tity references (denoting characters). It depends on HTML version which entities are defined, and it depends on a browser which entities are actually supported.
— In HTML, one can also use the notation Ä for the character Ä.
Generally, in any SGML based system, or "SGML application" as the jargon goes, a numeric character reference (or, actually, just characterreferences) of the form &#number ; can be used, and it refers to thecharacter which is in code position n in the character code defined for the "SGML application" in question. This is actually very simple:you specify a character by its index (position, number). But in SGMLterminology, the character code which determines the interpretation of&#number ; is called, quite confusingly, the document character set. ForHTML, the "document character set" is ISO 10646 (or, to be exact, a subset thereof, depending on HTML version). A most essential point isthat for HTML, the "document character set" is completely independentof the encoding of the document! The so-called character entity refer-ences like Ä in HTML can be regarded as symbolic names definedfor some numeric character references.
— In the C programming language, one can usually write 0304 to de- note Ä within a string constant, although this makes the program char-acter code dependent.
As you can see, the notations typically involve some (semi-)mnemonic name or the code number of the character, in some number system. (The ISO 8859-1 code number for our example character Ä is 196 in decimal, 304 in octal, C4 in hexadecimal). And there is some method of indicating that the Jukka K. Korpella letters or digits are not to be taken as such but as part of a special notationdenoting a character. Often some specific character such as the backslash is used as an "escape character". This implies that such a character cannotbe used as such in the language or format but must itself be "escaped"; forexample, to include the backslash itself into a string constant in C, you needto write it twice ( ).
In cases like these, the character itself does not occur in a file (such as an HTML document or a C source program). Instead, the file contains the"escape" notation as a character sequence, which will then be interpreted ina specific way by programs like a Web browser or a C compiler. One can ina sense regard the "escape notations" as encodings used in specific contexts upon specific agreements.
There are also "escape notations" which are to be interpreted by human readers directly. For example, when sending E-mail one might use A" (letter A followed by a quotation mark) as a surrogate for Ä (letter A with diere-sis), or one might use AE instead of Ä. The reader is assumed to understandthat e.g. A" on display actually means Ä. Quite often the purpose is to useASCII characters only, so that the typing, transmission, and display of thecharacters is "safe". But this typically means that text becomes very messy;the Finnish word Hämäläinen does not look too good or readable when writtenas Ha"ma"la"inen or Haemaelaeinen. Such usage is based on special (thoughoften implicit) conventions and can cause a lot of confusion when there is nomutual agreement on the conventions, especially because there are so many ofthem. (For example, to denote letter a with acute accent, á, a convention mightuse the apostrophe, a', or the solidus, a/, or the acute accent, a', or somethingelse).
How to mention (identify) a character There are also various ways to identify a character when it cannot be used as such or when the appearance of a character is not sufficient identification.
This might be regarded as a variant of the "escape notations for human readers"discussed above, but the pragmatic view is different here. We are not primarilyinterested in using characters in running text but in specifying which characteris being discussed.
For example, when discussing the Cyrillic letter that resembles the Latin letter E (and may have an identical or very similar glyph, and is transliterated as E according to ISO 9), there are various options: — "Cyrillic E"; this is probably intuitively understandable in this case, and can be seen as referring either to the similarity of shape or to the translit- Character Code Issues eration equivalence; but in the general case these interpretations do notcoincide, and the method is otherwise vague too — "U+0415"; this is a unique identification but requires the reader to know the idea of U+nnnn notations — "cyrillic capital letter ie" (using the official Unicode name) or "cyrillic IE" (using an abridged version); one problem with this is that the namescan be long even if simplified, and they still cannot be assumed to beuniversally known even by people who recognize the character — "KE02", which uses the special notation system defined in ISO 7350; the system uses a compact notation and is marginally mnemonic (K = kirillica'Cyrillics'; the numeric codes indicate small/capital letter variation andthe use of diacritics) — any of the "escape" notations discussed above, such as "E=" by RFC 1345 or "Е" in HTML; this can be quite adequate in a context where the reader can be assumed to be familiar with the partic-ular notation.
Information about encoding The need for information about encoding It is hopefully obvious from the preceding discussion that a sequence of octets can be interpreted in a multitude of ways when processed as characterdata. By looking at the octet sequence only, you cannot even know whethereach octet presents one character or just part of a two-octet presentation ofa character, or something more complicated. Sometimes one can guess theencoding, but data processing and transfer shouldn't be guesswork.
Naturally, a sequence of octets could be intended to present other than character data, too. It could be an image in a bitmap format, or a computerprogram in binary form, or numeric data in the internal format used in com-puters.
This problem can be handled in different ways in different systems when data is stored and processed within one computer system. For data transmis-sion, a platform-independent method of specifying the general format and theencoding and other relevant information is needed. Such methods exist, al-though they not always used widely enough. People still send each other datawithout specifying the encoding, and this may cause a lot of harm. Attach-ing a human-readable note, such as a few words of explanation in an E-mail Jukka K. Korpella message body, is better than nothing. But since data is processed by programswhich cannot understand such notes, the encoding should be specified in astandardized computer-readable form.
The MIME solution Media types Internet media types, often called MIME media types, can be used to specify a major media type ("top level media type", such as text),a subtype (such as html), and an encoding (such as iso-8859-1). They wereoriginally developed to allow sending other than plain ASCII data by E-mail.
They can be (and should be) used for specifying the encoding when data issent over a network, e.g. by E-mail or using the HTTP protocol on the World Character encoding ("charset") information The technical term used to denote a character encoding in the Internet media type context is "character set", abbreviated "charset". This has caused a lot of confusion, since "set" caneasily be understood as repertoire! Specifically, when data is sent in MIME format, the media type and encod- ing are specified in a manner illustrated by the following example:Content-Type: text/html; charset=iso-8859-1This specifies, in addition to saying that the media type is text and subtype ishtml, that the character encoding is ISO 8859-1.
The official registry of "charset" (i.e., character encoding) names, with references to documents defining their meanings, is kept by IANA athttp://www.iana.org/assignments/character-sets.
Several character encodings have alternate (alias) names in the registry.
For example, the basic (ISO 646) variant of ASCII can be called "ASCII" or "ANSI_X3.4-1968" or "cp367" (plus a few other names); the preferred namein MIME context is, according to the registry, "US-ASCII". Similarly, ISO 8859-1 has several names, the preferred MIME name being "ISO-8859-1". The "native" encoding for Unicode, UCS-2, is named "ISO-10646-UCS-2" there.
MIME headers The Content-Type information is an example of informa- tion in a header. Headers relate to some data, describing its presentation andother things, but are passed as logically separate from it. Possible headers andtheir contents are defined in the basic MIME specification, RFC 2045. Ad- equate headers should normally be generated automatically by the softwarewhich sends the data (such as a program for sending E-mail, or a Web server)and interpreted automatically by receiving software (such as a program forreading E-mail, or a Web browser). In E-mail messages, headers precede the Character Code Issues message body; it depends on the E-mail program whether and how it displaysthe headers. For Web documents, a Web server is required to send headerswhen it delivers a document to a browser (or other user agent) which has senta request for the document.
An auxiliary encoding: Quoted-Printable (QP) The MIME specification defines, among many other things, the general purpose "Quoted-Printable" (QP) encoding which can be used to present any sequence of octets as a sequence of such octets which correspond to ASCII characters. This implies that the sequence of octets becomes longer, and if itis read as an ASCII string, it can be incomprehensible to humans. But whatis gained is robustness in data transfer, since the encoding uses only "safe"ASCII characters which will most probably get through any component in thetransfer unmodified.
Basically, QP encoding means that most octets smaller than 128 are used as such, whereas larger octets and some of the small ones are presented as follows:octet n is presented as a sequence of three octets, corresponding to ASCIIcodes for the = sign and the two digits of the hexadecimal notation of n. If QPencoding is applied to a sequence of octets presenting character data accordingto ISO 8859-1 character code, then effectively this means that most ASCII characters (including all ASCII letters) are preserved as such whereas e.g. theISO 8859-1 character ä (code position 228 in decimal, E4 in hexadecimal) is encoded as =E4. (For obvious reasons, the equals sign = itself is among thefew ASCII characters which are encoded. Being in code position 61 in decimal,3D in hexadecimal, it is encoded as =3D.) Notice that encoding ISO 8859-1 data this way means that the character code is the one specified by the ISO 8859-1 standard, whereas the characterencoding is different from the one specified (or at least suggested) in that stan-dard. Since QP only specifies the mapping of a sequence of octets to anothersequence of octets, it is a pure encoding and can be applied to any characterdata, or to any data for that matter.
Naturally, Quoted-Printable encoding needs to be processed by a program which knows it and can convert it to human-readable form. It looks ratherconfusing when displayed as such. Roughly speaking, one can expect mostE-mail programs to be able to handle QP, but the same does not apply tonewsreaders (or Web browsers). Therefore, you should normally use QP inE-mail only.
Jukka K. Korpella How MIME should work in practice Basically, MIME should let people communicate smoothly without hin- drances caused by character code and encoding differences. MIME should han-dle the necessary conversions automatically and invisibly.
For example, when person A sends E-mail to person B, the following should happen: The E-mail program used by A encodes A's message in some particularmanner, probably according to some convention which is normal on the systemwhere the program is used (such as ISO 8859-1 encoding on a typical modern Unix system). The program automatically includes information about this en-coding into an E-mail header, which is usually invisible both when sending andwhen reading the message. The message, with the headers, is then delivered,through network connections, to B's system. When B uses his E-mail pro-gram (which may be very different from A's) to read the message, the programshould automatically pick up the information about the encoding as specifiedin a header and interpret the message body according to it. For example, if Bis using a Macintosh computer, the program would automatically convert themessage into Mac's internal character encoding and only then display it.
Thus, if the message was ISO 8859-1 encoded and contained the Ä (upper case A with dieresis) character, encoded as octet 196, the E-mail program usedon the Mac should use a conversion table to map this to octet 128, which is theencoding for Ä on Mac. (If the program fails to do such a conversion, strangethings will happen. ASCII characters would be displayed correctly, since they have the same codes in both encodings, but instead of Ä, the character corre-sponding to octet 196 in Mac encoding would appear - a symbol which lookslike f in italics.) Problems with implementations - examples Unfortunately, there are deficiencies and errors in software so that users often have to struggle with character code conversion problems, perhaps cor-recting the actions taken by programs. It takes two to tango, and some moreparticipants to get characters right. This section demonstrates different thingswhich may happen, and do happen, when just one component is faulty, i.e. whenMIME is not used or is inadequately supported by some "partner" (software involved in entering, storing, transferring, and displaying characterdata).
Typical minor (!) problems which may occur in communication in Western European languages other than English is that most characters get interpretedand displayed correctly but some "national letters" don't. For example, char-acter repertoire needed in German, Swedish, and Finnish is essentially ASCII Character Code Issues plus a few letters like "ä" from the rest of ISO Latin 1. If a text in such a lan- guage is processed so that a necessary conversion is not applied, or an incorrectconversion is applied, the result might be that e.g. the word "später" becomes"spter" or "spÌter" or "spdter" or "sp=E4ter", to mention just a few possibil-ities. People familiar with such problems might be able to read the distortedtext too, but others may get seriously confused.
Practical conclusions Whenever text data is sent over a network, the sender and the recipient should have a joint agreement on the character encoding used. In the optimal case, this is handled by the software automatically, but in reality theusers need to take some precautions.
Most importantly, make sure that any Internet-related software that you use to send data specifies the encoding correctly in suitable headers. There are two things involved: the header must be there and it must reflect the actualencoding used; and the encoding used must be one that is widely understoodby the (potential) recipients' software. One must often make compromises asregards to the latter aim: you may need to use an encoding which is not yetwidely supported to get your message through at all.
It is useful to find out how to make your Web browser, newsreader, and E-mail program so that you can display the encoding information for the page,article, or message you are reading. (For example, on Netscape use View PageInfo; on News Xpress, use View Raw Format; on Pine, use h.) If you use, say, Netscape to send E-mail or to post to Usenet news, make sure it sends the message in a reasonable form. In particular, make sure it does not send the message as HTML or duplicate it by sending it both as plain text and as HTML (select plain text only). As regards to characterencoding, make sure it is something widely understood, such as ASCII, some ISO 8859 encoding, or UTF-8, depending on how large character repertoire In particular, avoid sending data in a proprietary encoding (like the Macintosh encoding or a DOS encoding) to a public network. At the very least, if you do that, make sure that the message heading specifies theencoding! There's nothing wrong with using such an encoding within a singlecomputer or in data transfer between similar computers. But when sent toInternet, data should be converted to a more widely known encoding, by thesending program. If you cannot find a way to configure your program to dothat, get another program.
Jukka K. Korpella As regards to other forms of transfer of data in digital form, such as diskettes, information about encoding is important, too. The problem is typi-cally handled by guesswork. Often the crucial thing is to know which programwas used to generate the data, since the text data might be inside a file in,say, the MS Word format which can only be read by (a suitable version of) MSWord or by a program which knows its internal data format. That format, oncerecognized, might contain information which specifies the character encodingused in the text data included; or it might not, in which case one has to askthe sender, or make a guess, or use trial and error - viewing the data usingdifferent encodings until something sensible appears.
Jukka KorpelaThis text is an abridged version of the author's document at http://www.
cs.tut.fi/ jkorpela/chars.html, which contains some additional details aswell as links to further information on the topics discussed.

Source: http://www.eutypon.gr/eutypon/pdf/e2001-07/e07-a03.pdf