This book, The Unicode Standard, Version 4.0
, is the authoritative source of information on the Unicode character encoding standard.Version 4.0 expands on and supersedes all other previous versions. The text of the standard has been extensively rewritten to improve its structure and clarity.Major additions to Version 4.0 since Version 3.0 include:
- extensive additions of CJK characters to cover dictionaries and historic usage
- many new symbols for mathematical and technical publication
- substantially improved specification of conformance requirements, incorporating the character encoding model
- encoding of supplementary characters
- formalized policies for stability of the standard
- clarification of semantics of special characters, including the byte order mark
- major expansion of Unicode Character Database properties and of specifications for text boundaries and casing
- more minority scripts, including Limbu, Tai Le, Osmanya, and Philippine scripts
- more historic scripts, including Linear B, Cypriot, and Ugaritic
- tightened definition of encoding terms, including UTF-32
Furthermore, many individual characters were added to meet the requirements of users and implementers alike. The Unicode Standard maintains consistency with the international standard ISO/IEC 10646. Version 4.0 of the Unicode Standard corresponds to ISO/IEC 10646:2003.0.1 About the Unicode StandardThis book, together with the Unicode Standard Annexes described in Appendix B, and the Unicode Character Database, defines Version 4.0 of the Unicode Standard. The book gives the general principles, requirements for conformance, and guidelines for implementers, followed by character code charts and names.Concepts, Architecture, Conformance, and GuidelinesThe first five chapters of Version 4.0 introduce the Unicode Standard and provide the fundamental information needed to produce a conforming implementation. Basic text processing, working with combining marks, and encoding forms are all described. A special chapter on implementation guidelines answers many common questions that arise when implementing Unicode.Chapter 1 introduces the standard's basic concepts, design basis, and coverage, and discusses basic text handling requirements.Chapter 2 sets forth the fundamental principles underlying the Unicode Standard and covers specific topics such as text processes, overall character properties, and the use of combining marks.Chapter 3 constitutes the formal statement of conformance. This chapter also presents the normative algorithms for two processes: the canonical ordering of combining marks and the encoding of Korean Hangul syllables by conjoining jamo.Chapter 4 describes character properties in detail, both normative (required) and informative. Tables giving additional character property information appear in the Unicode Character Database.Chapter 5 discusses implementation issues, including compression, strategies for dealing with unknown and unsupported characters, and transcoding tother standards.Character Block DescriptionsChapters 6 through 15 contain the character block descriptions that give basic information about each script or group of symbols and may discuss specific characters or pertinent layout information. Some of this information is required in order to produce conformant implementations of these scripts and other collections of characters.Chapter 6 introduces writing systems and describes the general punctuation characters.Chapter 7 presents the European Alphabetic scripts, including Latin, Greek, Cyrillic, Armenian, Georgian, and associated combining marks.Chapter 8 presents the Middle Eastern, right-to-left scripts: Hebrew, Arabic, Syriac, and Thaana.Chapter 9 covers the South Asian scripts, including Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala, Tibetan, and Limbu. Chapter 10 covers the Southeast Asian scripts, including Thai, Lao, Tai Le, Myanmar, Khmer, and Philippine scripts.Chapter 11 presents the East Asian scripts, including Han, Hiragana, Katakana, Hangul, Bopomofo, and Yi.Chapter 12 presents other scripts, including Ethiopic, Mongolian, Osmanya, Cherokee, Canadian Aboriginal Syllabics, Deseret, and Shavian.Chapter 13 describes archaic scripts, including Ogham, Old Italic, Runic, Gothic, Ugaritic, Linear B, and Cypriot.Chapter 14 presents symbols, including currency, letterlike and technical symbols, mathematical operators, and musical symbols.Chapter 15 describes other topics such as private-use characters, surrogate code points, and special characters.Charts and Han Radical-Stroke IndexThe next two chapters document the Unicode Standard's character code assignments, their names and important descriptive information, and provide a Han radical-stroke index that aids in locating specific ideographs encoded in Unicode.Chapter 16 gives the code charts and the Character Names List. The code charts contain the normative character encoding assignments, and the names list contains normative information as well as useful cross references and informational notes.Chapter 17 provides a radical-stroke index to East Asian ideographs.AppendicesThe appendices contain detailed background information on important topics regarding the history of the Unicode Standard and its relationship to ISO/IEC 10646.Appendix A describes the history of Han Unification in the Unicode Standard.Appendix B provides abstracts of Unicode Technical Reports and lists other important Unicode resources.Appendix C details the relationship between the Unicode Standard and ISO/IEC 10646.Appendix D lists the changes to the Unicode Standard since Version 3.0.The appendices are followed by a glossary of terms, a bibliography, and two indices: an index to Unicode characters and an index to the text of the book.0.2 The Unicode Character Database and Technical ReportsThe Unicode Character Database is a collection of data files that contain character code points, character names and character property data.See Appendix B for a summary overview of important Unicode Technical Standards, Unicode Technical Reports and Unicode Standard Annexes.On the CD-ROMThe CD-ROM also contains additional information, such as sample code, which is maintained on the Unicode ftp site at: ftp.unicode.For the complete contents of the CD-ROM see its ReadMe.txt file.0.3 Notational ConventionsThroughout this book, certain typographic conventions are used.Code PointsIn running text, an individual Unicode code point can be expressed as U+n, where n is from four to six hexadecimal digits, using the digits 0-9 and uppercase letters A-F (for 10 through 15, respectively). There should be no leading zeros, unless the code point would have fewer than four hexadecimal digits; for example, U+0001, U+0012, U+0123, U+1234, U+12345, U+102345.
- U+0416 is the Unicode code point for the character named .
In tables, the U+ may be omitted for brevity.A range of Unicode code points is expressed as U+xxxx-U+yyyy or xxxx..yyyy, where xxxx and yyyy are the first and last Unicode values in the range, and the long dash or two dots indicate a contiguous range inclusive of the endpoints. For ranges involving supplementary characters, the code points in the ranges are expressed with five or six hexadecimal digits.
- The range U+0900-U+097F contains 128 Unicode code points.
- The Plane 16 private use characters are in the range 100000..10FFFD.
Character NamesAll Unicode characters have unique names, which are identical to those of the English language edition of International Standard ISO/IEC 10646. Unicode character names contain only uppercase Latin letters A through Z, digits, space, and hyphen-minus; this convention makes it easy to generate computer-language identifiers automatically from the names. Unified CJK ideographs are named -X, where X is replaced with the hexadecimal Unicode code point--for example, -4E00.The names of Hangul syllables are generated algorithmically; for details, see Hangul Syllable Names in Section 3.12, Conjoining Jamo Behavior.In running text, a formal Unicode name is shown in small capitals (for example, ), and alternative names (aliases) appear in italics (for example, umlaut).Italics are also used to refer to a text element that is not explicitly encoded (for example, pasekh alef) or to set off a non-English word (for example, the Welsh word ynghyd).SequencesA sequence of two or more code points may be represented by a comma-delimited list, set off by angle brackets. For this purpose angle brackets consist of U+003C - and U+003E - . Spaces are optional after the comma, and U+ notation for the code point is also optional; for example, "". When the usage is clear from the context, a sequence of characters may also be represented with generic short names, for example as in "", or the angle brackets may be omitted.In contrast to sequences of code points, a sequence of one or more code units may be represented by a list set off by angle brackets, but without comma delimitation or U+ notation. For example, the notation "" represents a sequence of bytes, as for the UTF-8 encoding form of a Unicode character. The notation "" represents a sequence of 16-bit code units, as for the UTF-16 encoding form of a Unicode character.MiscellaneousPhonemic transcriptions are shown between slashes, as in Khmer /khnyom/. Phonetic transcriptions are shown between square brackets, using the International Phonetic Alphabet.In cases of ambiguity, is used. Dates before the common era are labeled with .The term byte, as used in this standard, always refers to a unit of eight bits. This corresponds to the use of the term octet in some other standards.Extended BNFThe Unicode Standard and technical reports use an extended BNF format for describing syntax. As different conventions are used for BNF, Table 0-1, Extended BNF, lists the notation used here.0.3 Notational Conventions In other environments, such as programming languages or mark-up, alternative notation for sequences of code points or code units may be used.Character Classes. A code point class is a specification of an unordered set of code points. Whenever the code points are all assigned characters, it can also be referred to as a character class. The specification consists of any of the following:
- A literal code point
- A range of literal code points
- A set of code points having a given Unicode character property value, as defined in the Unicode Character Database (see PropertyAliases.txt and PropertyValueAliases.txt)
- Non-boolean properties given as an expression = or A, , such as "General_Category=Titlecase_Letter"
- Boolean properties given as an expression = true or
- A, true, such as "Uppercase=true"
- Combinations of logical operations on classes
Further extensions to this specification of character classes are used in some Unicode Standard Annexes and Unicode Technical Reports. Such extensions are described in those documents, as appropriate.A partial formal BNF syntax for character classes as used in this standard is given by the following.char_class := "" char_class - char_class ""// set difference
:= "" item_list ""
:= "" property ("=" "A,") property_value ""
item_list := item (","? item)?
item := code_point // either literal or escaped
:= code_point - code_point // inclusive range
Whenever any character could be interpreted as a syntax character, it must be escaped. Where no ambiguity would result (with normal operator precedence), extra square brackets can be discarded. If a space character is used as a literal, it is escaped. Examples are found in Table 0-2, Character Class Examples.Symbols MeaningFor more information about character classes, see Unicode Technical Report #18, "Unicode Regular Expression Guidelines."OperatorsOperators used in this standard are listed in Table 0-3, Operators.0.4 ResourcesThe Unicode Consortium provides a number of online resources for obtaining information and data about the Unicode Standard, as well as updates and corrigenda. They are listed below.Box 391476
Mountain View, CA 94039-1476
Please check the Web site for up-to-date contact information, including telephone, fax, and courier delivery address.