Table of Contents
List of Figures xxiii
List of Tables xxvii
Foreword by Mark Davis xxxi
Preface xxxiii
Acknowledgments xxxix
Chapter 1 Introduction 1
1.1 Coverage 2
1.2 Design Goals 4
1.3 Text Handling 5
Chapter 2 General Structure 9
2.1 Architectural Context 9
2.2 Unicode Design Principles 13
2.3 Compatibility Characters 23
2.4 Code Points and Characters 25
2.5 Encoding Forms 28
2.6 Encoding Schemes 35
2.7 Unicode Strings 37
2.8 Unicode Allocation 38
2.9 Details of Allocation 41
2.10 Writing Direction 46
2.11 Combining Characters 48
2.12 Equivalent Sequences and Normalization 54
2.13 Special Characters and Noncharacters 57
2.14 Conforming to the Unicode Standard 59
Chapter 3 Conformance 65
3.1 Versions of the Unicode Standard 65
3.2 Conformance Requirements 70
3.3 Semantics 76
3.4 Characters and Encoding 78
3.5 Properties 81
3.6 Combination 91
3.7 Decomposition 95
3.8 Surrogates 97
3.9 Unicode Encoding Forms 98
3.10 Unicode Encoding Schemes 105
3.11 Canonical Ordering Behavior 109
3.12 Conjoining Jamo Behavior 117
3.13 Default Case Algorithms 123
Chapter 4 Character Properties 129
4.1 Unicode Character Database 130
4.2 Case--Normative 132
4.3 Combining Classes--Normative 133
4.4 Directionality--Normative 138
4.5 General Category--Normative 138
4.6 Numeric Value--Normative 139
4.7 Bidi Mirrored--Normative 141
4.8 Name--Normative 142
4.9Unicode 1.0 Names 144
4.10 Letters, Alphabetic, and Ideographic 144
4.11 Properties Related to Text Boundaries 145
4.12 Characters with Unusual Properties 145
Chapter 5 Implementation Guidelines 151
5.1 Transcoding to Other Standards 151
5.2 Programming Languages and Data Types 153
5.3 Unknown and Missing Characters 155
5.4 Handling Surrogate Pairs in UTF-16 157
5.5 Handling Numbers 158
5.6 Normalization 160
5.7 Compression 161
5.8 Newline Guidelines 161
5.9 Regular Expressions 166
5.10 Language Information in Plain Text 166
5.11 Editing and Selection 167
5.12 Strategies for Handling Nonspacing Marks 169
5.13 Rendering Nonspacing Marks 172
5.14 Locating Text Element Boundaries 178
5.15 Identifiers 179
5.16 Sorting and Searching 179
5.17 Binary Order 181
5.18 Case Mappings 184
5.19 Unicode Security 190
5.20 Default Ignorable Code Points 192
Chapter 6 Writing Systems and Punctuation 197
6.1 Writing Systems 198
6.2 General Punctuation 202
Chapter 7 European Alphabetic Scripts 225
7.1 Latin 226
7.2 Greek 237
7.3 Coptic 243
7.4 Cyrillic 245
7.5 Glagolitic 246
7.6 Armenian 247
7.7 Georgian 249
7.8 Modifier Letters 250
7.9 Combining Marks 252
Chapter 8 Middle Eastern Scripts 263
8.1 Hebrew 264
8.2 Arabic 269
8.3 Syriac 283
8.4 Thaana 291
Chapter 9 South Asian Scripts-I 295
9.1 Devanagari 296
9.2 Bengali 312
9.3 Gurmukhi 317
9.4 Gujarati 321
9.5 Oriya 322
9.6 Tamil 324
9.7 Telugu 330
9.8 Kannada 331
9.9 Malayalam 334
Chapter 10 South Asian Scripts-II 341
10.1 Sinhala 341
10.2 Tibetan 343
10.3 Phags-pa 353
10.4 Limbu 360
10.5 Syloti Nagri 363
10.6 Kharoshthi 364
Chapter 11 Southeast Asian Scripts 373
11.1 Thai 373
11.2 Lao 376
11.3 Myanmar 379
11.4 Khmer 382
11.5 Tai Le 393
11.6 New Tai Lue 394
11.7 Philippine Scripts 395
11.8 Buginese 397
11.9 Balinese 399
Chapter 12 East Asian Scripts 407
12.1 Han 408
12.2 Ideographic Description Characters 427
12.3 Bopomofo 431
12.4 Hiragana and Katakana 433
12.5 Halfwidth and Fullwidth Forms 434
12.6 Hangul 435
12.7 Yi 438
Chapter 13 Additional Modern Scripts 445
13.1 Ethiopic 445
13.2 Mongolian 448
13.3 Osmanya 457
13.4 Tifinagh 457
13.5 N'Ko 458
13.6 Cherokee 463
13.7 Canadian Aboriginal Syllabics 464
13.8 Deseret 465
13.9 Shavian 467
Chapter 14 Archaic Scripts 471
14.1 Ogham 472
14.2 Old Italic 473
14.3 Runic 475
14.4 Gothic 477
14.5 Linear B 478
14.6 Cypriot Syllabary 479
14.7 Phoenician 480
14.8 Ugaritic 482
14.9 Old Persian 483
14.10 Sumero-Akkadian 483
Chapter 15 Symbols 489
15.1 Currency Symbols 490
15.2 Letterlike Symbols 492
15.3 Number Forms 498
15.4 Mathematical Symbols 502
15.5 Invisible Mathematical Operators 507
15.6 Technical Symbols 508
15.7 Geometrical Symbols 512
15.8 Miscellaneous Symbols and Dingbats 514
15.9 Enclosed and Square 517
15.10 Braille 519
15.11 Western Musical Symbols 520
15.12 Byzantine Musical Symbols 525
15.13 Ancient Greek Musical Notation 526
Chapter 16 Special Areas and Format Characters 531
16.1 Control Codes 532
16.2 Layout Controls 534
16.3 Deprecated Format Characters 543
16.4 Variation Selectors 545
16.5 Private-Use Characters 546
16.6 Surrogates Area 548
16.7 Noncharacters 549
16.8 Specials 550
16.9 Tag Characters 554
Chapter 17 Code Charts 563
17.1 Character Names List 563
17.2 CJK Unified Ideographs 569
17.3 Hangul Syllables 570
Chapter 18 Han Radical-Stroke Index 1023
Appendix A Notational Conventions 1077
Appendix B Unicode Publications and Resources 1083
B.1 The Unicode Consortium 1083
B.2 Unicode Publications 1084
B.3 Unicode Technical Standards 1085
B.4 Unicode Technical Reports 1086
B.5 Unicode Technical Notes 1087
B.6 Other Unicode Online Resources 1088
Appendix C Relationship to ISO/IEC 10646 1091
C.1 History 1091
C.2 Encoding Forms in ISO/IEC 10646 1095
C.3 UCS Transformation Formats 1096
C.4 Synchronization of the Standards 1097
C.5 Identification of Features for the Unicode Standard 1097
C.6 Character Names 1098
C.7 Character Functional Specifications 1098
Appendix D Changes from Previous Versions 1099
D.1 Improvements to the Standard 1099
D.2 Versions of the Unicode Standard 1100
D.3 Clause and Definition Numbering Changes 1102
D.4 Changes from Version 4.1 to Version 5.0 1104
D.5 Changes from Version 4.0 to Version 4.1 1106
D.6 Changes from Unicode Version 3.2 to Version 4.0 1109
D.7 Changes from Unicode Version 3.1 to Version 3.2 1111
D.8 Changes from Unicode Version 3.0 to Version 3.1 1113
Appendix E Han Unification History 1115
E.1 Development of the URO 1115
E.2 Ideographic Rapporteur Group 1116
Appendix F Unicode Encoding Stability Policies 1119
F.1 Encoding Stability Policies for the Unicode Standard 1119
Glossary 1125
References 1153
R.1 Source Standards and Specifications 1153
R.2 Source Dictionaries for Han Unification 1161
R.3 Other Sources for the Unicode Standard 1161
R.4 Selected Resources: Technical 1171
R.5 Selected Resources: Scripts and Languages 1173
Indices 1179
I.1 Unicode Names Index 1179
I.2 General Index 1231
Annexes 1251
UAX 9: The Bidirectional Algorithm 1251
UAX 11: East Asian Width 1275
UAX 14: Line Breaking Properties 1283
UAX 15: Unicode Normalization Forms 1333
UAX 24: Script Names 1365
UAX 29: Text Boundaries 1373
UAX 31: Identifier and Pattern Syntax 1393
UAX 34: Unicode Named Character Sequences 1405
UAX 41: Common References for Unicode Standard
Forewords & Introductions
This book, The Unicode Standard, Version 5.0, together with the Unicode Character Database, is the authoritative source of information on Version 5.0 of the Unicode character encoding standard.
Version 5.0 of the standard is a significant departure from prior versions. It lays out much clearer requirements for supporting Unicode and provides more explicit guidance for implementers to quickly embrace the proliferation of new growth technologies and emerging markets while at the same time meeting users' needs for secure, robust software.
Why Buy This Book
In a major enhancement, Version 5.0 of the Unicode Standard is now available in a smaller, more convenient size while including much more textual content. Most notably, for the first time the book includes all of the Unicode Standard Annexes, which provide specifications for vital processes such as text normalization, bidirectional handling, and identifier parsing.Version 5.0 contains the knowledge gained from many years of worldwide implementation experience and has been enhanced significantly: the text incorporates 15 years of user feedback, provides thorough answers to the many questions users of Unicode have raised, and is much more accessible--with greatly improved figures and tables, and with the text revised for clarity.
- Four-fifths of the figures are new.
- Two-thirds of the definitions are new.
- One-half of the Unicode Standard Annexes are new.
- One-third of the conformance clauses are new.
- One-fourth of the tables are new.
In addition, the text of Version 5.0 reflects advances in the computer implementation of writing systems. Itsubstantially improves the descriptions of rendering Indic scripts to meet the demands of this area of growing market importance--Unicode-based implementations are supported by the government of India, and this book explains how to build them. Version 5.0 also highlights the newly established core CJK subset of characters, IICore, which is critical for rendering and interoperability in the East Asian market.
In short, The Unicode Standard, Version 5.0, enables developers to implement quickly the latest advances for worldwide software users while opening new opportunities in high-growth markets. The changes from Versions 3.0 and 4.0 to Version 5.0 are major and important--this is the one book all Unicode implementers must have.
Why Upgrade to Version 5.0
Version 5.0 of the Unicode Standard brings significant improvements beyond Versions 3.0 and 4.0. The industry has noticed and is quickly moving to Version 5.0--Windows Vista runs on 5.0; ICU, Google, and Yahoo! all have plans to upgrade to 5.0. Internet and W3C protocols are built on Unicode and are continually adapting to the latest versions. The International Standard ISO/IEC 10646 is also synchronized with Version 5.0.
This latest version of the Unicode Standard is the basis for Unicode security mechanisms, the Unicode collation algorithm, the locale data provided by the Common Locale Data Repository, and support for Unicode in regular expressions. Improved expression of the Unicode encoding model makes it much clearer how implementers need to support the representation of Unicode text in UTF-8 and other encoding forms. Character properties have been systematized and greatly extended to help implementers in support of Unicode text processing. The standard has also established principles of stability for casefolding and identifiers, crucial for interoperability and backward compatibility for formal language use and in other contexts that depend on exact usage and matching of identifiers.
Version 5.0 delivers a stable, practical character processing model in sync with today's information technology needs. Unicode now offers:
- Round-trip compatibility with the Chinese standards GB18030 and HKSCS
- The specification of the newly established core CJK subset of characters, IICore
- Refinements to casing and bidirectional behavior to meet industry requirements
- Improved Indic rendering guidelines
- Better guidance on the handling of combining characters, Unicode strings, variation selectors, line breaking, and segmentation
Implementers who want to keep pace with the industry and take advantage of a stable foundation for security, to align with the latest collation and locale data definitions, and, most importantly, to expand their market reach need to upgrade to Version 5.0 as soon as possible.
Detailed Change Information
See Appendix D, Changes from Previous Versions, for detailed information about the changes from previous versions of the standard, including character counts, stability guarantees, and updates to the Unicode Character Database and Unicode Standard Annexes.
Version 5.0 of the Unicode Standard corresponds to ISO/IEC 10646:2003 plus Amendments 1 and 2 to that standard and four characters to support Sindhi from Amendment 3.
Organization of This Book
This book and the Unicode Character Database define Version 5.0 of the Unicode Standard. The book gives the general principles, requirements for conformance, guidelines for implementers, character code charts and names, and the Unicode Standard Annexes.
Concepts, Architecture, Conformance, and Guidelines
The first five chapters of Version 5.0 introduce the Unicode Standard and provide the fundamental information needed to produce a conforming implementation. Basic text processing, working with combining marks, and encoding forms are all described. A special chapter on implementation guidelines answers many common questions that arise when implementing Unicode.
Chapter 1 introduces the standard's basic concepts, design basis, and coverage and discusses basic text handling requirements.Chapter 2 sets forth the fundamental principles underlying the Unicode Standard and covers specific topics such as text processes, overall character properties, and the use of combining marks.
Chapter 3 constitutes the formal statement of conformance. This chapter also presents the normative algorithms for three processes: the canonical ordering of combining marks, the encoding of Korean Hangul syllables by conjoining jamo, and default casing.
Chapter 4 describes character properties in detail, both normative (required) and informative. Tables giving additional character property information appear in the Unicode Character Database.
Chapter 5 discusses implementation issues, including compression, strategies for dealing with unknown and unsupported characters, and transcoding to other standards.
Character Block Descriptions
Chapters 6 through 16 contain the character block descriptions that give basic information about each script or group of symbols and may discuss specific characters or pertinent layout information. Some of this information is required to produce conformant implementations of these scripts and other collections of characters.
Code Charts
Chapter 17 gives the code charts and the Character Names List. The code charts contain the normative character encoding assignments, and the names list contains normative information as well as useful cross references and informational notes.
Han Radical-Stroke Index
Chapter 18 provides a Han radical-stroke index for the IICore subset of CJK ideographs. This index aids in locating specific, common ideographs encoded in the Unicode Standard.
Appendices
The appendices contain detailed background information on important topics regarding the history of the Unicode Standard and its relationship to ISO/IEC 10646.
Appendix A documents the notational conventions used by the standard.
Appendix B provides abstracts of Unicode Technical Reports and lists other important Unicode resources.
Appendix C details the relationship between the Unicode Standard and ISO/IEC 10646.
Appendix D lists the changes to the Unicode Standard since Version 4.0.
Appendix E describes the history of Han unification in the Unicode Standard.
Appendix F reproduces the text of the policies of the Unicode Consortium regarding character encoding stability.
Glossary, References, and Indices
The appendices are followed by a glossary of terms, a bibliography, and two indices: an index to Unicode characters and an index to the text of the book.
Unicode Standard Annexes
The Unicode Standard Annexes are printed in the back of this book, following the indices. These annexes form an integral part of the Unicode Standard. Conformance to a version of the Unicode Standard includes conformance to its Unicode Standard Annexes.
Unicode Standard Annex #9, "The Bidirectional Algorithm," describes specifications for the positioning of characters in mixed-directional text, such as Arabic or Hebrew.
Unicode Standard Annex #11, "East Asian Width," presents the specification of an informative property for Unicode characters that is useful when interoperating with East Asian legacy character sets.
Unicode Standard Annex #14, "Line Breaking Properties," presents the specification of line breaking properties for Unicode characters.
Unicode Standard Annex #15, "Unicode Normalization Forms," describes specifications for four normalized forms of Unicode text.
Unicode Standard Annex #24, "Script Names," specifies an assignment of script names to all Unicode code points.
Unicode Standard Annex #29, "Text Boundaries," describes guidelines for determining default boundaries between certain significant text elements: grapheme clusters ("user-perceived characters"), words, and sentences.
Unicode Standard Annex #31, "Identifier and Pattern Syntax," describes specifications for recommended defaults for the use of Unicode in the definitions of identifiers and in pattern-based syntax.
Unicode Standard Annex #34, "Unicode Named Character Sequences," defines the concept of a Unicode named character sequence and a set of rules constraining possible names applied to character sequences.
Unicode Standard Annex #41, "Common References for Unicode Standard Annexes," contains the listing of references shared by other Unicode Standard Annexes.The 5.0.0 version of each UAX is included on the CD-ROM. All versions, including the most up-to-date versions of all Unicode Standard Annexes, are available on the Unicode Web site:
Any currently listed errata will be fixed in subsequent versions of the standard.
loading...