UTF-8 (from English Unicode Transformation Format, 8-bit - “ Unicode Transformation Format, 8- bit”) is one of the generally accepted and standardized text encodings that allows you to store Unicode characters using a variable number of bytes (from 1 to 6).

The UTF-8 standard is officially enshrined in RFC 3629 and ISO / IEC 10646 Annex D. The coding has found wide application in UNIX-like operating systems and web space ^[1] . The very same UTF-8 format was invented on September 2, 1992 by Ken Thompson and Rob Pike and implemented in Plan 9. ^[2] For BOM, it uses the byte sequence EF ₁₆ , BB ₁₆ , BF ₁₆ (which is itself a three-byte implementation of the FEFF ₁₆ character ).

One of the advantages is compatibility with ASCII - any of their 7-bit characters are displayed as they are, and the rest give the user garbage (noise). Therefore, if the Latin letters and the simplest punctuation marks (including the space) occupy a significant amount of text, UTF-8 gives a gain in volume compared to UTF-16. ^{[3] [4]}

Coding principle

For numbers from U + 0000 to U + 007F, the UTF-8 encoding fully corresponds to the 7-bit US-ASCII c 0 in the high bit and occupies one byte.

The coding algorithm in UTF-8 is standardized in RFC 3629 and consists of 3 points:

1. Determine the number of octets (bytes) required for the encoded character number in accordance with the table:

2. Prepare the high-order bits of the first octet (0xxxxxx for one octet, 110xxxxx - two, 1110xxxxx - three, etc.). For the remaining octets, the two most significant bits are 10 (10xxxxxx).

3. Fill the remaining bits (in clause 2 are marked with x) in octets by the number of the Unicode character expressed in binary. Start with the low-order bits of the character number, putting them in the low-order bits of the last octet of the code. And so on, until all the bits of the character number are transferred to the free bits of the octets.

Example

BOM code for UTF-8 = EF BB BF (16) = 1110 1111 1011 1011 1011 1111 (2)

The table below presents the values ​​in hexadecimal notation. In practice, for each value, the only correct representation is selected according to the algorithm standardized in RFC 3629 (with a minimum length of bytes, large ones are not allowed; and presented for clarity and tests by encoders).

UTF-8 and encoding / decoding errors

The examples below are for quick orientation in cases of incorrect decoding of the text (the so-called krakozyabry ^[en] ).

This is how the phrase “A person will now see only what he expects to see.” If it is perceived by a decoder in Windows-1251 encoding, and not UTF-8:

ЧеР"овек СЃРµР№С ‡ Р ° СЃ СѓРІРёРґРЁС‚ Р "РОС € СЊ то, С З СРР РРРРРРРРРРС Ñ СРРРРРРРС With.

The phrase “A person will now see only what he expects to see.” When double encoding UTF-8 to UTF-8:

ЧеР"Р С • Р Р † РµРС" СЃРµРв "-РЎвЂРР В ° РЎРѓ РЎС" Р Р † Р С'Р Т'Р С'С‚ Р В "Р С'С€ РЎРЉ РЎвЂљР С •, РЎвЂЎРЎвЂљР С • Р С • Р¶РС'Р Т'Р В ° Р ВµРЎвЂ РЎС "Р Р † Р С'Р Т'РµСвЂРРРР.

Self-sync and UTF-16

Self-synchronization in UTF-8 can be considered when random bytes are fed to your program and you need to determine the beginning of the first character. The primary sign is a flush high bit of a byte - this is an ASCII character. If it is set, then skip those bytes that have the bit cleared before the most significant one. In other cases, you can continue character-by-character decoding.

UTF-8 has the property of self-synchronization when processing 8-bit bytes. An alternative to UTF-8 is UTF-16 encoding, which is already processed in 16-bit words. There may be a doubt that UTF-16 is not self-synchronizing. At the moment, the overwhelming majority of data is transmitted in integral octets - 8 bits or nothing (see IPv4, IPv6, SATA for modern hardware and ATA with PATA for the recent one). Under these conditions, UTF-8 has the advantage of characterizing self-synchronization over UTF-16 when it comes to hardware data transfer or byte stream operation (reading Unicode data from an arbitrary position). If the work is carried out in the RAM of one machine, then UTF-16 is also self-synchronizing (if the equipment is capable of delivering whole 16-bit words).

Windows-1251