AI Glossary / Natural language and speech / Intermediate
Variable-length encoding
Definition
Definition: Variable-length encoding represents different items using code sequences of different lengths. In the source glossary's text-processing context, UTF-8 is the main example: a Unicode scalar value is encoded in one, two, three or four bytes, depending on its value.
What it means
What it means: Counting stored bytes is not the same as counting letters, Unicode code points, displayed symbols or model tokens. A text can contain four code points but occupy ten bytes in UTF-8. This difference matters when software limits field sizes, stores offsets or divides data into chunks.
Variable length describes the representation, not whether a word has many letters. Nor does it mean that a decoder is free to choose arbitrary lengths each time. UTF-8 uses defined rules so a well-formed byte sequence can be decoded unambiguously. Other variable-length codes, including compression codes, use different rules and serve different purposes.
How it works
How it works: In UTF-8, ASCII-range values occupy one byte. Other scalar values use a leading byte and continuation bytes. The leading pattern identifies the sequence length, while the continuation bytes carry additional bits. Not every possible byte combination is valid UTF-8; decoders must recognize malformed input rather than interpret arbitrary bytes as valid characters.
UTF-16 is also variable-width, using one or two 16-bit code units per scalar value. UTF-32 uses one 32-bit code unit per scalar value. None of these facts means that a user-perceived character must consist of one scalar value: combining marks and some emoji sequences make that a separate issue.
Three practical examples
Three practical examples:
1. A customer form allows a fixed number of storage bytes. A name containing non-ASCII text may reach that limit with fewer displayed characters than an ASCII-only name. The interface should explain and enforce the actual limit consistently.
2. An annotation tool records a span using Unicode code-point offsets, while a downstream service expects UTF-8 byte offsets. Without conversion, highlights after multibyte characters can point to the wrong location.
3. A streaming importer receives a UTF-8 character split across network chunks. An incremental decoder retains incomplete trailing bytes until the next chunk arrives, instead of assuming every chunk is an independently complete text string.
Think of it this way
Think of it: A shelf can hold packages of different widths. Counting packages does not tell you their total occupied width, and cutting the shelf into equal-width sections can cut through a package. This resembles byte counting and chunk boundaries, although encoded text follows precise mathematical rules rather than physical packaging constraints.
Worked example
Worked example: Take the four-code-point string Aé中😀, using precomposed é, U+00E9. In UTF-8, A occupies one byte, é two, 中 three and 😀 four. The total is 1 + 2 + 3 + 4 = 10 bytes. The hexadecimal sequence is 41 C3 A9 E4 B8 AD F0 9F 98 80.
The initial three code points occupy six bytes. Truncating to five bytes would retain A and é but only the first two bytes of 中, creating an incomplete UTF-8 sequence. A valid prefix within a five-byte budget therefore stops after é, using three bytes, unless the application adopts another explicit policy.
These byte counts were checked with Python. A visually similar é can instead be represented by e followed by a combining acute accent; that sequence has two code points and three UTF-8 bytes. Encoding length and visual appearance therefore require separate reasoning.
Illustrative case study
Illustrative case study: Fictional Maple Records exports multilingual customer labels to a legacy field with a byte limit. Its first implementation trims the encoded data at a fixed byte position. Some labels emerge with replacement characters because a multibyte sequence was split.
The team defines the storage encoding, validates decoding and chooses complete text boundaries that fit the byte budget. It also tests combining sequences so a valid code-point boundary does not leave an awkward displayed fragment. The original label is retained in the authoritative record. This fictional case illustrates text integrity, not a recommendation to shorten people's names or a measured production outcome.
Common misconceptions and mistakes
Common misconceptions and mistakes: UTF-8 does not mean eight bits for every character. A four-byte UTF-8 sequence is not four separate characters. Variable-length encoding is not automatically compression optimized for the frequency of symbols in a particular dataset. A byte limit is not a model-token limit. Converting between Unicode encodings does not translate a language or repair text already decoded incorrectly. Finally, cutting only at code-point boundaries avoids broken encodings but can still split a multi-code-point grapheme cluster.
Business use
Business use: Agree on encoding and length units at every integration boundary. Document whether positions and limits use bytes, code units, code points, grapheme clusters or tokens. Test multilingual names, accented forms and emoji rather than relying only on English examples. Preserve originals when applying truncation or normalization, and make any irreversible transformation visible in the workflow. These practices help distinguish actual source errors from corruption introduced by import, export or storage assumptions.
People also ask
People also ask: Is UTF-8 fixed-width because each byte has eight bits?
No. Its code unit is eight bits, but a scalar value can require several code units.
Will switching to UTF-32 make every displayed character one unit?
No. It gives one unit per scalar value, while a displayed grapheme can contain multiple scalar values.
Does variable length make decoding ambiguous?
Not when a valid encoding's rules provide an unambiguous representation. UTF-8 defines the required byte patterns; malformed input must be handled explicitly.
Check your understanding
Check your understanding: A UTF-8 string contains one ASCII letter and one code point that requires four bytes. Does it necessarily fit a four-byte field?
Answer: No. Those two code points occupy five bytes in total, excluding any additional framing. A character count of two does not establish that the byte limit is satisfied.
Related terms
- UTF-8
- Unicode
- Code point
- Normalization
- code unit
- grapheme cluster
- byte offset
- character encoding
Learn next
measure both code-point length and UTF-8 byte length for a multilingual test string. Inspect an accented letter in composed and decomposed forms. Then test a streaming decoder with a chunk boundary inside a multibyte sequence and explain why the boundary must not be treated as a text error by itself.
More in Natural language and speech
- Unlikelihood training - Unlikelihood training adds an objective that lowers the probability of selected undesirable outputs.
- Unvoiced sound - An unvoiced sound, also called a voiceless sound, is produced without the regular vocal-fold vibration that supplies voicing.
- Utterance - An utterance is a particular instance or segment of language produced in communication, commonly a stretch of speech by a speaker in context.
- Vauquois triangle - Definition: The Vauquois triangle is a conceptual diagram of machine-translation approaches.
- Velar sound - Definition: A velar sound is a speech sound whose primary constriction involves the back of the tongue and the velum, or soft palate.
- Verb - Definition: A verb is a grammatical class of word that commonly expresses an action, event, process or state and helps organize a clause.
Sources
Sources: Unicode Consortium, UTF-8, UTF-16, UTF-32 and BOM FAQ: https://www.unicode.org/faq/utf_bom.html . RFC 3629, UTF-8 definition and valid byte sequences: https://www.rfc-editor.org/rfc/rfc3629 . Jurafsky and Martin, Speech and Language Processing, Chapter 2, Unicode and text representation: https://web.stanford.edu/~jurafsky/slp3/2.pdf . The byte calculations were executed with Python; Maple Records is fictional. This entry emphasises character encoding rather than frequency-based compression coding.
Previous: Variable kernel density estimation Next: Variable neighborhood search