AI Glossary / Natural language and speech / Beginner

Code point

Definition

A code point is a numerical position in a coded character set. In Unicode, code points run from U+0000 to U+10FFFF. They provide abstract identifiers used in text representation, but not every code point is assigned to a character.

What it means

The notation U+0061 identifies the lowercase Latin letter a. The number is usually written in hexadecimal. It is not a drawing of the letter: different fonts can render the same character using different glyphs.

A code point also differs from a byte. An encoding such as UTF-8 specifies how Unicode scalar values are represented as bytes. One encoded value may require several bytes, so counting bytes does not generally count characters.

Even counting code points does not necessarily count the units a reader perceives as characters. A base letter and combining accent can form one visible unit using multiple code points. Some emoji sequences also contain several. AI text processing needs these distinctions when measuring length, aligning annotations and preparing multilingual input.

How it works

A text system decodes stored bytes according to an encoding, obtaining a sequence of encoded character values. It can then inspect code points, apply documented normalization rules, segment text and pass appropriate units to a tokenizer.

Unicode distinguishes code points from scalar values: the surrogate range U+D800 through U+DFFF is excluded from scalar values and is reserved for UTF-16 encoding mechanics. This is one reason “every code point is a directly encodable character” is inaccurate.

Normalization can give canonically equivalent strings a consistent representation. It is not the same as removing accents or translating text. Compatibility normalization makes additional changes that may erase distinctions, so the selected form should match the application.

Language-model tokens are another layer. A token can represent part of a word, multiple characters or bytes according to the tokenizer. A code-point count alone does not establish the model's token count.

Three practical examples

• A multilingual search system normalizes equivalent accented spellings before comparing keys.
• A text-annotation tool specifies whether offsets count bytes, code points or another unit so that highlighted spans remain aligned.
• A data-cleaning review inspects unexpected code points when visually similar strings fail an equality comparison.

Think of it this way

Think of a catalog number identifying a character, while a font supplies its visual artwork and an encoding determines how its identifier travels in storage.

The analogy separates identity, appearance and representation. A reader-perceived character can involve several cataloged components, and some Unicode positions serve technical purposes rather than representing a visible symbol.

Worked example

The same displayed word, different sequences

The word café can be represented using the sequence U+0063, U+0061, U+0066, U+00E9. That contains four code points. In UTF-8, its bytes in hexadecimal are 63 61 66 C3 A9, a total of five bytes.

An equivalent spelling uses U+0063, U+0061, U+0066, U+0065 and U+0301. The final two code points are the letter e and a combining acute accent. This sequence has five code points and six UTF-8 bytes: 63 61 66 65 CC 81.

With appropriate rendering, both represent the same four reader-perceived letters. A raw sequence comparison nevertheless distinguishes them. Applying Unicode NFC to both produces the same canonical representation for this example.

The operation preserves the accent; it does not turn café into cafe. A model tokenizer may still divide the normalized word according to its own vocabulary. Therefore the counts four, five and six above refer to specific representations, not a universal “text length.”

Illustrative case study

A fictional document-search team finds that some names copied from different systems fail to match despite looking identical. Its first proposal removes all non-ASCII characters, which would damage many names rather than address the actual representation issue.

The team inspects the code-point sequences and identifies canonically equivalent forms. It applies a documented normalization step for matching, preserves original display text and verifies behavior with multilingual examples.

It also updates annotation offsets consistently, since changing a representation can change code-point positions. The lesson is to diagnose the representation layer before altering linguistic content. Visual similarity alone is not a reliable description of the stored sequence.

Common misconceptions and mistakes

• One code point always equals one visible character. Combining sequences and other constructions can use several.
• A code point is a byte value. Encodings map values to one or more code units or bytes.
• Unicode assigns a visible character to every position. Some positions are unassigned or have technical roles.
• Normalization removes accents. Canonical normalization preserves the represented text's canonical distinctions.
• Code-point count equals token count. Tokenization follows a separate model-specific scheme.

Business use

Code-point literacy helps teams avoid broken search, inconsistent text limits and misaligned annotations. Document the encoding at ingestion and the unit used for every stored offset or length constraint. Preserve original text when transformations serve matching or analysis rather than display.

For model preparation, align normalization and tokenization between training and inference. Test examples with combining marks and scripts beyond basic Latin. A pipeline that works for short ASCII strings may still mishandle valid multilingual text without any obvious file-format error.

People also ask

Does changing the font change the code point?

Normally no. It changes the glyph used to display the encoded character.

Can two different code points look similar?

Yes. Visual resemblance does not establish character identity or canonical equivalence.

Is UTF-8 itself a list of characters?

No. It is an encoding form that represents Unicode scalar values using sequences of bytes.

Check your understanding

Why do the two café representations have different code-point counts but the same intended displayed word?

Answer: One uses a precomposed accented letter, while the other uses a base letter plus combining accent. These sequences are canonically equivalent even though their stored components differ.

Related terms

• Unicode
• UTF-8
• Glyph
• Normalization
• Tokenization
• code unit
• grapheme cluster

Learn next

Read Unicode normalization and grapheme clusters. Compare code-point and byte counts for a word with a combining mark, then specify which count an annotation system should use and how it will stay consistent after normalization.

• Unicode
• Normalization

More in Natural language and speech

• Closed class - A closed class is a grammatical word category whose membership is relatively stable and does not readily expand through ordinary creation of new words.
• Closure, stop - A stop closure is the interval during production of a stop consonant when the relevant articulators form a complete obstruction in the vocal tract.
• Cloze task - A cloze task asks a person or model to supply missing material in a text using the available context.
• Code switching - Code switching is the use of more than one language or language variety within a conversation, message or utterance.
• Codebook - A codebook is a defined collection of representations identified by codes or indices.
• Codec - A codec is a method or implementation for encoding and decoding a signal, commonly to store or transmit it efficiently.

Sources

Unicode Consortium, Glossary: https://www.unicode.org/glossary/#code_point. Code-point and surrogate terminology.

Unicode Standard Annex #15: https://unicode.org/reports/tr15/. Canonical and compatibility normalization.

Jurafsky and Martin, Words and Tokens: https://web.stanford.edu/~jurafsky/slp3/2.pdf. NLP context. The café byte counts were independently calculated; the fictional search case is illustrative.

Previous: Code-excited linear prediction Next: Code switching