AI Glossary / MLOps and deployment / Advanced
Binary Ordered Compression for Unicode
Definition
Binary Ordered Compression for Unicode (BOCU) is a Unicode text-compression approach designed to preserve code-point ordering in the encoded byte representation. Its BOCU-1 profile combines a compact, stateful encoding with specific rules for control characters and text interchange.
What it means
Unicode assigns numerical code points to characters. A text encoding turns those values into bytes. BOCU exploits the fact that nearby characters in a string often belong to nearby regions of the Unicode code space, making differences economical to represent.
Binary ordered refers to comparing encoded byte sequences in a defined order. It does not mean alphabetical order in every language. Code-point ordering and culturally appropriate sorting are separate matters.
BOCU-1 is described in Unicode Technical Note #6. That document is informational; publication as a technical note is not equivalent to endorsement as a Unicode standard. The base algorithm and the particular BOCU-1 profile should also be distinguished when discussing implementation.
How it works
BOCU-1 maintains a small state used as a reference value. For ordinary characters, it encodes the difference between the current code point and that reference, then updates the reference according to specified rules. The reference is not always simply the last character.
Some ranges receive special treatment. Spaces and control characters also have defined behavior: a space does not change the state, while C0 control characters reset it. These rules are part of the format, not optional encoder preferences.
The resulting bytes must be decoded with matching state. Starting in the middle of an arbitrary byte sequence is not generally sufficient to identify character boundaries and reconstruct text. Proper reset points and record boundaries therefore matter.
The order-preserving property also has conditions. BOCU-1 includes a special reset-only command whose use can break ordering. Applications relying on binary comparisons must follow the relevant restrictions rather than assuming every permitted stream arrangement remains sortable.
Three practical examples
1. Short-string storage experiments: A database team can evaluate BOCU-1 for many short multilingual strings, comparing complete storage and decoding costs with its existing encoding.
2. Legacy data conversion: An archive containing BOCU-1 text needs a matching decoder and an explicit encoding declaration. Treating the bytes as UTF-8 can produce errors or corrupted text.
3. AI corpus preparation: A data team can decode stored text into its pipeline's expected representation, then test exact code-point recovery before tokenization. Compression should not silently change punctuation, combining marks or record boundaries.
Think of it this way
Think of recording locations as offsets from a moving reference point, while arranging the written offset codes so that comparable complete records retain a chosen numerical order.
The analogy does not imply that distances in code-point space reflect similarity in meaning. Adjacent Unicode values are not necessarily interchangeable letters, and preserving their numerical order does not implement a language's dictionary-sorting rules.
Worked example
Worked example
This example traces BOCU-1's reference-state logic, not its final byte encoding. Use the initial reference 0x0040, which is decimal 64. Greek small alpha is U+03B1, decimal 945.
For alpha, the difference from the initial reference is 945 - 64 = 881. Under the ordinary block-based update rule, alpha's reference becomes 0x03C0, decimal 960: the middle reference for its 128-code-point block.
Greek small beta is U+03B2, decimal 946. If beta follows alpha, its difference is 946 - 960 = -14. This shows why a run in the same region can use small differences after its initial setup.
If a space occurs between alpha and beta, the space leaves the reference unchanged, so beta still has difference -14. If a newline occurs instead, that control character resets the reference to 64, and beta's difference becomes 946 - 64 = 882.
These differences must still pass through the exact BOCU-1 byte-mapping rules. Writing 881 or -14 as ordinary decimal text is not BOCU-1 encoding. The example explains state dependence without pretending to supply a complete encoder or a measured compression ratio.
Illustrative case study
Illustrative case study - fictional
A fictional catalog service inherits a collection of compressed multilingual names. A migration script initially concatenates independently encoded records and treats the result as one continuous stream.
Testing reveals that decoding depends on where each record begins and which state applies. The team returns to the documented record structure, decodes each string correctly and verifies its code-point sequence against known samples.
It separately checks sorting requirements. Customers expect language-sensitive name ordering, so binary code-point order alone is insufficient. The team retains explicit collation rules after conversion. This invented case reports no product result; it illustrates encoding-state and sorting distinctions.
Common misconceptions and mistakes
- Binary order is not automatically the alphabetical order users expect in a particular language.
- BOCU-1 is stateful; arbitrary byte offsets are not reliable decoding start points.
- Most printable ASCII characters do not keep their ordinary ASCII byte values in BOCU-1.
- Compression does not perform Unicode normalization or translation.
- A technical note's historical benchmarks are not guaranteed savings for a new dataset.
Business use
BOCU is mainly relevant when investigating specialized text storage or maintaining an existing format. A deployment decision should consider compatibility and decoding complexity alongside byte savings.
Test realistic scripts, mixed-language strings, short records and control characters. Verify exact round trips before comparing compression. If sorting encoded strings is important, document initial-state and reset conventions and test the required ordering explicitly. For web-facing text, follow the supported encoding of the actual platform; specialized internal storage does not justify serving an unsupported character encoding to users.
People also ask
Is BOCU-1 the same as UTF-8?
No. Both represent Unicode text, but their byte encodings and state requirements differ.
Does it preserve what a human considers the same text?
It preserves the encoded code-point sequence when decoded correctly. Canonical equivalence and normalization are separate Unicode concerns.
Can ordinary compression work on Unicode text?
Yes. Unicode's compression guidance discusses general-purpose compression as a practical option, particularly for longer text. A specialized character encoding is not required simply because the content is multilingual.
Check your understanding
A system sorts BOCU-1 strings in binary order. Does that alone guarantee the expected alphabetical ordering for every supported language?
Answer: No. Code-point ordering differs from language-sensitive collation. The application needs an appropriate collation policy.
Related terms
- Unicode
- UTF-8
- Code point
- Lossless compression
- character encoding
- stateful encoding
- collation
- Unicode normalisation
Learn next
Compare the code-point sequences of a precomposed accented character and a base character plus combining mark. Then separate three questions: exact encoding recovery, canonical equivalence and user-facing sort order.
More in MLOps and deployment
- Audio codec - An audio codec is a method, together with an implementation of that method, for encoding and decoding digital audio.
- Autonomic computing - Autonomic computing is an approach to building computing systems that manage aspects of their own operation according to human-defined goals and policies.
- Average bitrate - Average bitrate is the total number of bits used or transferred divided by the elapsed duration, usually expressed in bits per second.
- Bit rate - Bit rate is the number of binary digits, or bits, transferred or represented per unit of time, normally measured in bits per second.
- Bitstream format - A bitstream format is the set of rules that defines how information is arranged and interpreted in a sequence of bits.
- Brotli - Brotli is a general-purpose lossless compression format and algorithm family for representing data in fewer bytes when its patterns permit.
Sources
Sources: Scherer and Davis, Unicode Technical Note #6, BOCU-1: https://www.unicode.org/notes/tn6/ ; Unicode compression FAQ: https://www.unicode.org/faq/compression.html . The profile's state pseudocode, order restrictions and technical-note status were reviewed. The state calculation is original; linked C source files could not be rendered by the retrieval tool, so no implementation audit is claimed.
Previous: Binary independence model Next: Binary regression