Does a biword index store two documents together?

No. Biword refers to an adjacent pair of terms, usually associated with document identifiers.

Can it answer every longer phrase exactly by intersection alone?

No. The required pair occurrences may be at incompatible positions.

Is it the same as a positional index?

No. A positional index records term locations; a basic biword index can record pair presence without those locations.

AI Glossary / Retrieval and knowledge systems / Intermediate

Biword index

Definition

A biword index is a search index whose entries represent pairs of consecutive terms. Instead of recording only where an individual word occurs, it records which documents contain a particular adjacent two-term sequence. This can make two-word phrase lookup straightforward. Combining several biword lookups can generate candidates for a longer phrase, but document-level pair matches alone do not prove that the entire longer phrase occurs consecutively.

What it means

A document containing artificial intelligence includes the biword artificial intelligence, while intelligence artificial is a different pair. An ordinary term index can establish that both words occur somewhere in a document, but cannot by itself establish adjacency.
The meaning of consecutive depends on tokenization and normalization. If preprocessing removes words or treats punctuation as a boundary, the indexed pairs reflect that policy. A biword index is a representation of the processed token stream, not automatically a literal character-level account of the original document.

How it works

Tokenize a document, form overlapping pairs, and add its identifier to the postings list for each pair. A four-token sequence a b c d yields a b, b c, and c d. Repeated occurrences can share one document-level posting when only presence is needed.
For a two-term phrase, look up its pair. For a longer query, intersect the relevant pair postings to obtain candidates, then verify complete alignment using positions or source text. A separate single-term index is usually useful because a pair index does not naturally cover all single-word queries.

Three practical examples

A glossary search can index common two-word terms, such as neural network, for direct phrase retrieval.
A document archive can use pair postings to narrow candidates before checking a longer quoted passage.
A search-engine experiment can compare the storage and query-time tradeoffs of selected phrase entries and a general positional index.

Think of it this way

Think of a catalog that records neighboring word pairs on separate cards. A card for each required pair shows that the pieces exist, but does not prove that all the pieces connect in one place. The analogy is limited because a real index can store additional information, such as positions, which may resolve that uncertainty directly.

Worked example

Consider three original miniature documents, using lowercase space-separated tokens and no punctuation boundaries. D1 is red green blue. D2 is red green x green blue. D3 is red x green blue. Build document-level postings for the two pairs needed by the query red green blue.
The pair red green appears in D1 and D2, giving postings {D1, D2}. The pair green blue appears in all three documents, giving {D1, D2, D3}. Their intersection is {D1, D2}. D3 is correctly excluded because its red and green are not adjacent.
However, only D1 contains the full three-word phrase. In D1, red green begins at token position 1 and green blue begins at position 2, so the overlapping pairs align. In D2, red green begins at position 1 but green blue begins at position 4. Both pair conditions are true, yet they belong to different locations. D2 is a false positive for the full phrase.
If these are the only documents and D1 is the only true phrase match, the pair-intersection candidate set has precision 1/2 and recall one. A verification stage removes D2, leaving the correct result. The intermediate false positive is acceptable if the system clearly treats intersection as candidate generation and verifies before presenting exact phrase matches.
Now query just green blue. Under the stated tokenization, the biword posting is sufficient to show that the exact two-token sequence appears in all three documents. The difficulty arises when joining multiple pair occurrences without positional alignment, not from a failure to record the indexed pair itself.
Across the three documents there are 2 + 4 + 3 = 9 adjacent-pair occurrences before deduplication. That is different from the number of distinct pair keys or document postings.

Illustrative case study

Fictional case: a training-content website adds quoted-phrase search using pair intersections. A tester notices that a long query returns an article where the opening pair occurs near the top and the closing pair appears much later.
The team adds a verification step and preserves a small adversarial example like D2 in its acceptance checks. It also documents whether punctuation and removed words affect adjacency. Search labels now match the actual guarantee provided by the index and verification stages.

Common misconceptions and mistakes

Do not assume that finding all adjacent pairs somewhere in one document proves a longer phrase occurs there. Their positions must connect appropriately. Conversely, an exact indexed pair lookup can establish that pair's occurrence under the indexing policy without needing the longer-phrase reasoning.
Do not confuse a biword index with a language model that estimates the probability of the next word. Both involve pairs, but their stored information and purposes differ. Also avoid silently crossing document boundaries when forming pairs; the final word of one document and the first word of another should not become a phrase in either document.

Business use

A biword index can support selected frequent phrase queries or complement another index. Its usefulness depends on query patterns, vocabulary growth, and the cost of verifying longer phrases.
Measure both candidate counts and final query latency. Preserve a documented tokenization policy and rebuild or migrate derived indexes when that policy changes. Test repeated words, punctuation, short documents, and phrases whose component pairs appear far apart. Choose index structures based on the guarantees the product needs rather than assuming that storing more phrase keys always produces a better search system.

Check your understanding

Why is D2 a candidate but not a true match for red green blue?

Answer: It contains both required pairs, but red green starts at position 1 and green blue at position 4. The pairs do not overlap to form the requested consecutive sequence.

Related terms

Learn next

Add D4 with text red green red green blue. Identify every start position for the two required biwords and determine whether at least one aligned chain forms the full query. Explain why repeated pair occurrences make document-level presence less informative than positions.

Sources

Primary textbook source: Manning, Raghavan, and Schuetze, Introduction to Information Retrieval, Biword indexes: https://nlp.stanford.edu/IR-book/html/htmledition/biword-indexes-1.html . All miniature documents, positions, counts, and candidate calculations here are original examples.

Previous: Bivariate normal model Next: BL (logic)