Does a positional index store page layout?

Not necessarily. Its usual positions are token indices, with layout information requiring additional data.

Can it support phrases with repeated words?

Yes, when the query evaluator checks all required offsets and occurrences.

Is every proximity query ordered?

No. Ordered and unordered conditions are different operators or settings that the application must define.

AI Glossary / Retrieval and knowledge systems / Intermediate

Positional index

Definition

A positional index is an inverted search index that records where each term occurs within each document, usually as token positions. This lets a search system test word order, exact phrases, and proximity rather than merely checking whether words appear somewhere in the same document. The positions describe the indexed token stream and must be interpreted using the system's tokenization and boundary rules.

What it means

A basic posting might say that a document contains a term. A positional posting adds the locations of its occurrences. Those locations make the difference between knowing that red and blue both appear and knowing that red immediately precedes blue.
Token positions are not automatically character offsets, page numbers, or screen coordinates. Applications needing highlights or page navigation may store additional location information alongside positions.

How it works

During indexing, assign positions to tokens and store each term's sorted occurrence positions under the relevant document identifier. At query time, find candidate documents and compare term positions using the required offsets.
For a phrase of m tokens, an occurrence beginning at p requires the token at query offset i to occur at p+i. Repeated query terms need distinct compatible occurrences. Proximity queries can use related checks, but their distance, ordering, and boundary conventions must be stated. A positional index provides evidence; the query operator defines how that evidence qualifies as a match.

Three practical examples

A document portal can support quoted searches for an exact multiword expression.
An archive can find terms occurring near each other even when they do not form a fixed phrase.
A search-quality test can distinguish a document containing all query terms from one containing the requested term sequence.

Think of it this way

Think of an index that records seat numbers as well as which people are in a room. Knowing the seat numbers lets you check who sits next to whom and in what order. The analogy is limited because token positions depend on text processing and do not necessarily correspond to physical positions on a printed page.

Worked example

Consider two original documents using lowercase space-separated tokens, starting positions at one. D1 is red blue red green. D2 is red green blue red. In D1, red occurs at {1,3}, blue at {2}, and green at {4}. In D2, red occurs at {1,4}, green at {2}, and blue at {3}.
For the phrase red blue red, D1 has a valid start at p=1: red at 1, blue at 2, and red again at 3. D2 contains every distinct query term but has no valid start. Its red at 1 is followed by green, while its red at 4 has no later blue and red. Presence alone would accept both as candidates; the positions separate the exact phrase match.
For blue red, both documents match: D1 uses positions 2 and 3, and D2 uses 3 and 4. For red green, D1 matches at 3 and 4, while D2 matches at 1 and 2. The same term lists support several phrase queries without storing a separate entry for every full phrase.
Now define a specific unordered proximity condition: two occurrences match when their absolute token-position difference is at most two. Under this stated convention, D1 has qualifying red/green at positions 3 and 4. D2 has qualifying pairs at 1 and 2, and at 4 and 2. If the query additionally requires red before green, only D2's 1-and-2 pair qualifies there. This illustrates why proximity distance and direction must be specified separately.
Finally, imagine D3 is red and blue. If and is removed but its original position gap is preserved, red remains at 1 and blue at 3, so the exact phrase red blue does not match. Renumbering after removal could make them adjacent in the processed stream instead. Neither result should be accidental: the indexing and query policy must define which interpretation users receive. All traces here are manual examples, not a claim about a particular product's stopword behavior.

Illustrative case study

Fictional case: a research library adds exact phrase search to an existing keyword index. Tests pass for simple phrases but fail when a query repeats the same word. The implementation checks term membership without enforcing each required occurrence.
The team rewrites the positional test to retain query offsets and adds examples with repeated terms, punctuation, and removed words. It documents whether phrases can cross sentence or field boundaries. The resulting behavior is checked against the original text under the declared tokenization rules.

Common misconceptions and mistakes

Do not assume that every term occurring somewhere in a document implies the requested phrase occurs. Do not reuse one occurrence to satisfy multiple different positions in a repeated-term phrase. The query red red requires two adjacent token occurrences, not a single red with a frequency count greater than zero.
Also avoid treating token distance as character distance. Stopword handling, punctuation, and multi-valued fields can affect positions. A proximity operator in one system may count intervening words while another uses position differences, so identical-looking numeric settings can have different meanings.

Business use

Positional indexes support precise text retrieval and can reduce irrelevant results for users who know a phrase or need local context. They require more stored occurrence information than document-presence indexes, so storage and query-processing costs should be measured.
Keep tokenization, normalization, and position conventions consistent across indexing and querying. Test repeated words and boundary cases, and make phrase and proximity semantics understandable in the interface. If the tokenizer changes, assess whether the existing position data must be rebuilt. Evaluate correctness before optimizing compressed-list traversal or skipping strategies.

Check your understanding

Why does D2 fail red blue red even though it contains two red occurrences and one blue?

Answer: Their positions are incompatible with consecutive offsets 0, 1, and 2. The occurrence counts are sufficient in quantity but not in order and spacing.

Related terms

Inverted index
Proximity search (text)
Tokenization
Biword index
Positional posting
Phrase query
Proximity search
Stopword
Character offset
Term frequency

Learn next

Create a document containing red red blue red and list its positions. Find all matches for red red, red blue, and red blue red. Then state whether your matching rules allow phrases to cross a sentence boundary and explain what extra indexing convention would enforce that decision.

Sources

Primary textbook source: Manning, Raghavan, and Schuetze, Introduction to Information Retrieval, Positional indexes: https://nlp.stanford.edu/IR-book/html/htmledition/positional-indexes-1.html . All miniature documents, offsets, and proximity conventions here are original and explicitly specified.

Previous: Positional encoding Next: Positive class