Can a term be a phrase?

Yes, if the vocabulary and tokenizer define it that way.

Does the matrix have to fit densely in memory?

No. Sparse representations are common because many term-document pairs have zero values.

Can matrix factorization be applied?

Yes. Low-rank methods can derive compressed representations, but they change the representation and require their own evaluation.

AI Glossary / Mathematics and optimization / Intermediate

Term-document matrix

Definition

A term-document matrix represents a collection of documents using numerical associations between terms and documents. In the orientation used here, rows are terms, columns are documents and each cell records a count or weight for one term in one document.

What it means

The matrix turns text into a form that supports numerical comparison. A document becomes a column vector describing which vocabulary terms it contains and how strongly they are represented. A term becomes a row vector describing its distribution across documents.

Raw occurrence counts are one possible cell value. Binary presence indicators and weights such as term frequency - inverse document frequency are other choices. The matrix's meaning therefore depends on its vocabulary, text processing and weighting rules.

Many software interfaces use the transpose, with documents as rows and terms as columns, often called a document-term matrix. Neither orientation is inherently better. Confusion arises when an operation assumes one layout while receiving the other.

How it works

Begin by defining what counts as a document and a term. Terms might be normalized words, subword tokens or selected phrases. Tokenization, capitalization handling and vocabulary filtering determine which occurrences are counted together.

Count each vocabulary term in each document, then optionally transform the counts. A frequency weighting can reduce the influence of ubiquitous terms or adjust for document length. Record the exact formula because similarly named schemes can differ.

Most documents contain only a small portion of the vocabulary, making the matrix sparse. Sparse storage can avoid explicitly recording every zero. Similarity measures, classifiers and matrix decompositions can then operate on the numerical representation.

A basic count matrix discards word order. It also does not automatically distinguish senses of an ambiguous word. Additional features or other representations are needed when those distinctions matter to the task.

Three practical examples

A search prototype compares a query vector with document vectors using a defined weighting and similarity rule.
A document classifier uses vocabulary counts as features for routing incoming requests.
A corpus analyst examines which terms are concentrated in particular document groups, while checking effects of document length and preprocessing.

Think of it this way

Think of a checklist with one row per ingredient and one column per recipe, recording how often each ingredient appears. Recipes with similar columns share ingredients.

The analogy explains count-based comparison. A list of ingredients does not describe preparation order, just as a basic term-document matrix does not preserve sentence structure or guarantee similarity of meaning.

Worked example

Building counts and comparing two documents

Use the vocabulary [cat, dog, runs]. Let D1 be “cat runs cat,” D2 be “dog runs” and D3 be “cat dog.” Ignore case and count only these exact words.

With terms as rows and documents as columns, the cat row is (2, 0, 1), the dog row is (0, 1, 1) and the runs row is (1, 1, 0). Thus D1's column is (2, 0, 1) and D2's is (0, 1, 1).

Their dot product is 1. Their lengths are √5 and √2, giving cosine similarity 1/√10, approximately 0.316. D3's vector is (1, 1, 0); its cosine with D1 is 2/√10, approximately 0.632. Under this raw-count comparison, D1 is more similar to D3 than to D2.

The column sums are 3, 2 and 2, matching the retained-token counts. Reordering D1 to “runs cat cat” leaves its column unchanged. This verifies both the representation and a limitation: the count vector cannot distinguish those sequences. Different weighting or vocabulary choices could change the similarity ranking.

Illustrative case study

A fictional knowledge-base team builds a term-document matrix to find related articles. Its first results favor long articles that repeat broad terms, and a few short queries become all-zero vectors because their words are absent from the stored vocabulary.

The team reviews weighting, length normalization and vocabulary coverage. It adds explicit handling for zero vectors rather than attempting to divide by their zero length in cosine calculations. It then evaluates search results against relevant example queries.

The lesson is that the matrix is a representation, not a finished search-quality guarantee. Document boundaries, preprocessing and comparison rules determine what information remains available to the downstream system.

Common misconceptions and mistakes

Every cell must be a raw count. Binary values and transformed weights are also common.
Term-document and document-term always use the same orientation. Confirm row and column meanings.
Similar count vectors guarantee equivalent meanings. Word order, negation and ambiguity can be lost.
A zero cell means a document does not discuss a concept. It only means the defined term has zero weight under the chosen representation.
Vocabulary choices are harmless formatting. They directly change the feature space.

Business use

Term-document matrices provide transparent baselines for text classification, retrieval and corpus exploration. Their entries can often be traced directly to vocabulary choices and observed text, which helps explain model inputs.

Preserve the vocabulary ordering, tokenizer configuration and weighting statistics with the model. Fit data-dependent transformations within the appropriate training procedure to avoid evaluation leakage. Monitor new documents for vocabulary gaps and changes in language use. Compare the baseline with alternatives using the actual task's relevance or prediction criteria, rather than assuming that a more complex representation is automatically necessary.

Check your understanding

Why do D1 and its reordered version have identical columns in the example?

Answer: The representation counts occurrences of each vocabulary term and ignores order. Both strings contain two instances of cat and one of runs.

Related terms

Learn next

Read tf-idf and cosine similarity. Add a fourth document containing a word outside the example vocabulary, then explain the difference between ignoring that word and expanding the matrix with a new term row. Check how any learned weighting statistics would change.

Sources

Jurafsky and Martin, Speech and Language Processing, Embeddings chapter, August 19, 2026 draft: https://web.stanford.edu/~jurafsky/slp3/5.pdf. Term-document representation.

Manning, Raghavan and Schütze, Introduction to Information Retrieval: https://nlp.stanford.edu/IR-book/html/htmledition/matrix-decompositions-and-latent-semantic-indexing-1.html. Matrix orientation and decomposition context. Counts, cosine calculations and fictional search case are original examples.

Previous: Term discrimination Next: Term partitioning