Does each row represent a single occurrence?

Usually a row aggregates many occurrences of the same target word; occurrence-specific designs are different representations.

Must columns be the same words as rows?

No. That is a common construction, but context features can be selected or defined differently.

Is the matrix itself a probability table?

Raw counts are not normalized probabilities. A probability interpretation requires a specified normalization and sampling meaning.

AI Glossary / Mathematics and optimization / Beginner

Word-context matrix

Definition

A word-context matrix records how target words occur with selected contexts in a text collection. Each row represents a target word, each column represents a context feature, and an entry usually begins as a co-occurrence count. A row can then serve as a count-based vector representation of that word. What counts as context depends on an explicit rule, such as neighboring words within a fixed window.

What it means

The matrix summarizes usage patterns rather than storing dictionary definitions. If two words repeatedly appear in similar contexts, their rows may be similar, supporting a distributional comparison.
A common construction uses the same vocabulary for rows and columns, producing a square word-by-word matrix. Other designs use different target and context vocabularies, directional features, or selected context types, so a word-context matrix need not always be square.
The counts depend on the corpus and preprocessing. Tokenization, case handling, sentence boundaries, window size, and treatment of repeated words all affect what the entries mean. A matrix without those choices documented is difficult to reproduce or interpret.

How it works

Choose target and context vocabularies and initialize counts. For each target occurrence, inspect the positions permitted by the context rule and increment the matching target-context entries. A two-sided one-word window, for example, includes the immediately preceding and following tokens when available.
The resulting rows can be compared directly or transformed through a documented weighting scheme before comparison. Sparse storage is often useful because many possible target-context pairs never appear in the corpus.
Keep the count matrix distinct from a learned context-embedding matrix. The former indexes actual context features and stores counts or derived weights; the latter stores learned latent coordinates and need not have the same dimensions or interpretation.

Three practical examples

A language-analysis project can build transparent count vectors to explore which words share local contexts in a domain corpus.
A teaching exercise can compare different window sizes and observe how the captured relationships change.
A retrieval or representation experiment can use a co-occurrence matrix as a starting point for weighting, dimensionality reduction, or comparison with learned embeddings.

Think of it this way

Think of a tally sheet with a row for each word and a column for each kind of company it keeps. Every allowed encounter adds a mark. The analogy explains co-occurrence, but being seen together does not establish friendship, synonymy, or a causal relationship. Similarly, a count matrix records observed linguistic proximity, not a complete theory of meaning.

Worked example

Use three invented sentences: red fox runs; red bird flies; blue bird runs. Treat spaces as token boundaries, keep sentences separate, and use the immediately adjacent word on either side as context. Exclude the target position itself.
Select rows fox and bird, and columns [red, blue, runs, flies]. Fox has red on its left and runs on its right, giving row [1,0,1,0]. Bird occurs twice: once between red and flies, and once between blue and runs. Its row is [1,1,1,1].
The fox row sums to 2 even though fox occurs only once. Row totals count context events, not necessarily target-token frequency. The bird row sums to 4 for two target occurrences.
If the full vocabulary is used on both axes, each three-word sentence has two adjacent pairs. A symmetric two-sided window counts both target-context directions for each pair, so the full matrix has 3×2×2=12 total count events.
Changing to a right-only window changes the selected rows: fox becomes [0,0,1,0], and bird becomes [0,0,1,1]. Neither result is wrong; they implement different definitions of context. The example's six-word vocabulary and counts are synthetic and do not support real-language conclusions.

Illustrative case study

In a fictional product-support analysis, a team compares word-context matrices from two months and reports a sudden change in terminology. It later discovers that the newer pipeline crosses sentence boundaries while the older one stops at them.
The analysts rebuild the comparison using a consistent rule and inspect source examples for the largest remaining changes. They also account for the different volume and composition of the two text collections.
The lesson is that a changed matrix can reflect changed preprocessing or sampling rather than changed language. The fictional exercise does not claim that all observed differences disappear after standardization.

Common misconceptions and mistakes

Do not confuse a word-context matrix with a term-document matrix. Their columns represent different things: contexts in one and documents in the other.
Do not interpret a zero as proof that two words cannot co-occur. It means no event was counted under the selected data and rule.
Do not assume the matrix is always symmetric. It can be symmetric under a matching vocabulary and symmetric counting convention, but directional contexts or unequal vocabularies change that.
Finally, frequent words can dominate raw counts. A weighting scheme may change their influence, but it changes the representation and should be recorded rather than silently substituted for raw counts.

Business use

The business value is a transparent view of language patterns that can support terminology exploration or a baseline representation. Its limitations include corpus bias, rare-event sparsity, ambiguity, and the loss of detailed sentence structure.
Keep the corpus scope and counting specification with the matrix. When comparing periods, customer groups, or model versions, check whether the observed differences survive consistent preprocessing and sensible normalization. Inspect actual text examples before turning a numerical association into a product conclusion.

Check your understanding

Under a two-sided one-word window, a target appears between two identical context words. How many context events does that occurrence contribute to that context column?

Answer: Two, if the rule counts token positions separately. Counting only distinct context types would be a different rule and must be stated.

Related terms

Learn next

Construct the full six-by-six matrix for the worked corpus and verify the total count of 12. Then compare right-only and two-sided windows. Study count weighting before comparing a raw co-occurrence row with a learned embedding.

Sources

Jurafsky and Martin, Speech and Language Processing, chapter 5, section 5.3: word-context matrix construction and count-based embeddings. https://web.stanford.edu/~jurafsky/slp3/5.pdf

Previous: Wold's theorem Next: Word embedding