Does the same word always receive a different vector?

Not necessarily. Identical inputs and deterministic settings can reproduce the same representation, and different contexts can still produce similar vectors.

Are contextual embeddings limited to BERT?

No. Different model families can produce context-dependent representations.

Does every vector dimension have a clear human meaning?

Usually not. Information is distributed across dimensions, so individual coordinates should not casually be labeled as specific concepts.

AI Glossary / Retrieval and knowledge systems / Intermediate

Contextual embeddings

Definition

Contextual embeddings are numerical representations whose values depend on the surrounding input. In language models, a token's contextual representation can change between occurrences of the same word because the model processes its context. This contrasts with a static word embedding that assigns one fixed vector to a word type.

What it means

The word bank can refer to a financial institution or the side of a river. A contextual model can represent these occurrences differently after considering the words around them. Context can also affect grammar, reference and other aspects of usage, not just dictionary sense.

The resulting vector is a learned representation, not a readable definition or a verified statement about meaning. Its usefulness depends on the model, layer, input and task. Different vectors do not necessarily imply different word senses, and similar vectors do not guarantee equivalent meanings.

How it works

The model converts text into tokens and initial representations, then processes them through context-sensitive layers. Depending on the architecture, a token may incorporate information from earlier tokens, both sides of a sequence or other permitted inputs. The resulting hidden states serve as contextual representations.

A downstream system may use token vectors directly or combine them into sentence or passage representations. Pooling and training objectives affect what those combined vectors capture. ELMo and BERT are influential examples of contextual language representations with different architectures. Contextual embedding is therefore a broad concept, not another name for one particular transformer model.

Three practical examples

1. A named-entity system uses surrounding words to distinguish a person's name from an ordinary word with the same spelling.

2. A reading-comprehension model represents an ambiguous term in the context of the passage being analyzed.

3. A retrieval system uses an encoder trained for passage similarity to produce representations that reflect each passage's content. It evaluates the pooled representations rather than assuming every language-model hidden state is automatically suitable for search.

Think of it this way

Think of interpreting a word while reading the sentence around it. The letters stay the same, but the interpretation can change. A contextual embedding provides a numerical representation influenced by that setting. Unlike a dictionary entry, it is produced for a particular input and may encode several aspects of usage at once.

Worked example

Use three invented two-dimensional vectors to illustrate comparison, not outputs from a real model. Let bank in a finance sentence be f = (1, 0), bank in a river sentence be r = (0, 1), and lender in a finance sentence be l = (0.8, 0.6).

All three vectors have length one because 0.8 squared plus 0.6 squared equals 1. Cosine similarity is therefore just their dot product. Similarity between f and l is 0.8, while similarity between f and r is zero. In this toy representation, the finance occurrence of bank is closer to lender than to the river occurrence of the same spelling.

A static lookup that assigned both bank occurrences the same vector would instead make their cosine similarity one. Real contextual models use far more dimensions, and their coordinates do not normally come with simple labels such as finance and river. The example explains the principle; it does not predict exact similarities or demonstrate a particular model's ability.

Illustrative case study

Fictional case: Willow Language analyses internal maintenance notes where seal can describe a component or an action. A static feature baseline confuses some uses. The team tests a contextual encoder with a supervised task head and separate evaluation examples.

Reviewers inspect whether surrounding sentences provide enough information and whether specialist abbreviations remain problematic. They also compare processing cost and errors on unfamiliar equipment descriptions. This invented case describes a validation process; it reports no measured accuracy gain or claim that context alone resolves every ambiguous term.

Common misconceptions and mistakes

Contextual does not mean the representation includes all relevant context. Input windows, truncation and architecture limit what the model can access. A causal model generally cannot use future tokens when representing an earlier position during ordinary forward processing.

The input embedding table and the final contextual hidden states are also different objects. Another mistake is assuming that raw token vectors can be averaged into an excellent search embedding without appropriate training or evaluation. Finally, contextual representations can carry biases and domain mismatches from training, even when they distinguish word usages more flexibly than static vectors.

Business use

Contextual embeddings can support language classification, extraction and retrieval where wording changes meaning with its surroundings. They can help a system use information beyond isolated keywords, but benefits should be demonstrated on the organization's actual task.

Document tokenization, context length, pooling and model version so representations can be reproduced. Compare performance on ambiguous language, short fragments and specialist terms. When using vectors in a search index, account for model changes that may make old and new representations incompatible. A numerical similarity remains evidence for a matching procedure, not proof that a document is correct or relevant.

Check your understanding

In the toy example, why can lender be closer to one occurrence of bank than the other occurrence of bank is?

Answer: The representations depend on usage, not spelling alone. The invented finance vectors align more closely than the finance and river vectors. Real models must be evaluated to determine whether their geometry captures the distinctions a task needs.

Related terms

Static embeddings
ELMo
Cosine similarity
Pooling
Static embedding
token representation
hidden state
contextual encoder
BERT
sentence embedding

Learn next

Compare static and contextual representations of the same ambiguous word in several sentences. Keep the model, layer and comparison method fixed. Then study tokenization and pooling before using sentence or document vectors, and evaluate whether the resulting representations serve the intended downstream task.

Pooling

Sources

Peters and colleagues, Deep Contextualized Word Representations, NAACL 2018, ELMo: https://aclanthology.org/N18-1202/

Devlin and colleagues, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019: https://aclanthology.org/N19-1423/

The vectors and Willow Language case are original hypothetical illustrations, not extracted model outputs or published experimental results.

Previous: Context window Next: Contextual searching