Does anisotropy mean all embeddings are identical?

No. Vectors may share preferred directions while retaining meaningful differences in other components.

Does normalization to length one make embeddings isotropic?

No. It preserves directions, so directional concentration can remain.

Is centering always beneficial?

No. It changes similarities and may remove useful information. Its value must be evaluated for the model, data, and task rather than assumed from the geometry alone.

AI Glossary / Natural language and speech / Advanced

Anisotropy

Definition

Anisotropy means that a distribution has different properties in different directions. In language embeddings, it often describes vectors concentrated in preferred directions instead of being spread evenly through the representation space. This geometry can affect how similarity scores behave: two unrelated items may have a substantial cosine similarity because their vectors share a strong common directional component.

What it means

The term comes from a broader mathematical and scientific idea, but this entry focuses on NLP representations. Anisotropy is a property of a collection of vectors under a stated analysis, not a label for the meaning of one word.

Different diagnostics examine different aspects of geometry, such as average pairwise cosine similarity or the distribution of variance across directions. They are not interchangeable definitions in every study. A result depends on the model, layer, sampled texts, token selection, and preprocessing, so “the embeddings are anisotropic” should be accompanied by those details.

How it works

An embedding maps an item or contextual occurrence to a vector. If many vectors share a substantial component in one direction, their dot products and cosine similarities can be large even when the information that distinguishes them lies elsewhere.

Researchers can inspect similarities among randomly selected representations and examine directional variation. Centering removes a common mean; other transformations may rescale directions or remove selected components. These operations change the representation and can also remove useful information, so their effect should be measured on the intended task.

Unit-length normalization does not automatically solve anisotropy. It changes vector magnitudes while preserving their directions. If vectors point into a narrow region before normalization, making their lengths equal does not spread those directions uniformly. Conversely, a small average cosine alone does not prove complete directional uniformity.

Three practical examples

1. Semantic search: A team investigating uniformly high similarity scores may inspect the embedding geometry before interpreting a high cosine as strong evidence of relevance.

2. Model analysis: Researchers comparing token representations across layers can account for background similarity, so a shared directional tendency is not mistaken for identical contextual meaning.

3. Retrieval evaluation: A team can compare a proposed centering or whitening step with the original vectors on held-out queries, checking whether the transformation improves useful ranking rather than only a geometric diagnostic.

Think of it this way

Think of many arrows pointing roughly east, with small variations toward north or south. Two arrows can look similar mainly because they share the large eastward component. The analogy helps explain directional concentration, but real embedding spaces have many dimensions and complex distributions. A single compass sketch cannot show every form of anisotropy or determine which variation carries useful meaning.

Worked example

Take two invented vectors, u = (10, 1) and v = (10, -1). Their dot product is 100 - 1 = 99. Each has length sqrt(101), so their cosine similarity is 99/101, approximately 0.9802.

Most of that similarity comes from their shared first coordinate. Dividing each vector by its length leaves the cosine unchanged: unit normalization does not remove their common direction.

Their mean is (10, 0). After subtracting it, the vectors become (0, 1) and (0, -1), whose cosine is -1. Centering has radically changed the comparison. However, these centered points still occupy only one line in a two-dimensional space; they have not become a uniformly distributed sample over directions.

Nor does the original cosine of 0.9802 mean a 98.02 percent probability of semantic equivalence. It is a geometric quantity. The centered score does not prove semantic opposition either. This deliberately tiny example illustrates a shared component and the effect of preprocessing, not a realistic estimate of anisotropy or a validated semantic metric.

Illustrative case study

Fictional case: Birch Knowledge tests a document-retrieval prototype and notices that many unrelated passages receive high raw cosine scores. A developer proposes removing a common component and declaring all remaining scores more meaningful.

The team instead compares rankings and relevant-document retrieval on a fixed evaluation set. It records the model version, representation layer, pooling method, and transformation fitted on training data. The fictional case demonstrates an evaluation decision; it reports no measured gain and does not imply that postprocessing always improves embeddings.

Common misconceptions and mistakes

Do not equate anisotropy with failure. A directional structure can coexist with useful task information. Whether a transformation helps depends on what the downstream system needs.

Do not assume all models or layers share the same geometry because one study found a particular pattern. Also avoid claiming that zero-mean vectors are automatically isotropic, or that unit normalization removes dominant directions. These operations address different properties. A diagnostic should be named and interpreted within its mathematical assumptions rather than used as a universal quality score.

Business use

For business systems using embeddings, the practical issue is whether similarity supports the intended retrieval or grouping task. Geometry analysis can explain puzzling score distributions and prevent unsupported confidence interpretations.

Before modifying a deployed representation, evaluate relevant queries, rare terminology, and important subgroups of documents. Fit any data-dependent transformation on the appropriate training material and apply it consistently to query and document vectors. Changing only one side of a similarity comparison can make the spaces incompatible, even when the transformation looks sensible in isolation.

Check your understanding

Two vectors have cosine similarity 0.98. Can that number be read directly as a 98 percent chance that their texts mean the same thing?

Answer: No. Cosine measures an angle-related geometric relationship. Anisotropy and other factors can influence it, and a probabilistic interpretation would require a separately justified calibration procedure.

Related terms

Embedding
Cosine similarity
Centering
isotropy
representation geometry
whitening
principal component
contextual representation

Learn next

Compare angular concentration, mean removal, and covariance structure on a small vector set. Then evaluate any proposed transformation with a real retrieval objective instead of relying on one geometric statistic.

Sources

Source: Kawin Ethayarajh, How Contextual Are Contextualized Word Representations?, original 2019 paper, read for anisotropic contextual representations and model-layer-dependent analysis: https://aclanthology.org/D19-1006.pdf . Its findings concern the models and setup studied, not every current embedding system. The two-vector calculation and operational recommendations are original explanations.

Previous: Anisotropic filtering Next: Annealed importance sampling