Are word embeddings the same as one-hot vectors?

No. A one-hot vector identifies a vocabulary item without directly learning graded relationships between items.

Does a larger embedding always work better?

No. Dimensionality is one design choice; training, data, and task fit also matter.

Can one word have several representations?

Yes. Contextual models can produce different representations for different occurrences, and other methods can explicitly model multiple senses.

AI Glossary / Retrieval and knowledge systems / Intermediate

Word embedding

Definition

A word embedding is a numerical vector used to represent a word in a space where learned relationships can support language-processing tasks. Traditional static embeddings assign a word one vector, while contextual representations vary with its surrounding text. Modern systems often operate on subword tokens instead of complete words. An embedding encodes statistical information; it is not a dictionary definition or proof that a system understands the word.

What it means

Words such as camera and lens often occur in related contexts. A model can learn vectors that reflect aspects of these patterns, enabling mathematical operations on language representations. Similarity between vectors can help a system retrieve related material or supply useful features to another model.
The representation depends on training data, objective, and model architecture. Nearby vectors might reflect topical association rather than synonymy. Opposites can also occur in similar contexts. A vector therefore needs interpretation through the task and evidence, rather than through an assumed universal map of meaning.

How it works

Choose a vocabulary or tokenization scheme and training material appropriate to the intended task. In static approaches, a model learns a vector associated with each vocabulary item. Word2vec learns representations through predictive objectives involving words and their contexts; GloVe uses information from aggregated word co-occurrence statistics.
Contextual language models instead transform token representations using the surrounding sequence. The representation of bank in a discussion of rivers can differ from its representation in a discussion of loans. The exact layer and aggregation method matter when using these representations outside the original model.
Vectors can be compared with measures such as cosine similarity or supplied to downstream classifiers and retrieval systems. For multiword text, a separate composition or encoding method is needed: a word embedding alone does not specify how to represent an entire document.
Evaluate the representation on realistic examples, including ambiguous words, domain terminology, and relevant languages. Similarity scores are not probabilities of truth or relevance. Updating a model can change its coordinate system, so vectors from different versions should not be mixed without a supported alignment or migration procedure.

Three practical examples

A product-search team uses word representations as part of a system that connects customer vocabulary with catalog descriptions.
A document classifier uses embeddings to learn patterns associated with categories, while measuring performance on held-out documents.
A language researcher examines associations in an embedding space, investigating whether patterns reflect useful relationships or unwanted biases in the training material.

Think of it this way

Think of an embedding as a learned set of coordinates on a language map. Locations can help compare items, but the map was drawn from particular observations for a particular purpose. Its axes do not necessarily correspond to named concepts, and nearby locations do not guarantee interchangeable meanings.

Worked example

Consider invented two-dimensional vectors: lens = (3, 4), camera = (4, 3), and bread = (−4, 3). Each vector has length 5. The lens - camera dot product is 3 × 4 + 4 × 3 = 24, so their cosine similarity is 24 / 25 = 0.96.
The lens - bread dot product is 3 × (−4) + 4 × 3 = 0, giving cosine similarity zero. Under this deliberately constructed representation, lens is much closer in direction to camera than to bread.
Now rotate every vector by 90 degrees using the rule (x, y) → (−y, x). Lens becomes (−4, 3) and camera becomes (−3, 4). Their dot product remains 24, their lengths remain 5, and their cosine remains 0.96.
This calculation shows why individual coordinates need not have fixed human-readable meanings: the axes changed while the relationship stayed the same. These are teaching vectors, not measurements from a trained model, and the example does not establish that any production embedding will reproduce these relationships.

Illustrative case study

In this fictional case, Harbor Camera Supply wants customers searching for everyday expressions to find relevant equipment. Its team evaluates an embedding-based component alongside its existing catalog matching.
Tests reveal useful associations between some equipment terms, but also confusion between accessories and complete products. The team adds product-type constraints and evaluates query groups separately. It also retains exact matching for model numbers, where a nearby-looking term can identify an incompatible item.
The pilot is judged on retrieved product relevance and customer task completion. A persuasive nearest-neighbor demonstration is treated as an initial diagnostic, not as evidence that the whole search system is ready.

Common misconceptions and mistakes

A common mistake is assuming every embedding dimension names a concept. Dense learned coordinates usually do not have that simple interpretation. Another is equating proximity with synonymy: association, opposition, or shared context may produce closeness.
Static embeddings do not distinguish every sense of a word, while contextual representations still require evaluation. Neither approach eliminates training-data bias. Mixing vectors from unrelated models can make comparisons meaningless even when their dimensions match. Finally, representing a word numerically is only one component of a language application, not a complete reasoning or retrieval system.

Business use

Businesses can use embeddings as reusable features for search, classification, clustering, and other language workflows. Selection should consider vocabulary coverage, languages, domain fit, computational cost, and the intended comparison method.
A useful evaluation includes terminology that matters operationally, ambiguous expressions, and examples where mistakes have different consequences. Version the model and preprocessing together, and reassess downstream behavior when either changes.

Check your understanding

Why does rotating all vectors preserve the example's cosine similarity?

Answer: The rotation preserves lengths and dot products.

Why might antonyms appear near each other?

Answer: They can occur in similar linguistic contexts.

Why must a search team evaluate complete retrieval results?

Answer: Useful word relationships do not ensure correct document or product ranking.

Related terms

Learn next

Learn next how word2vec and GloVe learn representations, then compare static and contextual embeddings. Study text pooling and retrieval evaluation before applying word-level similarities to complete documents.

Contextual embeddings

Sources

Jurafsky and Martin, Speech and Language Processing, Chapter 5, Embeddings: https://web.stanford.edu/~jurafsky/slp3/5.pdf Mikolov and colleagues, Efficient Estimation of Word Representations in Vector Space: https://arxiv.org/abs/1301.3781 Stanford NLP, GloVe project and original paper: https://nlp.stanford.edu/projects/glove/

Previous: Word-context matrix Next: Word error rate