Are sparse vectors embeddings?

Some are described as sparse embeddings, particularly when a model learns them, but sparsity alone does not specify a learning method.

Can sparse and dense vectors be combined?

Yes, with a defined scoring or rank-combination method.

Is cosine similarity always available?

A zero vector has zero norm, so ordinary cosine division is undefined and needs an explicit handling policy.

AI Glossary / Retrieval and knowledge systems / Intermediate

Sparse vector

Definition

A sparse vector is a vector in which most entries are zero. It can represent an object using a large set of possible features while activating only a small subset. In information retrieval, sparse vectors often represent terms and their weights. Sparsity describes the pattern of values, rather than whether the vector was produced by a traditional formula or a learned model.

What it means

Imagine a vocabulary containing many thousands of terms. A short document uses only a small fraction of them, so a vector with one position per vocabulary term contains many zeros. The nonzero positions identify active features, and their values can represent counts, weights, or learned strengths.
The vector still belongs to a defined feature space even when its stored representation lists only active entries. A short list of stored values does not make it a low-dimensional vector. The positions attached to those values are essential to interpreting and comparing it.

How it works

Choose a consistent feature-to-index mapping. A text representation might assign one position to each vocabulary item; another application might use categories, events, or learned features. Construct each vector using the same mapping and agreed preprocessing.
Instead of storing every zero, a sparse representation can store pairs of indices and nonzero values. For a collection, specialized matrix formats or an inverted index can organize these entries for efficient access. The exact storage and search methods depend on the application.
A dot product sums products at matching coordinates. Coordinates where either value is zero contribute nothing, so sparse computation can focus on overlapping active positions. This can reduce work, although actual speed depends on data structures, overlap, hardware, and implementation overhead.
Sparse retrieval is not limited to raw word counts. Weighting schemes can reduce the influence of common terms, and learned models can assign sparse weights, including to expansion terms. A nonzero feature therefore need not always mean that a word literally appeared in the input. Query and document encoders must still produce compatible coordinates and weights.

Three practical examples

A document search engine represents each article through weighted vocabulary terms and searches their overlap with a query.
A classifier encodes a small number of active categories from a much larger catalog of possible features.
A hybrid retrieval system combines sparse matching with a dense representation, evaluating how the two contribute to finding relevant material.

Think of it this way

Think of an enormous checklist on which only a few boxes have marks and scores. You can store the marked box numbers and scores instead of copying every blank box. The box numbers remain necessary: scores in different boxes refer to different features.

Worked example

Consider an original vector example with twelve coordinates numbered 1 through 12. Query q has value 2 at coordinate 2 and value 1 at coordinate 9; all other entries are zero. Document d has value 3 at coordinate 2, value 4 at coordinate 5, and value 2 at coordinate 9.
The query has ten zero entries, so its zero fraction is 10/12, approximately 83.33%. The document has nine zero entries, giving 9/12, or 75%. Both are represented sparsely here, although no universal numerical cutoff defines every use of the word sparse.
The dot product is 2 times 3 plus 1 times 2, which equals 8. Coordinate 5 contributes nothing because the query value there is zero. The query norm is the square root of 5 and the document norm is the square root of 29. Their cosine similarity is therefore 8 divided by the square root of 145, approximately 0.6644.
For a separate storage illustration, suppose a vector has 100,000 coordinates and 100 nonzero values. Storing every value as a four-byte number requires 400,000 bytes. Storing each nonzero as a four-byte index and four-byte value requires 800 bytes for those pairs, a 500-fold reduction before metadata and index overhead.
That idealized ratio is not a claim about a complete database. Record headers, indexing structures, alignment, and other data consume additional space. Sparse storage can also be less attractive when many coordinates are nonzero.

Illustrative case study

Fictional case: Willow Manuals builds search over equipment instructions. Exact model numbers and specialized terms are important, so the team evaluates a sparse representation alongside dense embeddings.
During testing, it discovers that a preprocessing change assigned different indices to some terms in new documents. Similarity scores no longer compare consistent features. The team versions the vocabulary and rebuilds affected vectors before judging retrieval quality.
It then measures relevance, storage, and response time on representative queries. The case illustrates why consistent representation matters; it does not report a real deployment or imply that sparse retrieval always outperforms dense retrieval.

Common misconceptions and mistakes

Sparse does not mean small dimension. A vector can have millions of possible coordinates and only a few active ones.
Zero is not automatically a missing observation. Its meaning follows the feature construction; treating unobserved information as numerical zero can require a separate modeling decision.
Sparse vectors are not always binary, nonnegative, or literal word counts. Conversely, storing a dense vector in an index-value format does not make its underlying values sparse. Compression format and mathematical sparsity are related but different concepts.

Business use

Sparse vectors are useful when a task has many possible features but relatively few active ones per object. They can support interpretable feature matching and efficient retrieval, especially when exact identifiers or domain vocabulary matter.
Evaluate representation quality and resource use together. More aggressive pruning may save memory while removing useful matches. Changes to tokenization, weighting, or feature indexing can require rebuilding stored representations. Keep these choices versioned so query vectors remain compatible with the collection they search.

Check your understanding

Why does coordinate 5 not affect the worked dot product?

Answer: Its query value is zero.

Why store indices with nonzero values?

Answer: Without indices, the system cannot know which features the values describe or match corresponding coordinates correctly.

Related terms

Sparsity
Bag of words
Inverted index
Dot product
Cosine similarity
dense vector
TF-IDF
sparse embedding
hybrid search

Learn next

Learn vector similarity and term weighting, then compare sparse storage with dense arrays on a small dataset. Next examine how vocabulary versioning, learned expansion, and pruning affect retrieval quality as well as memory use.

Sources

Primary definitions: Google Machine Learning Glossary, sparse vector and sparsity: https://developers.google.com/machine-learning/glossary#sparse_vector Primary implementation reference: Qdrant, Vectors, sparse-vector section: https://qdrant.tech/documentation/manage-data/vectors/

Previous: Sparse variational Gaussian process Next: Sparsity