Is a sparse vector shorter?

Its mathematical dimension remains the same; a compact storage representation may require fewer stored values.

Does sparsity imply low rank?

No. These are different properties. An identity matrix is sparse for large dimensions but has full rank.

Can activations be sparse while weights are dense?

Yes. They are different arrays and can have different zero patterns.

Is there a universal threshold for calling data sparse?

No. The useful threshold depends on the representation and intended computation.

AI Glossary / Governance, ethics and safety / Advanced

Sparsity

Definition

Sparsity describes how much of a numerical representation consists of zero entries. A common measure is the number of zeros divided by the total number of entries. A high zero fraction indicates sparsity. In AI, sparsity can describe input features, model weights or intermediate activations, so a useful claim states exactly which representation is being measured.

What it means

A document may use only a small fraction of the words in a vocabulary. If each vocabulary word has a position in a count vector, most positions will be zero. Those zeros mean that the corresponding words did not occur under the chosen representation; they are not necessarily missing observations.
Some uses of the term also include empty or null entries. The convention must be stated because an unknown value and a known zero can mean different things. Replacing missing values with zero without justification can change the problem.
Sparsity is a property of the representation. The same underlying information can have a sparse encoding or a dense embedding, with different computational and modeling implications.

How it works

For a representation with N entries and k nonzero entries, numerical sparsity is (N - k) / N. Density, under this convention, is k / N. Values close to zero are still nonzero unless a thresholding rule explicitly treats them otherwise.
Sparse storage formats avoid allocating a value slot for every implicit zero. They retain the values that are stored and information identifying their positions. Compressed sparse row storage, for example, uses values, column indices and row pointers. This bookkeeping also consumes memory.
Algorithms must support the chosen format to gain practical benefits. Some operations can make a sparse representation dense. Subtracting a nonzero feature mean from every entry turns implicit zeros into nonzero values, which is why centering needs care in sparse preprocessing. Similarly, setting model weights to zero does not automatically accelerate software that still performs dense computations.

Three practical examples

Text classification: A word-count vector has positions for a large vocabulary, but each short message activates only a small subset. Sparse storage can represent the absent counts compactly.
Categorical data: A one-hot representation of one selected category among many contains one active position. A category index is a compact encoding, but it should not automatically be interpreted as an ordered numerical measurement.
Model compression: A pruning process sets selected weights to zero. The team measures both the zero fraction and actual memory or runtime changes, because those benefits depend on storage and execution support.

Think of it this way

A seating plan can list every seat and mark most as empty, or record only the occupied seats and their locations. The second approach can be compact when few seats are occupied. However, the locations take space too, and an unknown booking status should not be mistaken for a confirmed empty seat.

Worked example

Consider a 100-by-100 matrix containing exactly 200 nonzero entries. It has 10,000 entries in total and 9,800 zeros, so its sparsity is 9,800 / 10,000 = 98%. Its density is 2%.
Assume each numerical value uses four bytes. A dense representation's value array requires 10,000 x 4 = 40,000 bytes.
For an illustrative compressed sparse row representation, assume four-byte values and four-byte integer indices. The 200 values require 800 bytes, the 200 column indices require another 800 bytes, and 101 row pointers require 404 bytes. The total is 2,004 bytes, about 5.01% of the dense value-array size.
This comparison excludes object overhead and other metadata, assumes no explicitly stored zeros or duplicate entries, and describes these particular data types. It does not predict a 20-fold speed improvement. Runtime depends on the operation, memory access, software and hardware, while actual allocation may differ from this simplified payload calculation.

Illustrative case study

Fictional company Birch Messages develops a classifier for short support requests. Its vocabulary representation is mostly zero, so the team stores word counts in a sparse format. Memory use remains manageable during initial preparation.
A later preprocessing step attempts to center every feature. The team realizes that subtracting a feature's nonzero mean changes its implicit zeros into nonzero values. The representation can become dense and lose the expected storage advantage.
The developers choose preprocessing compatible with the intended representation and verify both model quality and resource use. This fictional case illustrates why sparsity needs to be considered across the processing pipeline, rather than measured once at the input and assumed to persist.

Common misconceptions and mistakes

“Sparse means missing.” A zero count can be fully observed, while a missing value is unknown.
“Every small value is zero.” Exact and threshold-based sparsity are different measurements and should not be mixed.
“More sparsity always means better performance.” Important information may be lost through pruning, and indexing overhead can outweigh savings for insufficiently sparse data.
“A sparse model must run faster.” The execution path must exploit the pattern of zeros; unsupported or irregular sparsity may provide little benefit.
“Sparse storage always omits every zero.” Some libraries permit explicitly stored zeros, so stored-entry counts can differ from true nonzero counts.

Business use

Sparsity is useful when estimating resource needs for large feature spaces or compressed models. Specify the object being measured, the counting convention and the expected operations. Benchmark realistic data sizes and deployment conditions instead of deriving latency from the zero fraction alone.
If sparsity is introduced deliberately, compare accuracy and relevant error patterns before and after the change. Check whether preprocessing or format conversions create dense intermediate arrays. Resource savings matter only if the complete workflow remains useful and reliable.

Check your understanding

What is the density of the worked matrix?

Answer: Two percent.

Why count row pointers in the storage estimate?

Answer: Sparse formats need structural information in addition to numerical values.

Why can centering destroy sparsity?

Answer: An originally zero entry becomes minus the feature mean when that mean is nonzero.

Related terms

Learn next

Compare common sparse storage formats and their supported operations. Then distinguish numerical sparsity, low rank and missingness using small matrices before interpreting resource-saving claims.

Sources

Google Machine Learning Glossary, sparsity and sparse features. https://developers.google.com/machine-learning/glossary#sparsity SciPy, compressed sparse row array documentation. https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_array.html Scikit-learn, StandardScaler, sparse centering limitations. https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html The storage calculation and Birch Messages case are original teaching illustrations.

Previous: Sparse vector Next: Sparsity-of-effects principle