Is an empirical distribution always discrete?

A finite ordinary empirical distribution is discrete, even when measurements came from a continuous population.

Is it the same as a histogram?

A histogram groups observations into bins; the empirical distribution retains their individual values and frequencies.

Can it represent categories?

Yes. A numerical ordering is not required for the distribution itself.

AI Glossary / Statistics and probability / Intermediate

Empirical distribution

Definition

An empirical distribution is the probability distribution that assigns equal mass to each observation in a nonempty, unweighted dataset. Repeated values receive the combined mass of their occurrences. It represents the sample's observed frequencies without imposing a parametric family. It can describe numbers, categories, or complete multivariate records and is a basic building block of empirical averages, model training, and resampling.

What it means

Selecting one recorded observation uniformly at random is equivalent to drawing from the empirical distribution. If a category occurs in four of ten records, its empirical probability is four-tenths. The distribution describes what was recorded; it does not declare that unobserved outcomes are impossible in the population.
For numerical data, the empirical cumulative distribution function is one way to represent this distribution. For categories without a natural order, a frequency table is often more suitable. For paired or multivariate observations, the empirical distribution can preserve the combinations appearing in complete records.

How it works

Let x1 through xn denote the observations, allowing repeated values. For an event A, define P_n(A) as the number of observations belonging to A divided by n. Equivalently, P_n is the average of n point masses, one at each recorded observation.
An expectation under P_n is a sample average: E_Pn[g(X)] = sum g(x_i)/n. Choosing g(x) = x gives the sample mean for numeric data. Choosing a loss function gives the average empirical loss used in many machine-learning objectives. A small empirical loss is a training-data result and does not by itself establish good performance on new cases.
The ordinary empirical variance is the average squared deviation from the empirical mean, using denominator n. The familiar estimator with denominator n - 1 serves a different purpose: unbiased estimation of population variance under standard independent-sampling assumptions. Both quantities are useful, but their labels and denominators should be explicit.
Sampling with replacement from the observed records generates draws from the empirical distribution. This is the basis of the ordinary nonparametric bootstrap. Resampling complete paired records preserves their observed associations; independently resampling separate columns generally changes the joint distribution.
Weights produce a weighted empirical distribution when nonnegative weights are normalized to sum to one. Dependence or complex sampling can require specialized resampling and inference. Even without such complications, empirical probabilities have sampling uncertainty, especially for rare outcomes and small samples.

Three practical examples

A classification team describes observed class proportions and uses them to inspect imbalance before training.
An analyst computes average prediction loss as an expectation under the evaluation dataset's empirical distribution.
A bootstrap procedure repeatedly samples customer records with replacement to study the variability of a chosen statistic under an appropriate sampling model.

Think of it this way

Think of a bag containing one card for every recorded observation. Drawing a card uniformly reproduces the dataset's empirical distribution. Two cards carrying the same value make that value twice as likely as a value printed on only one card. Removing duplicate-valued cards changes the distribution.

Worked example

An original sample records five counts: 0, 0, 1, 2, and 2. Its empirical distribution assigns probability 2/5 to zero, 1/5 to one, and 2/5 to two. The probabilities sum to one.
Its mean is 0(2/5) + 1(1/5) + 2(2/5) = 1. Its variance is (0 - 1) squared(2/5) + (1 - 1) squared(1/5) + (2 - 1) squared(2/5) = 4/5 = 0.8.
The usual unbiased sample-variance estimator would divide the total squared deviation, four, by 5 - 1, giving one. That does not make the empirical variance wrong; the calculations answer different questions.
Suppose five independent draws are now taken from this empirical distribution with replacement. The chance that at least one draw equals two is 1 - (3/5) to the fifth power = 0.92224. This is a conditional resampling probability for the observed empirical distribution. It is not a proven probability for five new real-world observations from an unknown population.

Illustrative case study

Fictional case: Willow Returns stores one record per returned item and builds a simple simulation by resampling those records. A developer removes repeated values of return cost to make the dataset smaller.
The analyst notices that common costs now receive the same probability as rare costs. The team restores occurrence counts and either samples the full records or samples distinct values with probabilities proportional to their frequencies. It also keeps item type and cost together when joint behavior matters. The simulation becomes faithful to the observed dataset, while its limits for predicting future conditions remain documented.

Common misconceptions and mistakes

Mistake: giving every distinct value equal probability after discarding duplicates. Equal mass belongs to observations, not necessarily unique values.
Mistake: confusing empirical distribution with a fitted density. A finite empirical distribution places mass on observed points even when the underlying population is continuous.
Mistake: assuming unobserved events have zero population probability. Absence from the sample is not impossibility.
Mistake: treating bootstrap resamples as newly collected independent evidence. They reuse the original information.
Mistake: separating paired columns during resampling when their relationship is part of the question.

Business use

Businesses use empirical distributions to build transparent baselines, simulate observed workloads, calculate expected losses, and summarize customer or operational data. Their simplicity makes assumptions visible: the simulation repeats the observed pattern unless an additional model changes it. Document the observation unit, sampling period, weights, exclusions, and whether records are dependent. A larger historical dataset can still misrepresent future conditions if the process has changed.

Check your understanding

What happens if the two zeros and two twos in the example are collapsed into one occurrence each and treated equally?

Answer: The probabilities become one-third each, which changes the empirical distribution.

Why is bootstrap sampling done with replacement?

Answer: Each draw should again follow the same empirical distribution, allowing observations to appear repeatedly.

Related terms

Empirical cumulative distribution function
Probability mass function (pmf)
Sample mean
Empirical risk
Bootstrap
probability mass function
weighted sample
joint distribution
and generalization

Learn next

calculate expectations under a frequency table, distinguish empirical variance from unbiased variance estimation, and compare resampling complete records with resampling their columns independently.

Variance estimation

Sources

Sources: Aalto Dictionary of Machine Learning, empirical distribution, empirical risk, and bootstrap entries: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . SciPy official bootstrap documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html . The frequency, variance, and resampling calculations are original examples.

Previous: Empirical cumulative distribution function Next: Empirical dynamic modeling