Does more data always move the observed average closer to the truth?

No. A particular running average can move away temporarily even when convergence holds.

Does the law require finite variance?

Not for the standard independent, identically distributed finite-absolute-mean version, although finite variance supports a simple proof and useful bounds.

Can dependent data obey a law of large numbers?

Yes, under suitable additional conditions. Independence is a common sufficient setting, not a universal necessity.

AI Glossary / Mathematics and optimization / Intermediate

Law of large numbers

Definition

The law of large numbers describes how an average of many observations approaches an expected value under specified assumptions. A standard version applies to independent, identically distributed observations with finite absolute expectation. It explains long-run stability of averages, but it does not guarantee that every new observation makes an average more accurate.

What it means

Individual outcomes can remain unpredictable while their average becomes more stable. If independent trials share a fixed success probability, the fraction of successes tends toward that probability. The statement concerns the average or fraction, not a requirement that the absolute difference between observed and expected counts must shrink.
The assumptions matter as much as the conclusion. Repeating one recorded observation thousands of times does not create thousands of independent observations. Nor does collecting more data from a systematically unrepresentative source necessarily reveal the average of the population that a business actually cares about.
There are several laws of large numbers with different conditions. The familiar independent, identically distributed case is an accessible starting point, not the only possible setting in which averages can converge.

How it works

Write the sample average as X̄ₙ=(X₁+…+Xₙ)/n and the common expected value as μ. The weak law says that, for every fixed positive tolerance ε, the probability that |X̄ₙ−μ| exceeds ε tends to zero as n grows. The strong law describes almost-sure convergence of the sequence of averages.
For independent observations with common finite variance σ², the sample average has variance σ²/n. Chebyshev's inequality then gives P(|X̄ₙ−μ|≥ε)≤σ²/(nε²), a simple finite-sample bound supporting the weak law. This bound can be conservative.
Finite variance is useful for this particular argument, but the standard independent, identically distributed strong law only requires finite absolute expectation. Convergence itself does not supply a universal sample size that is adequate for every distribution and every tolerance.

Three practical examples

A simulation can average many independent runs to estimate an expected outcome, while separately assessing sampling uncertainty.
A quality-analysis exercise can estimate a stable event rate from appropriately collected observations under a justified sampling model.
A model evaluator can average losses on independently sampled test examples to estimate the expected loss of a fixed model under that test distribution.

Think of it this way

Think of averaging repeated readings that contain random fluctuations around a stable level. Individual readings remain variable, but their average can settle. The analogy requires care: a miscalibrated instrument can settle around the wrong level, and correlated readings may contain less new information than their count suggests.

Worked example

Consider an original mathematical model in which each independently inspected item has defect probability p=0.2. Let Xᵢ be 1 for a defect and 0 otherwise. Its mean is 0.2 and its variance is p(1−p)=0.16. The sample defect fraction has variance 0.16/n.
With n=400, the standard deviation of that fraction is √(0.16/400)=0.02. With n=1,600, it is 0.01. Multiplying the sample size by four halves this standard deviation under the stated model.
For an absolute tolerance of 0.05, Chebyshev's bound at n=400 is 0.16/(400×0.05²)=0.16. Thus the probability of an error of at least five percentage points is at most 16%. At n=1,600, the bound is 0.04, or 4%. These are upper bounds, not exact binomial tail probabilities or guarantees about one realized sample.
Now compare counts. The defect count's standard deviation is √(0.16n): it is 8 at n=400 and 16 at n=1,600. The fraction becomes more stable even though the count's typical absolute fluctuation grows. Finally, observing a run of nondefective items does not make the next independent item's defect probability exceed 0.2. The model contains no mechanism that compensates for an earlier streak.

Illustrative case study

In a fictional delivery-analytics team, an analyst collects many parcel records to estimate an average delay. A review reveals that most records came from one depot during a quiet week. Adding more records from that same setting makes the local estimate more precise without establishing that it represents all depots and seasons.
The team therefore revisits its sampling plan and reports which population the available average describes. It also checks whether multiple parcels share disruptions that create dependence. This hypothetical example shows why the law supports reasoning within an appropriate data-generating model rather than replacing the work of defining that model.

Common misconceptions and mistakes

Do not use the law to justify the gambler's fallacy: independent outcomes do not owe a correction after a streak. Do not assume convergence is monotonic or that a large sample removes systematic bias.
Distinguish the law of large numbers from the central limit theorem. The former concerns convergence of averages; the latter, under its own assumptions, describes the distribution of suitably scaled fluctuations. Also, choosing a model after repeatedly inspecting the same evaluation data requires additional reasoning; a law for the loss of one fixed model does not automatically validate unrestricted adaptive model selection.

Business use

For business measurement, state the target population, time period, observation unit, and plausible dependencies. Explain what additional data will improve: often it reduces random sampling variation, while measurement bias and changing conditions require different remedies.
Use a finite-sample uncertainty calculation appropriate to the actual design when deciding whether an estimate is precise enough. A general appeal to large numbers does not establish that a particular dataset is sufficient. In operational monitoring, yesterday's stable distribution may also cease to describe today's process after a policy, product, or customer change.

Check your understanding

Under an independent fixed-probability Bernoulli model, sample size increases from 100 to 900. By what factor does the standard deviation of the sample proportion change?

Answer: It becomes one-third as large, because it scales as 1/√n.

Related terms

Expected value
Sample mean
Independence
Central limit theorem
Sampling bias
identical distribution
convergence in probability
almost-sure convergence
Chebyshev's inequality

Learn next

Next, compare the variability of a sample sum with that of its average. Then study confidence intervals and concentration inequalities to connect long-run convergence with concrete uncertainty statements for finite datasets.

Sources

Primary teaching sources: Philip B. Stark, The Long Run and the Expected Value, original Berkeley material: https://www.stat.berkeley.edu/~stark/SticiGui/Text/expectation.htm . Amir Dembo, Probability Theory: STAT310/MATH230, strong-law section: https://stanford.edu/~montanar/TEACHING/Stat310A/lnotes.pdf . The defect-rate calculations are original.

Previous: Latin square design Next: Layer