Is a design effect of two the same as doubling the standard error?

No. It doubles variance and multiplies standard error by the square root of two.

Can the design effect be below one?

Yes. Some designs improve precision relative to the specified simple-random-sampling benchmark.

Does effective sample size tell us how many records to keep?

No. It is a precision comparison, not an instruction to discard observations or ignore the original design.

AI Glossary / Statistics and probability / Intermediate

Design effect

Definition

A design effect is the ratio of an estimator's variance under a specified sampling design to its variance under a reference simple random sampling design of the same size. It describes how features such as clustering, stratification and weighting change precision. A design effect above one indicates greater variance than the reference; a value below one indicates greater precision. It is specific to the estimator, outcome, population and reference definition.

What it means

Counting observations alone does not tell you how much information a survey contains. People sampled within the same organization may resemble one another, unequal weights can concentrate influence and stratification can improve coverage of important differences. The design effect compares the resulting uncertainty with a clearly defined simpler benchmark. It is not a measure of bias or representativeness. A survey can have a low design effect yet still miss an important population group or measure the wrong outcome.

How it works

Estimate the variance using methods appropriate to the actual sample design, such as Taylor linearization or suitable replicate weights. Calculate the comparable simple-random-sampling variance for the same estimand and sample size, including the relevant finite-population convention. Divide the first variance by the second. The corresponding standard-error ratio is the square root of the design effect, sometimes called the design factor. A rough effective sample size is n divided by the design effect when the reference variance scales approximately as one over n. That approximation does not replace the design information needed for valid inference.

Three practical examples

Adoption surveys: assess how sampling employees within selected workplaces affects precision of an AI-use estimate.
Survey planning: inflate a reference sample-size calculation using defensible assumptions about clustering and weighting.
Results reporting: explain why a weighted complex survey has a different standard error from a naive calculation treating every record as an independent equal-weight observation.

Think of it this way

Think of collecting information from many people who share the same immediate experience. Their responses still matter, but some information overlaps. A differently structured sample may spread the same number of interviews across more varied settings. The design effect expresses the precision difference for a particular estimate, rather than judging which respondents are more valuable.

Worked example

Suppose a hypothetical survey of 900 people estimates an adoption proportion near 10%, with a negligible sampling fraction. Its design-appropriate variance estimate is 0.000225, while the stated simple-random-sampling reference variance is 0.0001. The design effect is 0.000225/0.0001 = 2.25. Standard errors are 0.015 and 0.01, giving a ratio of 1.5, the square root of 2.25. Under the usual inverse-sample-size approximation, effective sample size is 900/2.25 = 400. Thus 900 actual respondents provide approximately the reference precision associated with 400 simple-random-sample observations for this estimate. They have not become 400 people, and that effective count need not apply to another outcome or subgroup. Ignoring the design would understate this estimated standard error by using 0.01 instead of 0.015.

Illustrative case study

Fictional case: Cedar Insights surveys employees at selected branches about assistant adoption. A first dashboard reports narrow uncertainty intervals using a simple independent-response formula. Survey analysts incorporate branch clustering and the supplied weights, then calculate outcome-specific design effects against a documented reference. Some estimates lose precision, while stratification helps others. The team reports actual sample sizes, the design-based uncertainty and limitations in coverage separately. This invented case illustrates appropriate variance estimation, not a measured result for a real workforce.

Common misconceptions and mistakes

The design effect multiplies variance, not the standard error directly. It is not always greater than one: well-chosen stratification can improve precision relative to the reference. The familiar cluster formula 1 + (m - 1)rho assumes a particular equal-cluster-size correlation setting and is not a universal formula for every complex survey. Similarly, a weight-only approximation based on weight variability does not capture every interaction between weights and outcomes. A single overall design effect can mislead when subgroup estimates have different sampling structures. Precision adjustment does not correct selection bias by itself.

Business use

Businesses commissioning surveys should request the sample-design information needed for analysis rather than only a respondent count and a spreadsheet of answers. During planning, examine plausible design effects instead of presenting one assumed value as certain. After collection, report uncertainty using the actual design and clarify the reference when an effective sample size is shown. For AI-related surveys, different behaviors and attitudes may cluster differently across workplaces, so the design effect for one measure should not automatically be copied to every other measure.

Check your understanding

Why is the effective sample size of 400 in the example only a qualified summary?

Answer: It uses the approximation that reference variance scales inversely with sample size. It pertains to the particular estimator and design comparison, and does not preserve all information about clustering, weighting, degrees of freedom or other outcomes.

Related terms

Stratification
Cluster Sampling
Effective sample size
Standard error
Complex survey
sampling variance
survey weight
intracluster correlation
replicate weights
finite-population correction

Learn next

distinguish a variance ratio from a standard-error ratio. Compare clustering, stratification and unequal weighting in small survey examples, then study the variance estimator used for the actual design. Examine why outcome-specific and subgroup-specific design effects can differ before using a single inflation factor in planning.

Sources

CDC/NCHS, NHANES Tutorials: Variance Estimation: https://wwwn.cdc.gov/nchs/nhanes/tutorials/varianceestimation.aspx CDC/NCHS, Vital and Health Statistics, Series 2, No. 92: https://www.cdc.gov/nchs/data/series/sr_02/sr02_092.pdf CDC, Population Survey or Descriptive Study sample-size guidance: https://www.cdc.gov/epiinfo/user-guide/statcalc/samplesize.html

Previous: Descriptive (or summary) statistics Next: Design of experiments