Does a narrow confidence interval mean the survey is accurate?

It may indicate small sampling uncertainty while other errors remain.

Can non-sampling errors be random?

Yes; the category includes more than systematic bias.

Does a full population collection solve every problem?

No; measurement and processing issues can remain.

AI Glossary / Statistics and probability / Intermediate

Survey errors

Definition

Survey errors are discrepancies or uncertainties that can arise when a survey's results differ from the quantities it is intended to measure. They include sampling error and non-sampling errors such as coverage, nonresponse, measurement, and processing problems. Assessing a survey requires attention to the whole collection and estimation process, not only its reported margin of error.

What it means

A survey can be precisely calculated yet misleading. Its sample may miss part of the population, some selected people may not respond, questions may be misunderstood, or data may be coded incorrectly. These issues affect different parts of the evidence chain.
Error is not always a deliberate mistake or a known numerical difference. Some errors are systematic, some contribute random variability, and many cannot be measured exactly from the completed responses alone. Quality assessment therefore combines statistical estimation with design review and validation.

How it works

Define the intended population and target quantity first. An estimate of current customers' usage cannot be evaluated against an unrelated group or period without recognizing that the targets differ. Clear definitions prevent conceptual mismatches from being hidden behind numerical precision.
Sampling error comes from observing a sample rather than every population unit. Under a probability design, analysts can estimate the variability of an estimator across possible samples using design information and observed data.
Coverage error arises when the frame does not align with the target population. Missing eligible units, duplicates, and ineligible entries can change who has a chance to be represented. A larger sample from an incomplete frame does not automatically repair the omissions.
Nonresponse error can arise when nonrespondents differ from respondents in ways relevant to the estimate. Distinguish unit nonresponse from unanswered individual items. Response rate alone does not determine the size or direction of resulting bias.
Measurement error concerns differences between recorded answers and the intended quantity. Ambiguous questions, recall difficulties, interviewer effects, response pressures, or unsuitable answer categories can contribute. More respondents do not necessarily average away a systematic wording problem.
Processing error can enter through coding, data entry, linkage, editing, or computation. Automated workflows can repeat an error consistently across many records, so reproducibility should be paired with checks on the intended logic.
Estimation choices introduce further assumptions. Weighting, imputation, and modeling may reduce some problems but can also add variability or bias if poorly justified. Document which information supports each adjustment and test sensitivity where practical.
Evaluate overall quality through targeted checks: frame comparisons, questionnaire testing, recontact or validation studies, missingness analysis, and independent calculation review. A conventional sampling confidence interval does not automatically cover every error component. For an estimator with a defined target, mean squared error combines variance and squared bias, but estimating all those components can be difficult.

Three practical examples

Product surveys: a pop-up shown only to active users may omit customers who stopped using the product.
Employee research: ambiguous questions about AI use can mix experimentation with routine deployment.
Customer reporting: a coding reversal can turn satisfied responses into dissatisfied ones even when sampling was well designed.

Think of it this way

Think of a survey result passing through a chain: define, select, ask, record, process, and estimate. A strong link does not repair a weak one elsewhere. A carefully calculated margin of error addresses only part of that chain.

Worked example

Consider an original simplified population with 600 customers in group A and 400 in group B. Suppose their true group means for monthly usage are 480 and 200 minutes. The population mean is (600 times 480 + 400 times 200)/1,000 = 368 minutes.
Imagine the achieved respondents include 300 from A and 100 from B, and their within-group means exactly match the population means. The unweighted respondent mean is (300 times 480 + 100 times 200)/400 = 410 minutes, which is 42 minutes above the target.
Using known population shares gives 0.60 times 480 + 0.40 times 200 = 368. This removes the composition discrepancy in this deliberately simplified example because the group means among respondents are stipulated to be correct.
In a real survey, respondents might differ from nonrespondents even within each group. Then weighting only by group would not necessarily remove bias. The calculation illustrates one mechanism, not a guarantee that calibration fixes nonresponse or other survey errors.

Illustrative case study

Fictional case: Birch Research reports high satisfaction with an AI service and a narrow sampling interval. A quality review finds that the invitation list excludes recently closed accounts and that one question combines speed with accuracy. Analysts identify these as separate coverage and measurement concerns. They revise the frame and questionnaire, then assess the new collection. The organization reports what the sampling interval covers and avoids describing it as a complete measure of survey accuracy.

Common misconceptions and mistakes

Do not treat every survey discrepancy as sampling error. A census can contain non-sampling errors. More completed responses can make a biased result look more precise. Nonresponse rate is not the same quantity as nonresponse bias. Weighting cannot recover an unmeasured difference automatically. Agreement between repeated calculations does not establish that the data or target definition were correct.

Business use

Use a survey-error framework to prioritize quality work before results enter decisions. Address likely sources of large error, not only those easiest to count. Publish methodological limitations beside key findings and retain enough documentation to trace changes in questions, coverage, processing, and estimation across reporting periods.

Check your understanding

Why is the respondent mean 410 rather than 368 in the example?

Answer: Group A is overrepresented among respondents.

Why does weighting work exactly there?

Answer: Respondent group means are assumed equal to the true group means.

Would the same adjustment always eliminate real-world nonresponse bias?

Answer: No.

Related terms

Sampling error
Coverage error
Measurement error
Mean Squared Error
Total survey error
nonresponse bias
processing error
weighting

Learn next

Study total survey error and questionnaire validation. Then examine how nonresponse adjustments depend on auxiliary information and why uncertainty from multiple error sources is harder to quantify than sampling variance alone.

Sources

Australian Bureau of Statistics, Types of error: https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/types-error Australian Bureau of Statistics, Errors in Statistical Data: https://www.abs.gov.au/websitedbs/d3310114.nsf/home/Basic%20Survey%20Design%20-%20Errors%20in%20Statistical%20Data The population-composition example is original.

Previous: Survey Next: Survey propagation