Can an analytic survey also estimate an overall average?

Yes, if its design and weighting support that estimate, although optimizing subgroup comparisons can affect overall precision.

Does it require equal sample sizes across groups?

No. Allocation depends on the comparisons, variability, costs and precision requirements. Equal allocation is only one possible design.

Can it establish causation?

An observational survey usually requires additional assumptions or a stronger design for causal claims. Comparing groups alone does not establish why they differ.

AI Glossary / Statistics and probability / Intermediate

Analytic survey

Definition

An analytic survey is a sample survey designed primarily to compare sectors or subgroups within the sampled population. Its design emphasizes the information needed for those comparisons, such as adequate subgroup sample sizes and comparable measurements, rather than focusing only on a precise overall population total or average.

What it means

The source glossary uses "analytic" to describe the survey's main design purpose. It does not mean merely that someone will analyze the results, since nearly every survey is analyzed. The distinction concerns what the survey is built to estimate and compare.

A survey intended to compare a small subgroup with a large one may need to sample the smaller group disproportionately. Otherwise, a large total sample can still leave too few observations in the subgroup of interest. Such choices affect weighting, precision and the kinds of population summaries that can be produced reliably.

How it works

Define the target population and the subgroup comparisons before selecting the sample. Specify the outcomes, meaningful differences and precision or power requirements for those comparisons. Check that the sampling frame identifies the relevant subgroups or supports an appropriate screening approach.

Choose a sampling design and allocation that provide enough information for the priority comparisons. Stratification and oversampling can help, but their benefits depend on subgroup prevalence, costs and design effects. Use consistent questions and collection procedures so differences do not merely reflect different measurement methods.

During analysis, account for selection probabilities, clustering, stratification and relevant adjustments for nonresponse. Produce subgroup estimates and contrasts with design-appropriate uncertainty. Population-wide estimates may require weighting that reverses disproportionate allocation. Plan how multiple comparisons will be handled and distinguish prespecified comparisons from exploratory findings. An analytic purpose does not make an observational survey an experiment or remove confounding among subgroup characteristics.

Three practical examples

1. A customer survey deliberately recruits enough users of a small service channel to compare their experiences with those using a larger channel.

2. A workforce survey allocates sample sizes to support comparisons across occupations, rather than allowing the largest occupation to dominate the sample.

3. A public survey designs questions and sampling around regional comparisons, evaluating whether each region has sufficient coverage and comparable measurement.

Think of it this way

Think of arranging a debate in which each group must have enough speakers to be heard. A sample proportional to population size may give a small group too few voices for a useful comparison. Giving that group more sample space helps the comparison, but an overall population summary must still respect the groups' actual sizes.

Worked example

Consider an original planning example with two population groups: A contains 90% of people and B contains 10%. A total sample of 800 allocated proportionally gives 720 observations in A and 80 in B. Assume independent simple random samples within large groups, negligible sampling fractions and a planning proportion of 0.5 in each group.

The approximate standard error of the difference in proportions is the square root of 0.25/720 + 0.25/80, or about 0.0589. Allocating 400 observations to each group instead gives the square root of 0.25/400 + 0.25/400, or about 0.0354. Under these assumptions, equal allocation improves precision for this particular subgroup difference.

Suppose the equal-allocation survey then estimates proportions of 0.60 for A and 0.40 for B. Their simple average is 0.50, but the population-weighted estimate is 0.9 times 0.60 + 0.1 times 0.40 = 0.58. The comparison-focused allocation and the overall population estimate require different weighting perspectives. These calculations do not include nonresponse, clustering or other real-design complications.

Illustrative case study

Fictional case study: Comparewell wants to compare experiences across two customer segments, one much smaller than the other. Its first proposal uses a large overall sample but provides little information about the small segment. Survey specialists revise the allocation to support the planned comparison.

The team reports segment estimates with uncertainty and uses appropriate population weights for the overall result. It tests whether the questionnaire is understood similarly in both segments and investigates differential response rates. Exploratory comparisons added later are labeled accordingly.

This fictional case illustrates a survey designed around analytic priorities. It does not establish that oversampling alone removes selection bias or guarantees valid interpretation of every observed segment difference.

Common misconceptions and mistakes

An analytic survey is not defined by using sophisticated software or regression after collection. The defining feature is its comparison-oriented design. A very large sample can still be inadequate for a rare subgroup, especially after missing data or clustering reduce effective information.

Oversampling does not mean the subgroup should count disproportionately in a population-wide estimate. Conversely, applying weights does not automatically fix poor coverage or nonresponse. Also, a subgroup difference is not necessarily an effect of subgroup membership; related characteristics and selection processes can explain part or all of the association.

Business use

Analytic surveys help organizations obtain evidence for comparisons central to a decision, such as whether experiences differ across regions, channels or customer types. They encourage planning around the actual questions rather than a single headline sample size.

Specify the priority contrasts, target population and reporting levels before fieldwork. Budget for enough observations in key groups, appropriate measurement testing and follow-up of nonresponse. Publish both the sampling approach and the weighting rationale. For AI-related surveys, define experience and usage categories carefully so comparisons reflect meaningful differences rather than inconsistent interpretations of broad labels such as "AI user." Protect small groups from disclosure when detailed comparisons are released.

Check your understanding

A rare subgroup receives half the sample even though it represents one-tenth of the population. Should the unweighted sample mean automatically be reported as the population mean?

Answer: No. The unequal selection probabilities must be accounted for using an appropriate estimator and the survey design.

Related terms

Subgroup analysis
Stratified sampling
Oversampling
survey design
domain estimation
survey weights
design effect and nonresponse

Learn next

compare proportional and disproportionate sample allocations for a planned subgroup contrast. Calculate how each affects subgroup precision and an overall estimate, then add realistic costs and expected response rates.

Sources

Primary sources: OECD, OECD Glossary of Statistical Terms, Analytic survey: https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf

Kalton, Methods for oversampling rare subpopulations in social surveys, Statistics Canada: https://www150.statcan.gc.ca/n1/en/catalogue/12-001-X200900211036

Statistics Canada, Questions to ask about surveys: https://www.statcan.gc.ca/en/customized-products-services/custom-surveys/questions-to-ask

Previous: Analysis of variance Next: Analytics