Must strata be the same size?

No. Population strata often differ greatly in size, and sample allocation can be proportional or deliberately unequal.

Does stratification guarantee representativeness?

No. It helps with the characteristics used to define strata, but coverage errors, nonresponse, poor within-stratum selection, and other problems can remain.

Can I stratify on several characteristics?

Yes, but many combined categories can become sparse. The design should balance useful coverage with feasible sample sizes and the needs of the analysis.

AI Glossary / Generative AI and language / Intermediate

Stratified sampling

Definition

Stratified sampling divides a population or dataset into distinct groups, called strata, and selects samples separately within those groups. The groups are defined using relevant characteristics, such as request type, language, region, or class label. This can improve coverage of important categories and, with a suitable design, the precision of estimates. It does not automatically remove every source of bias or make every possible subgroup adequately represented.

What it means

A simple random sample may contain few examples from a rare but important category. Stratification allows the sampling plan to specify how much evidence to collect from each defined group.

The allocation can be proportional to group size, or it can deliberately give smaller groups more representation. If the goal is a population-level estimate, unequal sampling fractions generally require appropriate weighting when results are combined.

Stratification is broader than language AI. In machine learning it also appears in train-test splitting, often to preserve label proportions. That use should be distinguished from designing a survey or deliberately enriching an evaluation set with rare cases.

How it works

Define the target population and a sampling frame that identifies eligible units. Assign each unit to a stratum using a clear rule, with strata that do not overlap and together cover the intended population.

Choose the number of units to sample from each stratum and use the specified random-selection procedure within it. Record population sizes, sample sizes, and selection probabilities as required by the analysis.

Evaluate or measure the selected units. Report within-stratum results when useful, and combine them using weights appropriate to the target quantity. Check whether the design also needs to account for clusters, repeated observations, or time order. Preserving one set of proportions does not make those dependencies disappear.

Three practical examples

1. A language-assistant evaluation samples separately from common questions, document questions, and rare workflow requests so each receives deliberate coverage.

2. A classifier evaluation uses a stratified split to maintain approximate class proportions in training and test partitions when that design matches the task.

3. A service survey samples customers within defined regions and uses the survey design when producing an overall estimate, rather than simply averaging regional samples with unequal selection fractions.

Think of it this way

Think of inspecting items from every labeled shelf in a warehouse instead of drawing only from the building as a whole. The shelf labels help ensure that an important small section is not missed by chance.

The analogy has limits. The labels must correspond to the question being studied, and the way items are chosen within shelves still matters. Inspecting every shelf does not reveal items absent from the warehouse list or guarantee that the inspected items represent each shelf fairly.

Worked example

Suppose an evaluation population contains 800 routine requests and 200 complex requests. A proportional sample of 100 would select 80 routine and 20 complex requests. Alternatively, a team might select 50 from each group to study complex requests more closely.

In the equal-allocation sample, assume the system succeeds on 45 of 50 routine requests and 30 of 50 complex requests. The stratum success estimates are 90% and 60%.

The unweighted sample success rate is 75/100 = 75%. To estimate success for the stated population mix, weight the stratum estimates by 0.8 and 0.2: 0.8 × 0.90 + 0.2 × 0.60 = 0.84, or 84%. This calculation assumes appropriate random sampling and valid group sizes; it is a point estimate, not a complete uncertainty analysis.

Illustrative case study

Fictional case: Birch Support evaluates an assistant on requests from three product lines. The smallest product line has distinctive terminology, so the team assigns it more evaluation cases than proportional sampling would provide.

Reviewers report results for each product line and use the known traffic mix for a separate overall estimate. They clearly label the deliberately enriched sample so readers do not confuse its raw average with expected performance across all traffic.

The design supports both targeted investigation and population reporting, provided the sampling and weighting assumptions are respected. The team also checks rare failures within each product line instead of assuming the first layer of stratification covers every important distinction.

Common misconceptions and mistakes

Stratified sampling does not necessarily mean equal sample sizes in every group. Proportional and disproportionate allocations serve different purposes.

It is not the same as cluster sampling. Stratification samples within the defined strata, while cluster designs select groups of units under a different structure.

Another mistake is averaging an intentionally balanced sample as though it had the population's natural proportions. That can answer a different question from the one intended. Finally, stratifying by class label does not automatically prevent leakage between related records or make random splitting suitable for time-dependent prediction.

Business use

Stratification can make an evaluation budget more informative by ensuring attention to meaningful categories. It is useful when an overall average might hide weak performance on a smaller but important part of the workload.

Decide whether the purpose is estimating typical performance, comparing subgroup behavior, or finding difficult cases. Those goals can require different allocations and reporting. Document the strata, sampling units, selection process, and weights. Keep overall and subgroup results visible together so neither a favorable average nor an unusually difficult test mix is misinterpreted.

Check your understanding

An evaluation samples equal numbers of routine and complex requests even though routine requests are 80% of traffic. Is its raw average automatically the traffic-wide success estimate?

Answer: No. It describes the sampled mix. A population estimate should account for the population proportions and the actual sampling design.

Related terms

Cluster Sampling
Selection bias
Stratum
sampling frame
probability sampling
proportional allocation
sampling weight
class imbalance
stratified split
subgroup evaluation

Learn next

Divide a small evaluation population into meaningful, nonoverlapping categories and record their sizes. Compare proportional and equal allocations under the same total budget. Calculate both raw and population-weighted averages from hypothetical subgroup results, then state which business question each average answers.

Sources

Sources: Statistics Canada, Probability sampling, https://www150.statcan.gc.ca/n1/edu/power-pouvoir/ch13/prob/5214899-eng.htm — primary sampling-design explanation. KNIME Data Science Glossary, https://www.knime.com/blog/data-science-glossary — supplied stratified-sampling terminology. Scikit-learn cross-validation guide, https://scikit-learn.org/stable/modules/cross_validation.html — primary machine-learning split context. Relevant sections were read. The two-group allocation and weighting calculation are original illustrations.

Previous: Stratified resampling Next: Stream processing