Is a pilot just a smaller main survey?

Not necessarily; it is designed around testing and improvement objectives.

Must every pilot use the same sample size?

No; the size and coverage depend on what needs to be learned.

Can pilot data be included later?

Sometimes, but only when design, comparability, and the intended analysis justify it.

AI Glossary / Statistics and probability / Intermediate

Pilot survey

Definition

A pilot survey is a preliminary survey conducted to test how a proposed survey works before full implementation. It can examine the questionnaire, recruitment, sampling procedures, data collection, processing, and fieldwork organization. Its principal purpose is to discover practical and measurement problems and improve the main survey, rather than automatically provide definitive population estimates.

What it means

A pilot is a rehearsal with evidence collection built into it. It asks whether respondents understand the questions, whether the intended people can be reached, whether procedures function together, and whether the resulting data can be processed as planned.
A questionnaire pretest may focus closely on wording or comprehension. A pilot survey can go further by exercising the wider operational process. The exact scope should be stated because organizations use the term for tests of different sizes and ambitions.

How it works

Start with explicit pilot objectives and decisions. Identify uncertainties that could undermine the main survey, such as ambiguous eligibility rules, confusing response options, excessive interview time, technical failures, or difficulty recruiting a key subgroup.
Select participants and settings that can reveal those problems. A pilot may use probability sampling when population-level operational estimates are needed, or purposive coverage of important situations when the main goal is detecting design weaknesses. The sampling choice limits what the findings can support.
Test the intended collection mode, language versions, invitation process, reminders, and support arrangements. Include realistic devices and accessibility conditions where relevant. A questionnaire that works during a supervised demonstration may fail in ordinary use.
Record process information as well as answers. Track invitations, eligibility, starts, completions, item omissions, time taken, help requests, routing errors, and reasons for abandonment. Define denominators clearly so that recruitment and completion measures are not confused.
Exercise data validation, coding, transfer, and analysis steps. Confirm that variable names, response values, skip patterns, and exported files support the planned analysis. If an AI system codes open responses, compare its outputs with a suitably reviewed sample rather than assuming successful file processing means valid classification.
Combine quantitative indicators with respondent and interviewer feedback. A small number of misunderstood questions can matter even when a headline completion percentage looks acceptable. Repeated issues should lead to specific revisions and, when necessary, another test.
Document what changed after the pilot and whether the changed instrument remains comparable. Decide in advance, where possible, whether pilot responses can enter the main dataset. Major wording, recruitment, or procedure changes may make direct pooling inappropriate.
Translate findings into an updated operational plan with uncertainty. Pilot estimates of yield, timing, and cost can be unstable, and main-study conditions may differ. Set realistic contingencies instead of treating every observed percentage as a guaranteed future rate.

Three practical examples

AI adoption research: a team tests whether business respondents interpret technology-use categories consistently.
Customer feedback: analysts verify mobile survey routing and identify questions that trigger abandonment.
Workforce research: investigators rehearse recruitment, multilingual administration, and data processing before a larger employee survey.

Think of it this way

Think of a dress rehearsal that tests the script, the stage equipment, and the handoffs between people. A smooth rehearsal is useful evidence, but it does not guarantee that a larger performance under different conditions will run identically.

Worked example

Suppose a pilot sends 120 invitations to known eligible people. Seventy-two start the questionnaire, and 54 submit complete responses. The start proportion is 72/120 = 60%. The completion proportion among starters is 54/72 = 75%. The completed-response yield per invitation is 54/120 = 45%.
These denominators answer different operational questions. The 75% figure should not be reported as the completed-response yield among everyone invited.
Assume the 54 completers average 18 minutes each. Their combined recorded completion time is 54 × 18 = 972 minutes, or 16.2 hours. This excludes time spent by noncompleters, recruitment staff, and support personnel.
If the main survey aims for 600 complete responses and the same 45% yield held, the simple planning calculation would be 600/0.45 = 1333.33, rounded up to 1334 invitations. That is a conditional planning estimate, not a promise. A different sample, revised questionnaire, or recruitment setting can change the yield, so the plan should include uncertainty and monitoring.

Illustrative case study

Fictional case: Acorn Business Research pilots an AI-use questionnaire with organizations of different sizes. Feedback reveals that respondents disagree about whether embedded software features count as AI. The team clarifies the definition, tests the revised wording, and updates its fieldwork plan. It reports the pilot's operational findings separately from the main survey's eventual population estimates.

Common misconceptions and mistakes

Do not assume a small pilot proves questionnaire validity or estimates all rare problems. The absence of an observed issue is not proof that it cannot occur.
Avoid treating convenience-pilot percentages as representative population statistics. Do not choose the pilot solely from colleagues who already understand the project.
A pilot should test the process realistically; quietly providing extra assistance can conceal difficulties the main survey will face.
Do not pool revised and original responses without considering comparability. Maintain a record of version changes and collection conditions.

Business use

Pilot surveys can prevent expensive large-scale collection errors, improve respondent experience, and produce more credible resource plans. Their value comes from actionable findings and documented revisions. Management should judge a pilot by what it reveals and improves, not by whether every initial design choice appears successful.

Check your understanding

Why are 60%, 75%, and 45% all correct in the example?

Answer: They describe starts per invitation, completions per start, and completions per invitation.

Why is 1334 only a planning estimate?

Answer: It assumes the pilot yield carries over to the main survey.

Related terms

Nonresponse
Data Quality
questionnaire pretest
cognitive interviewing
feasibility study
response rate
sampling frame
fieldwork
survey design

Learn next

Learn next how cognitive interviews reveal interpretation problems, how survey outcome-rate denominators are defined, and how pilot findings become explicit revisions and contingency plans.

Sources

Sources: Statistics Canada, Data Quality Guidelines, questionnaire development and testing, https://www150.statcan.gc.ca/n1/pub/12-539-x/steps-etapes/4147792-eng.htm ; OECD, Glossary of Statistical Terms, https://www.oecd.org/en/publications/oecd-glossary-of-statistical-terms_9789264055087-en.html . The official sources describe pilot testing of the survey process and statistical terminology. All recruitment figures, calculations, and the organizational case are original illustrations.

Previous: Pilot job Next: Pinball loss