AI Glossary / Statistics and probability / Advanced

Interviewer bias

Definition

Interviewer bias is systematic distortion in survey responses or recorded information caused by the interviewer's actions. It can arise through leading prompts, inconsistent treatment of respondents, unsuitable substitutions, or a recurring tendency to record answers incorrectly.

What it means

An interviewer participates in the measurement process. A respondent may change an answer in response to the interviewer's wording or reactions, and an interviewer may record a different meaning from the one expressed. These mechanisms can shift results in a particular direction without deliberate deception.
Bias should be distinguished from interviewer-related variability. Different interviewers can introduce different errors, increasing variation in estimates. Some errors may cancel in an aggregate while still distorting subgroup findings.

How it works

Consider the chain from selecting and contacting the intended respondent through asking, clarifying, interpreting, and recording. At each stage, identify behavior that could change who answers or what becomes data.
Use consistent questions, neutral explanations, appropriate respondent-selection procedures, and interviewer training. Pretesting and examination of interviewer-respondent interactions can reveal problems before or during collection.
When investigating interviewer differences, examine how cases were allocated. An interviewer assigned mainly to one location, shift, or respondent type can have different results even with identical interviewing behavior. Random allocation of comparable cases can help separate interviewer effects from case composition, although missing responses and other design features still require attention.

Three practical examples

• An interviewer introduces an AI tool as highly successful before asking whether it helped. The introduction may encourage favorable answers rather than measure an unprompted assessment.
• A survey taker repeatedly records uncertain answers as yes. The resulting dataset can overstate the proportion supporting a proposal.
• An interviewer substitutes an available colleague for the selected employee. If substitutes systematically differ from intended respondents, the resulting figures can be distorted through the interviewer's selection practice.

Think of it this way

The interviewer can tilt the measuring instrument while taking a reading. A consistent tilt can move results in one direction. But different readings from two instruments do not prove a tilt unless the things being measured are also considered.

Worked example

Start with an invented audit in which the correct yes-or-no classifications are known from an independent, accurate reference. Two interviewers each handle one hundred cases. Each set contains fifty true yes cases and fifty true no cases.
Interviewer A correctly records all fifty yes cases but mistakenly records ten no cases as yes. A therefore records sixty yes responses and forty no responses, producing an observed yes proportion of 60 percent instead of the reference 50 percent.
Interviewer B records every case correctly, producing fifty yes responses and fifty no responses. Combining both interviewers gives 110 recorded yes responses out of two hundred, or 55 percent. The reference proportion is one hundred out of two hundred, or 50 percent. The aggregate distortion is five percentage points.
If a future workload used A's same error mechanism for all cases with the same reference composition, the expected observed proportion would be 60 percent. If it used only B, it would be 50 percent. The mix of interviewers can therefore affect the resulting estimate. The specified mechanism explains this distortion.
Now consider a completely separate example with no recording or response errors. There are two case types: H has an 80 percent true yes rate, and L has a 20 percent true yes rate. Interviewer C receives eighty H cases and twenty L cases. Interviewer D receives twenty H cases and eighty L cases.
Suppose the observed counts exactly match those rates. C records sixty-four plus four = sixty-eight yes answers out of one hundred, or 68 percent. D records sixteen plus sixteen = thirty-two yes answers, or 32 percent.
The interviewer difference is thirty-six percentage points, larger than in the first example. Yet there is no interviewer distortion in this construction. Within H, both interviewers record 80 percent yes; within L, both record 20 percent. The aggregate difference comes entirely from the allocation of case types.
If both sets were standardized to half H and half L, each would have an illustrative standardized yes rate of 0.5 times 0.8 plus 0.5 times 0.2 = 50 percent. This calculation exposes the composition difference because the relevant case types are known here. In a real study, unmeasured differences may remain after adjustment.
For a final caution, imagine two equally sized groups with true mean scores of fifty. One interviewer's recording process adds five points to every answer, while the other's subtracts five. The pooled mean remains fifty, but the group means become fifty-five and forty-five. Cancellation at the overall level hides a ten-point artificial subgroup difference.
These examples deliberately specify reference values and mechanisms. Ordinary survey data rarely reveal the truth so conveniently; observed interviewer differences should prompt investigation rather than automatic blame.

Illustrative case study

Fictional company Maple Interviews sees unusually favorable ratings in interviews conducted by one team. Before attributing the pattern to interviewer bias, analysts check case assignments and discover that the team mainly interviewed experienced users.
They also review permitted interview records and identify inconsistent introductory wording. The company corrects the wording and plans a comparison using balanced case allocation. It keeps the composition finding separate from the evidence about interview delivery.

Common misconceptions and mistakes

Interviewer bias need not be intentional.
A difference between interviewer averages does not, by itself, identify an interviewer effect.
A large sample does not remove a systematic recording rule that shifts answers.
Training can reduce risks, but completion statistics alone do not demonstrate accurate measurement.
Removing interviewer identifiers can make the dataset look simpler while preventing useful quality assessment.

Business use

Preserve appropriate interviewer identifiers, assignment information, and quality-control records. Investigate unusual patterns alongside respondent composition and collection conditions. Use findings to improve procedures and evaluate changes, rather than rewarding interviewers solely for completion volume or favorable answers.

People also ask

Is this the same as hiring discrimination?

No; this entry concerns survey interviewing.

Can the overall mean look correct despite interviewer problems?

Yes, if distortions cancel.

Does random case allocation guarantee error-free interviews?

No; it helps assessment but does not eliminate measurement errors.

Check your understanding

What is the aggregate distortion in the first example?

Answer: Five percentage points.

Why do C and D differ without bias?

Answer: They receive different mixtures of H and L cases.

What does the offset example hide?

Answer: An artificial ten-point subgroup difference.

Related terms

• Interviewer error
• Measurement error
• Response Bias
• Nonresponse
• interviewer variance
• case allocation
• behavior coding
• survey quality

Learn next

Study interpenetrated interviewer assignments, response-validation studies, neutral probing, and models separating interviewer and respondent characteristics.

More in Statistics and probability

• Interval predictor model - An interval predictor model, or IPM, maps an input to an interval of predicted outputs rather than a single number.
• Interval scale - An interval scale is a measurement scale on which equal numerical differences represent equal differences in the measured attribute.
• Interview survey - An interview survey collects answers through verbal interaction with an interviewer, who records the responses.
• Inverse Dirichlet distribution - The inverse Dirichlet distribution, in the matrix-variate sense used here, is obtained by inverting each matrix in a type-I matrix Dirichlet collection.
• Inverse sampling - Inverse sampling is a sampling design in which observations continue until a specified condition based on the observed results is met.
• Irregular component of a time series - The irregular component of a time series is the remainder after estimated trend-cycle, seasonal, and relevant calendar effects have been removed.

Sources

OECD Glossary of Statistical Terms, Interviewer bias: https://www.oecd.org/content/dam/oecd/en/publications/reports/2008/09/oecd-glossary-of-statistical-terms_g1gh9ad7/9789264055087-en.pdf . Statistics Canada, interviewer variance and case assignment: https://www150.statcan.gc.ca/n1/pub/12-001-x/2022001/article/00005/01-eng.htm . U.S. Census Bureau, questionnaire evaluation: https://www.census.gov/about/policies/quality/standards/appendixa2.html . Audit examples and company scenario are original illustrations.

Previous: Interview survey Next: Interviewer error