AI Glossary / Statistics and probability / Advanced

Interviewer error

Definition

Interviewer error is a source of survey measurement error arising when an interviewer's behavior changes the respondent's answer. In the supplied NIST glossary, the term specifically covers responses to verbal statements, nonverbal cues, or other interviewer behavior. Broader discussions sometimes include recording mistakes, but the focus here is distortion introduced into the answer through the interaction itself.

What it means

The answer collected may depend partly on how a question is delivered, rather than only on the experience or attribute being measured. An approving reaction, a suggested answer, or an improvised explanation can change what a respondent reports.
A respondent may try to be agreeable, avoid embarrassment, or use the interviewer's explanation to interpret an ambiguous question. The resulting answer can be recorded faithfully yet still differ from the answer the intended measurement procedure would have obtained.

How it works

Start by separating the intended question from the way it is administered. Specify wording, reference periods, response categories, and acceptable clarification. Evaluate whether delivery changes the question's meaning or signals that one answer is preferred.
Pretesting can reveal confusing questions and difficult interactions. Behavior coding can document departures from the intended procedure, requests for clarification, or response problems. Such observations identify possible mechanisms; they do not automatically reveal the correct answer.
A well-designed comparison may vary the administration procedure while keeping respondent assignment comparable. When cases differ across interviewers, differences in answers may reflect case composition. Reinterviews also require care because genuine changes, recall, and the first interview itself can affect later answers.

Three practical examples

• An interviewer smiles and praises a positive evaluation of an AI assistant, then asks further evaluation questions. The reaction may encourage additional favorable responses.
• A question asks about usage during the previous seven days, but an interviewer explains it as usual usage. The response now concerns a different reference period.
• An interviewer suggests that most colleagues already use a tool before asking about the respondent's own usage. That statement may create pressure to report adoption.

Think of it this way

The interviewer is not merely a microphone. Interaction can alter the message. A perfect transcript preserves what was said, including any influence introduced while eliciting the answer.

Worked example

Consider a deliberately simplified teaching exercise with eight respondents. Assume their intended numerical answers under a specified neutral procedure are known: 2, 3, 4, 5, 6, 7, 8, and 9. Their sum is 44, so their mean is 5.5. Treat these as numerical quantities with meaningful equal units, not arbitrary category labels.
First suppose an encouraging prompt causes every respondent to report one unit more. The answers become 3, 4, 5, 6, 7, 8, 9, and 10. Their sum is 52 and their mean is 6.5.
Define each response error as the elicited answer minus its stipulated neutral reference. Every error is plus one. The mean signed error is one, the mean absolute error is one, and the mean squared error is one. The average is systematically shifted upward in this constructed case.
Now consider a different interaction pattern affecting the same reference answers. Suppose the first four responses increase by one and the last four decrease by one. The elicited answers become 3, 4, 5, 6, 5, 6, 7, and 8.
The sum is again 44 and the mean is again 5.5. The mean signed error is zero because four positive errors cancel four negative errors. However, the mean absolute error and mean squared error are still one. Correct agreement of the overall mean does not establish accurate individual answers.
The subgroup consequences are also different. The first four reference values average 3.5, and their elicited values average 4.5. The last four reference values average 7.5, and their elicited values average 6.5. The original subgroup gap of four units has been reduced to two units, even though the combined mean is unchanged.
Next imagine that a transcription system captures every elicited answer perfectly. Its transcription error is zero. That does not undo the interviewer influence that changed the answers before transcription. Improving recording accuracy addresses a different stage of the measurement process.
Finally, suppose the exercise instead involves two real interviewers receiving different respondents, and one obtains a mean of 6.5 while the other obtains 5.5. Those observed means alone cannot reproduce the causal conclusion from the teaching example. We would need evidence about assignment, respondent differences, the intended measurement, and the interaction mechanism.
The example stipulates the reference and response changes. In an actual survey, a neutral answer may be difficult to define or observe, and a disagreement between interview methods is not automatically proof that one is error-free.

Illustrative case study

Fictional company Cedar Feedback reviews interviews about an assistant's usefulness. Analysts notice that some interviewers offer examples of benefits before asking for an assessment, while others ask the question without those examples.
The team examines whether respondents understood the intended question, revises the guidance, and pilots a consistent introduction. It distinguishes evidence of changed delivery from evidence about the size or direction of any resulting error. It also avoids treating accurate automated transcripts as proof that the underlying answers were unaffected.

Common misconceptions and mistakes

Interviewer error is not limited to entering a wrong number into a form.
The absence of deliberate leading does not establish the absence of influence.
An unchanged overall average can hide distorted individual or subgroup responses.
Strict wording alone does not guarantee shared understanding. The questionnaire and permitted clarification procedures also need evaluation.
A difference between interviewers can reflect different respondent populations rather than an interviewer-induced change.

Business use

Include the interaction in survey quality assurance. Review question delivery, neutral clarification, and respondent understanding alongside recording accuracy. Preserve enough information about administration and case assignment to investigate differences. If AI assists with prompting or transcription, evaluate those functions separately: a tool that records answers accurately can still accompany a problematic prompting process.

People also ask

Can nonverbal behavior matter?

Yes, when it changes the respondent's answer.

Does error always shift the overall mean?

No; errors can cancel in the aggregate.

Does a perfect transcript eliminate interviewer error?

No; it preserves the answer actually elicited.

Check your understanding

What is the first pattern's mean signed error?

Answer: Plus one unit.

What is the second pattern's mean absolute error?

Answer: One unit despite zero mean signed error.

What happens to the subgroup gap?

Answer: It falls from four units to two.

Related terms

• Interviewer bias
• Response Bias
• Measurement error
• social desirability
• behavior coding
• questionnaire pretesting
• transcription error
• mode effects

Learn next

Study response-process evaluation, standardized and conversational interviewing, and designs that distinguish interviewer effects from differences in assigned respondents.

More in Statistics and probability

• Interval predictor model - An interval predictor model, or IPM, maps an input to an interval of predicted outputs rather than a single number.
• Interval scale - An interval scale is a measurement scale on which equal numerical differences represent equal differences in the measured attribute.
• Interview survey - An interview survey collects answers through verbal interaction with an interviewer, who records the responses.
• Inverse Dirichlet distribution - The inverse Dirichlet distribution, in the matrix-variate sense used here, is obtained by inverting each matrix in a type-I matrix Dirichlet collection.
• Inverse sampling - Inverse sampling is a sampling design in which observations continue until a specified condition based on the observed results is met.
• Irregular component of a time series - The irregular component of a time series is the remainder after estimated trend-cycle, seasonal, and relevant calendar effects have been removed.

Sources

NIST statistical glossary, Interviewer error: https://www.itl.nist.gov/div898/handbook/glossary.htm . Hunter and Landreth, Person-Based Data Collection in Practice: https://www.census.gov/content/dam/Census/library/working-papers/2009/adrm/ssm2009-12.pdf . U.S. Census Bureau, questionnaire evaluation: https://www.census.gov/about/policies/quality/standards/appendixa2.html . Response patterns and company scenario are original illustrations.

Previous: Interviewer bias Next: Intonation phrases