AI Glossary / Governance, ethics and safety / Intermediate

Non-response bias

Definition

Non-response bias occurs when people who provide data differ in relevant ways from those invited or eligible to respond who do not. The resulting observations can give a distorted picture of the intended population because participation is related to the quantity being studied.

What it means

Suppose an organization asks customers whether an AI service solved their problem. People who remain engaged with the service may be more likely to answer than those who abandoned it. The reported satisfaction rate then describes respondents unless further evidence supports a broader inference.

A low response rate raises questions, but it does not by itself prove large bias. The important issue is how nonrespondents differ in relation to the study's purpose. A high response rate can still leave a consequential group underrepresented.

Non-response differs from coverage problems that prevent people from entering the invitation frame at all. It also differs from an inaccurate answer supplied by someone who does respond. These mechanisms can occur together and require different checks.

How it works

Define the eligible population, the invitation process and what counts as a response. Track participation using the appropriate denominator and distinguish complete responses from missing answers to particular questions.

Investigate whether response patterns vary across relevant observed characteristics and, where feasible, use appropriate follow-up to learn about nonrespondents. Such comparisons provide clues, but similarity on a few recorded attributes does not prove similarity on the outcome of interest.

Weighting or other adjustments can help under stated assumptions and with suitable supporting information. They do not automatically recover the missing answers. Design improvements may include clearer invitations, accessible formats and reasonable follow-up. Report the response process and remaining uncertainty rather than treating silence as satisfaction, dissatisfaction or agreement.

Three practical examples

• A product team checks whether users who abandoned an AI onboarding flow are missing from its satisfaction survey.
• A workplace study offers suitable response routes for employees whose schedules make a live feedback session difficult.
• An evaluation distinguishes people who skipped a question about trust from those who explicitly selected a low-trust answer.

Think of it this way

Think of asking for opinions at a meeting after many participants have left. The remaining voices may be informative, but they do not automatically speak for those who departed.

The analogy illustrates selective participation. It does not establish why anyone left or what they would have said; those missing views require evidence rather than assumptions.

Worked example

Different response patterns change the apparent rate

In an invented population of 1,000 customers, 600 are satisfied and 400 are dissatisfied. Suppose 300 satisfied customers respond to a survey, while only 100 dissatisfied customers respond.

The respondents' satisfaction rate is 300 / 400 = 75%. The full population's satisfaction rate, known only because this teaching example supplies it, is 600 / 1,000 = 60%. The survey result is 15 percentage points higher.

The overall response rate is 40%, but that number alone does not reveal the direction or size of the bias. The distortion comes from the different response rates: 50% among satisfied customers and 25% among dissatisfied customers. In practice, the unknown views of nonrespondents cannot simply be filled in using these invented proportions.

Illustrative case study

Fictional company Maple Assist surveys customers through a message shown after they complete a successful interaction. The team initially describes the results as the views of everyone who attempted to use the assistant.

A review finds both a coverage issue and a response issue. People who never reached completion were not invited, and some invited users did not answer. The company separates these stages, designs a broader feedback approach and reports each denominator clearly.

The resulting analysis avoids treating all missing observations as one interchangeable category. Improving the response rate among completers would not, by itself, supply evidence about users excluded before the invitation.

Common misconceptions and mistakes

• A low response rate proves a particular amount of bias. The relevant differences between respondents and nonrespondents determine the effect.
• A large number of responses eliminates the issue. Many responses can still come from a systematically selective group.
• Nonresponse means a negative answer. Silence does not establish the missing person's view.
• Weighting guarantees representativeness. Adjustments rely on data and assumptions that should be examined.

Business use

Non-response review is important when businesses use surveys or voluntary feedback to assess AI quality, trust or satisfaction. Those most affected by a problem may be difficult to reach, while highly enthusiastic or frustrated participants may also be unusually motivated to respond.

Record who was eligible, who was invited and who answered. Investigate meaningful participation differences before converting respondent percentages into claims about all customers or staff. Combine thoughtful collection design with transparent reporting. The objective is to understand the missing evidence, not merely maximize a response count.

People also ask

Is non-response bias the same as participation bias?

The terms often overlap; the precise usage should be stated in the study.

Can it affect individual questions?

Yes. Item nonresponse can distort analyses even when someone completes most of a survey.

Does random selection prevent it?

Randomly inviting participants does not ensure that the people who respond remain representative.

Check your understanding

A random sample receives a survey, but participation is much higher among satisfied users. Does random invitation guarantee an unbiased satisfaction estimate?

Answer: No. Selective response can alter the observed sample after invitation. The response mechanism and its relevance to satisfaction need investigation.

Related terms

• Selection bias
• Coverage bias
• Missing data
• response rate
• survey weighting
• representativeness

Learn next

Read coverage bias and missing data. Draw the stages from eligible population to invitation, response and completed questions. At each stage, identify whose information can disappear and how that might affect the conclusion.

• Coverage bias
• Missing data

More in Governance, ethics and safety

• Model risk management - Model risk management is the process of identifying, assessing, reducing and monitoring risks arising from models and their use.
• Model Selection Bias - Model selection bias is distortion introduced when a model is chosen using data and the chosen result is interpreted without accounting for that selection.
• Multi-level security - Multi-level security is an approach to processing information with different security classifications and categories while allowing authorized users...
• Nondiscrimination - Nondiscrimination is the principle that people should not face unjustified disadvantage or exclusion on grounds that should not determine their treatment.
• Offline learning - Offline learning is training a model from an already collected dataset rather than continually updating it as new operational examples arrive.
• Opacity - Opacity is difficulty seeing or understanding how an AI system operates, reaches an output or affects a decision.

Sources

Google Machine Learning Glossary, selection bias and non-response bias: https://developers.google.com/machine-learning/glossary#selection_bias. The glossary identifies different participation rates as a source of selection problems. The population calculations and fictional survey review here are original illustrations, not estimates from a real customer study.

Previous: Non-relational database Next: Non-sampling error