AI Glossary / Governance, ethics and safety / Beginner

Automation bias

Definition

Automation bias is a tendency to give automated recommendations undue weight when making decisions, including accepting an incorrect recommendation despite other available evidence. It concerns human interaction with automation, not simply a statistical error inside a model.

What it means

An assistant may confidently classify an invoice as a duplicate while the underlying records show two different deliveries. Automation bias arises when the person reviewing the case treats the system's recommendation as a substitute for examining those records.

This does not mean that trusting software is always unreasonable. Reliable tools often improve decisions. The problem is reliance that is poorly matched to the tool's actual evidence, limitations and situation. Rejecting every recommendation would create different errors and discard useful assistance.

The term applies beyond modern generative AI. Earlier research studied automated decision aids in tasks such as aviation. A chatbot's polished explanation adds a contemporary setting, but the central issue remains how people allocate attention and judge conflicting information.

How it works

A user sees a recommendation, interprets its apparent authority and decides how much independent checking to perform. The interface, workload, familiarity and organisational expectations can all shape that decision. A system presented as definitive may discourage consideration of contrary information.

Researchers distinguish commission errors, in which a person takes an inappropriate action following an automated cue, from omission errors, in which needed action is missed when an aid supplies no warning. These categories describe different ways reliance can go wrong; neither implies that every human mistake was caused by automation.

To evaluate the issue, observe the combined human-and-tool workflow. Include cases where the tool is correct, confidently wrong or silent despite a relevant problem. Record whether reviewers identify the evidence and respond appropriately. A model's standalone accuracy and a count of approval clicks cannot establish effective oversight.

Three practical examples

• An accounts reviewer accepts a duplicate-invoice alert without checking distinct delivery references visible in the source records.
• A travel coordinator ignores a published closure because an automated itinerary still routes visitors through that entrance.
• A content editor assumes an article needs no review because a screening tool issued no warning, despite an obvious unsupported quotation.

Think of it this way

Think of following a navigation device toward a road that is visibly closed. The route suggestion is useful evidence about a journey, but the physical barrier supplies information that the suggested route has failed to incorporate.

The analogy is limited: many AI decisions have no obvious equivalent of a road barrier, and the reviewer may need specialist knowledge or access to records. Meaningful oversight must make relevant contradictory evidence available.

Worked example

Measure responses when the aid is wrong

In an invented review exercise, an automated invoice checker makes recommendations on 100 cases. Independent adjudication establishes that 90 recommendations are correct and 10 are wrong. Reviewers accept 88 correct recommendations and eight wrong ones, while rejecting two correct and two wrong ones.

The reviewer approval rate is 96%, but that is not a quality score. Final decisions are correct on 88 accepted correct recommendations plus two corrected wrong recommendations: 90 of 100. The team has not improved overall accuracy in this exercise, despite putting a person into every case.

The eight accepted errors deserve investigation. They could reflect poor source visibility, misunderstanding or undue reliance; the count alone does not prove a psychological cause. Comparing review behavior with evidence access and suitable unassisted or differently presented conditions can help identify what changed.

Illustrative case study

Fictional Willow Accounts introduces AI-assisted invoice review. Supervisors initially celebrate fast approvals. During a training exercise, reviewers approve a deliberately incorrect duplicate label even though the delivery references differ.

Interviews reveal that the interface hides those references behind an extra screen, and staff believe overrides will be treated as mistakes. The team places the relevant evidence beside the recommendation, clarifies override responsibility and tests both incorrect alerts and missed duplicates.

It evaluates the resulting decisions rather than assuming that a new warning label solved the problem. This case illustrates that automation bias can be encouraged by workflow design and incentives. Telling employees to be careful without changing those conditions leaves the underlying difficulty unresolved.

Common misconceptions and mistakes

• A human approval step guarantees independent judgment. Approval can be perfunctory or uninformed.
• High model accuracy makes checking unnecessary. Rare errors can still matter, and accuracy may differ in unfamiliar situations.
• Disagreement proves the human is right. Human judgment also needs evidence.
• Automation bias and algorithmic bias are identical. One concerns undue human reliance; the other can concern systematic patterns in a system's outputs.

Business use

Businesses can use this concept to assess whether a proposed human review stage has a realistic function. Reviewers need time, relevant information, competence and authority to challenge the output. They also need a clear understanding of what they are expected to verify.

Design evaluations around final task outcomes and the consequences of errors, not only acceptance rates or review speed. Where useful, let a reviewer form an initial assessment before seeing the recommendation and test whether that improves this particular workflow. Such design choices need evaluation; no single interface technique eliminates automation bias in every task.

People also ask

Is automation bias limited to AI?

No. It also occurs with conventional automated decision aids.

Does it mean people should trust their intuition instead?

No. Decisions should integrate appropriate evidence, recognizing limitations in both human and automated judgment.

Is it the same as complacency?

They overlap, but complacency emphasizes insufficient monitoring, whereas automation bias also includes favoring an incorrect automated recommendation.

Check your understanding

A reviewer accepts an automated rejection even though the displayed source record satisfies the stated requirement. Is the presence of a reviewer enough to demonstrate useful oversight?

Answer: No. The review must use the relevant evidence and have the ability to correct the recommendation. Investigate why the contradiction was overlooked.

Related terms

• Confirmation bias
• Human oversight
• automation complacency
• decision support
• human factors
• calibrated trust

Learn next

Read human oversight and automation complacency. For one AI-assisted decision, list the evidence a reviewer can access, the errors they must catch and what happens when they disagree with the tool.

More in Governance, ethics and safety

• Attack - An attack is a deliberate attempt to compromise a system, its information or its operation.
• Audit log - An audit log is a record of selected system events and activities that provides evidence for later review.
• Authenticity - Authenticity is the property of being genuine with respect to a stated origin or identity.
• Automation Complacency Bias - Automation complacency bias concerns over-reliance on automated systems that weakens active checking and can leave people less prepared to perform or...
• Autonomy - Autonomy in an AI system is its ability to select and carry out actions toward a goal without a person directing every step, within a defined scope.
• Availability - Availability is the property of information or a service being accessible and usable by authorized users when needed, within an acceptable time.

Sources

Google, Machine Learning Glossary, automation bias: https://developers.google.com/machine-learning/glossary/#automation-bias. NASA, Automation Bias and Countermeasures in Flight Crews: https://ntrs.nasa.gov/citations/20020043049. NASA, Automation: Decision Aid or Decision Maker?: https://ntrs.nasa.gov/api/citations/19980048379/downloads/19980048379.pdf. Numerical and business scenarios are original teaching examples, not reported study results.

Previous: Automation Next: Automation Complacency Bias