Are heuristics always harmful?

No. Mental shortcuts can be useful and adaptive; their limitations depend on the task and environment.

Can an AI model have cognitive biases?

Its outputs may resemble named human patterns, but that does not establish the same mental mechanism. Distinguish observed behavior from claims about cognition.

Is cognitive bias the same as statistical bias?

No. Statistical bias concerns properties of an estimation procedure; cognitive bias concerns judgment and decision processes.

AI Glossary / Governance, ethics and safety / Beginner

Cognitive Bias

Definition

A cognitive bias is a systematic pattern in judgment or decision-making that can lead people away from an appropriate evaluation of evidence. Such patterns can involve attention, memory, interpretation or mental shortcuts, and they can affect how AI systems are designed, evaluated and used.

What it means

People cannot examine every possible piece of information before every decision. Shortcuts help them act efficiently, but some conditions make those shortcuts misleading. A vivid incident may feel more common than it is, or an initial number may influence later estimates more than the evidence warrants.

Cognitive bias is not a synonym for low intelligence, bad character or every mistake. A single incorrect decision can result from missing information, unclear instructions or ordinary uncertainty. Identifying a systematic bias requires more than disagreeing with someone's conclusion.

In AI work, the issue includes the humans around the model: people choose targets, label examples, interpret evaluation results and decide whether to follow recommendations. Software does not remove those judgment points simply by performing part of the calculation.

How it works

Different biases operate through different processes. Confirmation bias involves favoring information that fits existing beliefs. Availability bias involves overweighting readily recalled examples. Anchoring involves insufficient adjustment from an initial reference point. Automation bias involves undue reliance on automated advice.

These patterns can interact with workflow design. A prominent AI recommendation can become the initial reference point for a reviewer; a selectively chosen demonstration can shape what a team remembers; a preferred project outcome can influence which test failures receive attention.

Investigation should connect an observed pattern to evidence about the decision process. Compare conditions, inspect what information was available and consider alternative explanations. Improving the process can involve independent assessment, explicit criteria and review of contrary evidence. Awareness training may help, but knowing the names of biases does not automatically prevent them.

Three practical examples

A model-development team searches repeatedly for tests that favor its preferred method while giving less attention to failures on representative tasks.
A reviewer begins with an AI-generated estimate and makes only a small adjustment despite strong evidence supporting a different value.
A product manager judges typical system reliability from one memorable demonstration instead of examining the wider evaluation record.

Think of it this way

Think of looking through a window that directs attention toward one part of a scene. The visible part can be real, yet it may not provide the complete view needed for the decision.

The analogy does not imply that every person sees through the same distortion or that an outside observer is automatically objective. It highlights the need to inspect how evidence is selected and interpreted, including one's own process.

Worked example

A selected subset changes the conclusion

In an invented model comparison, a team evaluates two assistants on 100 predefined tasks. Model A succeeds on 82 and model B on 78. The project lead prefers B and presents a subset of 20 tasks where B succeeds on 18 and A on 12.

Within that subset, B's success rate is 90% and A's is 60%. Those calculations are correct, but they do not describe performance across the original task set, where A scores 82% and B 78%.

The selective presentation is a warning sign for confirmation-driven judgment, although the numbers alone do not establish the lead's mental process. Perhaps the subset represents a newly justified business priority. If so, that change should be explicit and independently assessed rather than quietly replacing the original evaluation after results are known.

Illustrative case study

Fictional Elm Analytics selects a support model after an enthusiastic pilot. During wider testing, team members explain away failures as unusual while treating successes as proof that the model is ready. A reviewer notices that the acceptance criteria were never written down.

The team defines representative tasks and failure conditions before the next comparison, includes people who did not build the prototype and documents evidence against its preferred option. It also examines whether time pressure and incentives are discouraging criticism.

The company does not diagnose individuals as irrational or assume the new procedure eliminates all bias. The case illustrates how a structured decision process can make selective reasoning more visible and give competing evidence a fairer hearing.

Common misconceptions and mistakes

Cognitive bias means deliberate dishonesty. It often operates without conscious intent.
Experts are immune. Expertise does not remove every judgment bias.
Any disagreement demonstrates bias in the other person. The evidence and decision process need examination.
AI eliminates cognitive bias. Human choices remain throughout the system lifecycle, and AI presentation can influence them.
Learning a list of biases guarantees better decisions. Procedures, incentives and information access also matter.

Business use

Businesses can use the concept to improve decisions rather than assign blame. Set evaluation criteria before reviewing preferred results, preserve contrary evidence and distinguish independent observations from repeated versions of the same story. Give reviewers the authority and time to question an AI output.

For consequential choices, document the alternatives considered and what evidence would change the decision. Review whether the interface encourages premature acceptance or hides uncertainty. The aim is not to remove all intuition, but to support judgment with an evidence process appropriate to the stakes and the limits of human attention.

Check your understanding

A team changes its evaluation criteria only after seeing that its preferred model loses. Does the new score alone justify the revised decision?

Answer: No. The changed objective needs a clear rationale and suitable independent assessment. Otherwise selective interpretation may be driving the conclusion.

Related terms

Confirmation bias
Automation bias
Statistical bias
anchoring
availability heuristic
human factors
decision-making

Learn next

Read confirmation bias and anchoring. Review one recent AI project decision and identify the original criteria, contrary evidence and alternatives considered. Ask which process change would make a future decision easier to challenge constructively.

Confirmation bias

Sources

NIST SP 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence: https://doi.org/10.6028/NIST.SP.1270. Artificial Intelligence Ontology, Cognitive Bias: https://raw.githubusercontent.com/berkeleybop/artificial-intelligence-ontology/main/aio.owl. These support the concept and AI lifecycle context; model-comparison figures and the analytics case are original illustrations.

Previous: Cognitive automation Next: Cognitive computing