AI Glossary / Governance, ethics and safety / Intermediate

Safety

Definition

Safety is the condition and practice of preventing unacceptable harm from a system's operation, failures, or interactions with its surroundings. For AI, it requires examining how outputs and actions can affect people, property, and the environment, then designing, evaluating, and operating the complete system to control those hazards.

What it means

An AI component does not become safe simply because its predictions are usually accurate. A rare error may have severe consequences, and an accurate prediction can still trigger an inappropriate action. The surrounding workflow, permissions, equipment, operators, and recovery options shape the outcome.
Safety is also context dependent. A mistake in a fictional game and the same type of mistake in an industrial control process have different implications. Claims should name the intended use and conditions, the harms considered, and the supporting evidence. No general conversational assurance establishes safety for a new deployment.

How it works

Identify the system's purpose, boundaries, users, and affected parties. Describe hazards: conditions that could contribute to harm. Examine how those conditions might arise from model errors, unsuitable objectives, flawed interfaces, timing, data problems, unexpected use, or interacting components.
Design controls that prevent dangerous states where possible and limit consequences when prevention fails. Validate assumptions and test both ordinary operation and relevant failures. Define intervention authority, monitoring, incident response, and conditions for restricting or stopping use. Revisit the analysis when the model, data, integration, or operating environment changes. The analysis must connect evidence to the actual system rather than to a model name alone.

Three practical examples

• A warehouse robot uses independently designed protective measures and operating limits around people, rather than relying solely on its visual classifier.
• An AI maintenance assistant labels uncertain suggestions and routes consequential actions through the established engineering approval process.
• A scheduling system is tested for dangerous combinations of individually acceptable recommendations, because system-level interactions can create hazards that isolated predictions miss.

Think of it this way

Think of safety as designing the entire journey, including vehicle, route, driver, signals, and emergency response. A reliable engine helps but cannot settle every question. The analogy is bounded: particular AI applications require their own domain expertise and evidence, so a general checklist cannot substitute for an applicable safety analysis.

Worked example

A fictional simulation contains 1,000 obstacle encounters. A perception model misses 10 obstacles, giving a detection rate of 990 divided by 1,000, or 99%. An additional protective layer prevents movement in 8 of those 10 missed cases. Two simulated encounters still allow the hazardous movement.
The observed hazardous-movement proportion is therefore 2 divided by 1,000, or 0.2%, for this test. It is not the probability of injury: exposure, contact, and consequences have not been modeled. Nor does 99% detection establish acceptable safety. The remaining cases, test representativeness, dependence between protective layers, and severity require examination. These invented figures explain why component accuracy and system harm are different measures; they are not recommended limits for a physical system.

Illustrative case study

At fictional Linden Automation, a team proposes an AI assistant that can recommend changes to a packing line. During review, engineers find that two individually reasonable recommendations could combine badly when a sensor is delayed. The team revises the operating rules and tests the interaction in simulation before considering further use.
Its record describes the hazard, assumptions, protective measures, evidence, and unresolved questions. The fictional case illustrates why safety work includes timing and integration rather than only model accuracy. It does not claim certification or demonstrate that the revised system is ready for production.

Common misconceptions and mistakes

Do not equate safety with reliability. A system can reliably execute a harmful objective. Security is also distinct: protection against malicious interference contributes to safety, but accidents and unsuitable design can cause harm without an attacker.
Zero observed incidents are not proof of zero risk, especially with limited exposure or incomplete reporting. Human review is not an automatic safeguard either; reviewers need time, information, competence, and authority to intervene. Finally, shutdown is not always harmless. The safe response to failure depends on the task and may require a controlled transition rather than abrupt termination.

Business use

Safety analysis helps determine whether a proposed AI role is appropriate, what evidence is required, and where responsibility lies. It can influence procurement requirements, release gates, operating procedures, and incident learning. Business owners should understand the consequences of failure and ensure that technical teams have access to domain expertise. Claims should remain proportionate to demonstrated conditions and applicable requirements.

People also ask

Can a highly accurate model still be unsafe?

Yes. Error consequences, unsuitable objectives, and integration failures can create serious hazards.

Is safety only relevant to robots?

No. Software recommendations and information outputs can influence consequential human decisions and workflows.

Can safety be handled only before launch?

No. Changing conditions, incidents, and system updates can invalidate earlier assumptions, requiring continued monitoring and review.

Check your understanding

An assistant correctly predicts that equipment will be busy and recommends skipping a required inspection to avoid delay. Does predictive accuracy establish safety?

Answer: No. The recommendation can create a hazard despite the correct prediction. The objective, permitted actions, and operating requirements must also be appropriate.

Related terms

• Reliability
• Security
• Robustness
• hazard
• risk assessment
• human oversight
• fail-safe behavior
• system safety

Learn next

Select a hypothetical AI workflow and trace one harmful outcome backward through the system. Identify the conditions that enable it and which controls interrupt each path. Include human actions and dependencies, then ask what evidence would show that each control works under the relevant conditions.

More in Governance, ethics and safety

• Risk tolerance - Risk tolerance is the amount and type of uncertainty or potential harm that an organization or person is prepared to accept while pursuing an objective.
• Robot ethics - Robot ethics examines the moral questions raised by designing, building, deploying, and interacting with robots.
• Robust AI - Robust AI is AI designed and evaluated to keep functioning acceptably when inputs, operating conditions, or other relevant circumstances vary.
• Sampling bias - Sampling bias is systematic distortion arising when the process used to collect a sample gives an unrepresentative view of the population or task of interest.
• Scalability - Scalability is a system's ability to handle growth in workload while meeting defined performance, quality, and resource requirements.
• Selection And Sampling Bias - Selection And Sampling Bias is the combined label used in the Artificial Intelligence Ontology for distortion introduced by a selection process that fails...

Sources

Sources: NIST AI RMF, safety and related trustworthiness characteristics: https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/ . NASA, Safety and Decision Support, software and system safety: https://www.nasa.gov/reference/jsc-safety-decision-support/ . Amodei and colleagues, Concrete Problems in AI Safety: https://arxiv.org/abs/1606.06565 . Dobbe, System Safety and Artificial Intelligence: https://arxiv.org/abs/2202.09292 .

Previous: Safe policy iteration Next: Saliency