Is AI safety only about future highly capable systems?

No; present-day systems can create operational harms too.

Is safety the same as security?

They overlap, but safety also concerns unintended failures without an attacker.

Can a benchmark prove universal safety?

No; conclusions depend on coverage, assumptions and the deployment context.

AI Glossary / Governance, ethics and safety / Advanced

AI safety

Definition

AI safety is the study and practice of preventing or reducing harm arising from AI systems and their use. It includes identifying hazardous behavior, testing relevant conditions, limiting unsafe actions and supporting appropriate intervention. A safety claim must describe its context and evidence. Calling a model accurate, helpful or advanced does not establish that the system using it is safe.

What it means

Safety concerns what can happen to people, property, the environment or other protected interests when a system operates. It extends beyond the model's individual answer to the actions and dependencies surrounding that answer. A recommendation shown to a trained reviewer has a different pathway to harm from the same recommendation executed automatically.
The field includes immediate operational problems and research on more capable systems and systemic risks. Different communities emphasize different parts of that scope. It is useful to specify the hazard being discussed rather than assume that everyone using the phrase has the same setting or time horizon in mind.

How it works

Safety work begins with the intended use, affected people, operating conditions and credible failure scenarios. Teams then connect safeguards and tests to those scenarios, examine remaining uncertainty and define how the system should respond when conditions fall outside its supported use.
NIST's AI RMF distinguishes safety from related qualities such as reliability and security. Its safety discussion emphasizes early design work, relevant testing, monitoring and the ability to intervene or modify operation. Security failures can create safety consequences, but accidental failures can also do so.
A practical safety argument links claims to evidence and limitations. It should explain what a test covers, what it leaves unresolved and what would invalidate earlier conclusions. Evidence must be revisited when capabilities, data, users or the surrounding workflow change.

Three practical examples

An AI-assisted warehouse system is evaluated for situations where its perception is uncertain. The team considers the behavior of the complete machine and its protective mechanisms, not only image-classification accuracy.
A customer assistant is prevented from automatically executing consequential actions until the relevant workflow has been assessed. The team checks whether escalation reaches a person who can actually resolve the problem.
A teaching platform reviews generated experiments for unsafe suggestions before publication. It examines the instructions as a reader might follow them, rather than judging only whether they sound fluent or educational.

Think of it this way

Think of asking whether a bridge is suitable for particular loads and conditions. A favorable measurement helps, but its meaning depends on what was measured, how the structure will be used and which failure modes were considered. Similarly, a successful AI demonstration supplies evidence about that demonstration, not every future operating condition.

Worked example

A fictional simulator contains 1,000 cases for an AI-assisted control workflow. Reviewers classify 20 as hazardous situations requiring intervention and 980 as ordinary situations. Suppose the system handles all 980 ordinary cases correctly but responds appropriately to only 15 of the 20 hazardous cases.
Its overall correct-response rate is (980 + 15) / 1,000 = 99.5%. Its appropriate-response rate within the hazardous subset is 15 / 20 = 75%, leaving five missed hazardous cases. The impressive overall number conceals a weakness concentrated in the scenarios of greatest concern.
The test designer deliberately selected the cases. Therefore, the 2% hazardous-case share in this test does not establish their frequency in actual operation. Nor does a 75% rate in 20 cases precisely establish the real-world intervention rate.
If a revised system handles all 20 hazardous cases correctly, that is encouraging evidence for those cases, not proof of zero risk. The team still needs fresh cases, justified coverage and an assessment of untested failure pathways. These invented numbers demonstrate evaluation interpretation; they are not acceptable-performance thresholds for any real control system.

Illustrative case study

Fictional Alder Assist pilots an AI system that suggests maintenance steps to experienced staff. The pilot review assumes that staff check suggestions against approved instructions before acting. The system performs well on the selected pilot tasks.
A later expansion introduces less experienced users and a simplified interface that hides the source instructions. The model has not changed, but the basis for the earlier safety judgment has. Incident reports show that users sometimes interpret a confident suggestion as an approved procedure.
The team reassesses the workflow, restores access to authoritative instructions and tests whether users recognize uncertainty and obtain qualified help. It also identifies tasks outside the supported scope. The case demonstrates that safety depends on people and interfaces as well as the model, and that an earlier pilot result cannot silently authorize a different use.

Common misconceptions and mistakes

Equating safety with average accuracy: failures can differ greatly in consequence.
Assuming human review automatically resolves risk: reviewers need time, information, competence and authority.
Treating a lack of observed incidents as proof: important hazards may be rare, unreported or absent from the test.
Confusing safety with content filtering alone: permitted actions, interfaces and operating conditions also matter.
Describing an entire system as safe without qualifications: the claim needs a defined scope and supporting evidence.

Business use

Make safety claims concrete enough to review. Identify the relevant hazards, users, operating limits and evidence supporting each safeguard. Connect monitoring to action, including who can restrict or stop the system when needed. Compare proposed AI use with realistic alternatives while keeping severe consequences visible. Where established sector-specific safety practices apply, AI development should fit into those practices rather than replace them with a generic model benchmark.

Check your understanding

Why is 99.5% overall performance misleading on its own?

Answer: Five of the twenty hazardous cases were missed.

Does the test establish a 2% real-world hazard frequency?

Answer: No; its cases were deliberately selected.

Why did Alder's changed interface require reassessment?

Answer: It altered the human checks assumed in the pilot.

Related terms

AI risk management
Robustness
Reliability
hazard analysis
human oversight
safety case
incident monitoring
AI alignment

Learn next

Choose a hypothetical AI workflow and trace one error through to a possible consequence. Identify which safeguard interrupts that pathway and what evidence would show that the safeguard works under realistic conditions.

Sources

NIST AI RMF 1.0, AI Risks and Trustworthiness, safety, reliability and security distinctions. https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/ Stanford CS120, Introduction to AI Safety, model-specific and systemic safety scope. https://web.stanford.edu/class/cs120/index.html NIST AI RMF 1.0, Framing Risk, context and measurement limitations. https://airc.nist.gov/airmf-resources/airmf/1-sec-risk/ The simulator calculation and Alder Assist case are original teaching illustrations.

Previous: AI risk management Next: AI Scaling Wall