AI safety
In short
AI safety is the work of stopping AI systems from causing harm, whether through accidents, deliberate misuse or behaviour their makers never intended. It covers testing models before release, technical research and government oversight.
- Also called
- Artificial intelligence safety, Safe AI
- Topic
- Safety, ethics and risk
- Level
- Beginner
- First described
- Concerns go back decades, but the modern research agenda is usually traced to "Concrete Problems in AI Safety" by Amodei, Olah, Steinhardt, Christiano, Schulman and Mané, posted in June 20162.
There is no single legal definition. The US standards body NIST says a safe AI system should not, under defined conditions, lead to a state in which human life, health, property or the environment is endangered1.
A 2016 paper by Dario Amodei, Chris Olah and colleagues helped set the technical agenda. It framed the problem as accidents: unintended and harmful behaviour that comes from poor design. It listed five research problems, including avoiding harmful side effects, avoiding reward hacking and coping when the real world differs from the training data2.
The term now spans everyday harms and much larger ones. The International AI Safety Report 2026, led by Yoshua Bengio and written by over 100 experts, sorts risks into three groups: malicious use, such as AI-assisted fraud and cyberattacks; malfunctions, such as unreliable answers and loss of control; and systemic risks, such as effects on jobs3.
In practice, safety work shows up as testing models for dangerous capabilities before release, training them to refuse harmful requests, and publishing safety frameworks. The 2026 report counts 12 companies that published or updated such frameworks in 2025, and notes that most of this remains voluntary3.
Governments have built institutions around it. The UK set up an AI Safety Institute and, in February 2025, renamed it the AI Security Institute, with a sharper focus on risks such as chemical and biological weapons, cyber-attacks, fraud and child sexual abuse4.
Example
Before a new chatbot is released, specialist testers try to get it to give step-by-step help with a cyber-attack. If it does, the developer adds safeguards and tests again. Capability evaluations of this kind are a standard part of AI risk management3.
Why it matters
AI tools are now built into search, banking, healthcare and public services. Safety work decides whether their failures are caught in testing or discovered by the public.
Questions people ask
What is the difference between AI safety and AI security?
They overlap. Safety is the broader aim of preventing harm from AI, including accidents. Security focuses on deliberate threats. When the UK renamed its AI Safety Institute the AI Security Institute in 2025, it said the new name reflected a focus on risks with security implications, such as weapons, cyber-attacks and fraud4.
Is AI safety only about the risk of AI wiping out humanity?
No. The International AI Safety Report 2026 covers fraud, deepfakes, manipulation, cyberattacks, unreliable outputs and job impacts, alongside loss of control. It notes that expert opinion on loss of control varies greatly3.
Who is responsible for AI safety in the UK?
There is no single AI regulator. The government's AI Security Institute has a team of scientists evaluating the most serious risks from advanced AI. Since 2025 it has said it will not focus on bias or freedom of speech4. Existing regulators, such as the ICO for data protection, apply their own rules to AI.
Related terms
Sources
- NIST AI 100-1 (January 2023), Artificial Intelligence Risk Management Framework (AI RMF 1.0), section 3.2
- Amodei et al. (2016), Concrete Problems in AI Safety, arXiv:1606.06565
- International AI Safety Report 2026 (chair: Yoshua Bengio), February 2026
- DSIT and AI Security Institute (14 February 2025), UK's AI Safety Institute becomes 'UK AI Security Institute', GOV.UK