Control (AI safety)

Most safety work, called alignment, tries to make sure a model wants the right things. Control asks a different question: if alignment has failed and the model is trying to get around your safeguards, would you still be safe²? Researchers at Redwood Research describe the two as separate lines of defence, so a disaster needs both to fail².

The approach was set out in a December 2023 paper by Greenblatt, Shlegeris, Sachan and Roger¹. They gave a strong but "untrusted" model (GPT-4) programming tasks, alongside a weaker "trusted" model (GPT-3.5) and a small budget of human checking¹. A red team then played the untrusted model, trying to sneak hidden bugs, called backdoors, past the safeguards¹.

The safeguards are called protocols. In "trusted editing", the weaker model rates how suspicious the stronger model's code looks and rewrites the suspicious parts before use. In "untrusted monitoring", a second copy of the strong model checks the code, with steps taken to stop the two copies colluding¹. Both did much better than simple baselines¹.

The UK AI Security Institute released ControlArena, an open library for running these experiments, in October 2025³. It sets an AI agent a legitimate main task, such as updating cloud deployment settings, alongside a hidden side task, such as weakening security, and measures whether the protocol catches it³. AISI says researchers at Anthropic, Google DeepMind and Redwood Research have used it³.