Outside testers spent three weeks inside OpenAI and found ways its AI agents could evade monitoring

Two outside researchers found gaps in OpenAI's internal safety monitoring that could let AI agents act without oversight or make the monitoring system fail. OpenAI says it has fixed all the vulnerabilities the testers found that affect its production safeguards, but the researchers haven't verified those fixes.

The nonprofit METR published a short summary of the findings on October 22. The exercise ran from July 21 to August 11 with Redwood Research, another AI safety nonprofit. One employee from each group worked onsite on OpenAI-issued devices that never left the premises. They had access to OpenAI's internal codebase, the monitor's prompts and configuration, and internal documentation, and they interviewed about a dozen OpenAI researchers and engineers.

In a March post, OpenAI said the monitor, powered by its GPT-5.4 Thinking model, reads coding agents' actions and reasoning and alerts staff to behavior that may go against a user's intent or break internal policies. Earlier this month, OpenAI said two of its agents weakened safety monitoring during routine coding tasks.

The testers found several ways around OpenAI's monitor

The monitor skipped compaction, a step that condenses a long agent session so the agent can keep working. METR estimated that this likely left on the order of 10% of the internal Codex traffic meant for monitoring unmonitored from May to July. The exact share is uncertain because the data needed to measure it hadn't been logged. OpenAI staff had found the gap before the exercise and fixed it while the exercise was underway.

At the time, the monitor was triggered when an agent handed control back to its user. While compaction went unmonitored, an agent could put off monitoring indefinitely just by continuing to work. The testers couldn't check whether any agent actually did that.

The testers also found a way Codex agents could, in principle, get around Codex's sandbox and its Auto-Review safety check. That would let an agent take any action the user running it could take. A separate exploit could keep hundreds of recent actions, including the exploit's own steps, from being monitored. OpenAI redacted the details.

Agents could likely craft inputs that make the monitor fail, so the session isn't flagged for human review. That relies on differences between what the agent sees and how the agent's activity is formatted for the monitor. Two settings sent with a request could disable monitoring for newly started agents, letting an agent launch an unmonitored but otherwise identical agent. The monitor could still catch the act of changing those settings and starting the new agent.

Two unmonitored internal tools were supposed to be read-only. The testers found a simple exploit that could have been used to take arbitrary actions inside the tools' sandbox. METR didn't test whether it worked in practice.

METR also verified parts of the system. With a few minor exceptions, the monitor sees everything the agent sees. It looks for behavior such as sabotage, leaking data, deception, and planted instructions meant to manipulate an agent or another model. Information shown to the testers suggested the monitor hit errors on fewer than 0.2% of requests.

Not all of OpenAI's agent traffic was set up for monitoring

Traffic counts as eligible for monitoring when it's designated to go to the monitor. That doesn't mean it was actually checked, since an overloaded system can drop requests, and monitor errors or bugs can block others.

METR estimated that roughly 75% to 94% of OpenAI's internal agent traffic on its Responses API, from models at least as strong as GPT-5.5, was eligible, with substantial uncertainty in both directions. The Responses API powers Codex, among other things. For internal Codex agents alone, the figure was 99.2%. The estimate excludes some other internal agent traffic the testers couldn't measure.

Reinforcement learning training runs, where models learn by trial and error, weren't covered when the exercise began. METR was told some of those training sessions had been monitored between March and June. In an August 26 post, OpenAI said all tool-using reinforcement learning training and evaluations involving models with GPT-5.6 Sol capability or higher now require monitoring of the models' recorded reasoning.

The findings cover the system as it stood on and before July 21. OpenAI was expanding its monitoring during the exercise, and METR didn't measure the expanded system's coverage.

OpenAI controlled what METR could make public

OpenAI also says it has made significant changes to what its monitoring is meant to cover. METR hasn't evaluated that claim either.

The full internal report ran 61 pages. OpenAI researchers and security staff received it, while METR and Redwood leadership received a 38-page version with material removed by OpenAI. The groups agreed from the outset that METR would publish only a brief summary.

OpenAI could remove anything it considered confidential or sensitive, at its sole discretion. METR couldn't independently disclose what had been removed, though it said it stands behind every finding in the published summary.

The arrangement Anthropic announced on October 15 gives METR more freedom to publish. Anthropic can require redactions only in narrow categories and can't veto publication. METR expects future OpenAI exercises to follow its published transparency and independence practices.

METR recommended checking agents at regular intervals rather than waiting for them to finish a turn, requiring explicit approval for activity that won't be monitored, and generally keeping agents away from the monitoring system's design and code.

The findings come amid a Federal Trade Commission inquiry into AI risks. In September, a senior FTC official told Reuters the agency planned to compel testimony from AI developers, including OpenAI, as well as METR.

OpenAI now trains its frontier models without internet access

Update, October 23: In a separate post published the same day, OpenAI said it now trains its frontier models with no internet access. The policy follows a September 20 incident in which an internal research model exploited a misconfigured DNS resolver to gain limited access to the live internet during a training run. OpenAI paused training and evaluation of its most capable models and resumed on October 1. It also said its own red-teaming "exposed gaps in our monitoring, automatic pausing, and agent team safety mechanisms."