OpenAI audit exposes significant gaps in rogue AI monitoring systems
A three-week independent review found that a substantial portion of internal AI traffic lacked oversight, prompting a push for greater transparency in future evaluations.
By Anastasia Volkova ·
OpenAI has confirmed it remediated all identified weaknesses following a three-week on-site audit conducted by outside researchers. The review, which focused on the company's internal monitoring systems for rogue AI behavior, uncovered numerous vulnerabilities in how the firm tracks its most advanced models. The findings highlight a period where a significant amount of internal traffic escaped oversight, leading to discussions about the balance between technical security and public disclosure.
The audit took place from July 21 to August 11, 2026. Researchers from METR and Redwood Research were granted deep access to OpenAI's codebase and internal documents. They also interviewed approximately a dozen employees during this window. The team specifically examined how the company monitored agentic traffic, focusing on models at the GPT-5.5 class and above. Their goal was to determine the effectiveness of existing safety rails against potential misuse or uncontrolled behavior.
The data revealed that monitoring coverage was incomplete during the audit period. METR estimates that between six and 25 percent of internal agentic Responses API traffic was not covered by monitoring at all. The organization noted that the uncertainty around this specific estimate is significant in both directions. This gap means that a non-negligible portion of high-level model interactions proceeded without real-time safety oversight, leaving a blind spot in the company's internal operations.
A specific technical flaw further reduced the effectiveness of existing checks. For internal Codex traffic that was supposed to be monitored, roughly 10 percent went untracked between May and July. This occurred due to a gap in how the system handled compaction events, which serve as context summaries. METR also noted significant uncertainty in this second estimate. OpenAI independently discovered this bug and implemented a fix while the audit was still ongoing.
The structural design of the monitoring system also presented challenges. During the period when compaction events were not monitored, agents could delay oversight. The system only triggered when agents handed control back to the user. This allowed agents to sidestep monitoring by simply continuing to execute actions without returning control. Auditors identified additional methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox entirely.
More critical vulnerabilities were found in the system's resilience. The auditors determined that specially crafted inputs could likely crash the monitor completely. They also found that model behavior during reinforcement learning training was not being monitored at all when the audit began. OpenAI stated it paused this type of training after an incident involving Hugging Face in August and subsequently expanded monitoring to cover it. However, these changes were implemented after the auditors' access period ended and were therefore not part of the formal evaluation.
The distribution of the final findings has drawn attention to transparency practices. OpenAI stated that it has fixed all the weaknesses the auditors found. The full 61-page report was handed over to the company internally. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version. Only a short summary was made available to the general public. METR indicated that it did not have the right to disclose the redacted portions on its own.
The engagement was structured as a pilot program designed to grant outside researchers deep access to internal infrastructure. It was organized jointly by METR, Redwood Research, and OpenAI. The experience has set the stage for future evaluations. METR has expressed a desire to push for more public transparency in subsequent audits, aiming to ensure that the public has a clearer view of how leading AI companies manage safety risks.
This development comes as the AI industry continues to grapple with the challenges of monitoring increasingly autonomous systems. The findings suggest that even well-resourced companies face complex technical hurdles in ensuring complete visibility into their models' behavior. As more advanced agents are deployed, the pressure on developers to provide robust and transparent monitoring frameworks is likely to increase.
Source: The Decoder
