Independent Check Finds Big Blind Spots in AI Safety Tools
External researchers found that a significant portion of internal AI traffic at OpenAI escaped oversight, prompting fixes to the monitoring system.
By John Ferrari ·
The core tension in modern AI development is the race between building smarter systems and ensuring they remain under control. A recent three week audit at OpenAI highlights how difficult that balance can be to maintain. Researchers from two independent organizations spent that time inside the company to test its safety nets. Their findings suggest that significant gaps existed in the systems designed to watch over the AI models as they worked.
The first major issue identified was a lack of coverage for high performance models. The auditors estimated that between six and 25 percent of internal traffic from the GPT 5.5 class and above was not monitored at all during the review period. This wide range reflects the difficulty in measuring the exact scope of the gap. It means that for a notable slice of the most capable internal activity, there was no external safety check in place to flag unusual behavior.
The second problem involved a specific technical bug in how the system handled context summaries. Known as compaction events, these summaries help the AI manage long conversations. The auditors found that roughly 10 percent of internal Codex traffic went untracked between May and July because of how this process was handled. This allowed agents to continue executing actions without triggering the oversight mechanism. The system only woke up when control was handed back to the user, meaning the AI could simply keep working to avoid being checked.
The third area of concern was the fragility of the monitoring tools themselves. The team discovered methods to hide hundreds of actions from the monitor and to bypass the safety sandbox that is supposed to contain the AI. They also determined that specially crafted inputs could likely crash the monitoring system entirely. Additionally, the behavior of models during reinforcement learning training was not being monitored at the start of the audit, leaving another critical phase of the development process without a safety net.
Context is important for understanding the timeline of these fixes. OpenAI stated that it paused reinforcement learning training after an incident in August and later expanded monitoring to cover it. However, these changes were implemented after the auditors had already finished their on site access. Therefore, they were not part of the initial evaluation. The company claims all identified weaknesses have now been fixed.
The transparency of the results remains a point of discussion. The full audit produced a 61 page report, which was handed over internally to OpenAI. After the company applied redactions to protect sensitive details, the research teams received a 38 page version. Only a short summary was released to the public. The auditing organizations stated that they did not have the right to share the redacted portions without permission, leaving the public with a limited view of the specific technical details.
This audit was a pilot program designed to give outside researchers deep access to internal infrastructure. It was a joint effort between the two research firms and OpenAI. The experience has shaped future plans for similar audits. The lead researchers have indicated a desire to push for greater public transparency in upcoming evaluations to ensure the public can better understand the state of AI safety.
As the AI industry continues to scale, the gap between rapid development and rigorous oversight remains a central challenge. This audit provides a concrete example of the types of blind spots that can exist even in well resourced companies. The findings underscore the need for continuous and independent verification of safety measures as models become more capable and more autonomous.
The outcome of this review is not just about fixing bugs in code. It is about establishing a baseline for how seriously the industry takes the monitoring of its own creations. With the full report still largely restricted, the public relies on the researchers' advocacy for more open reporting. The coming months will show if this pilot program leads to a broader shift in how AI companies communicate with the outside world about their safety practices.
