Anthropic's Claude Automates Safety Proofs for Critical Open-Source Software

Developers and software teams can now rely on a new automated method to verify the safety of critical code, significantly reducing the manual effort required to ensure software runs without errors. By leveraging Anthropic’s Claude model to generate mathematical proofs, the process identifies flaws that traditional testing methods often miss. This shift means that verifying code correctness is becoming a routine, automated task rather than a specialized, labor-intensive project reserved for high-stakes industries like aviation.

The initiative, led by Anthropic’s Frontier Red Team in collaboration with partner Theorem, focused on verifying existing software rather than generating new code. Over a period of just a few weeks, the model produced approximately 180 million lines of proof code for 123 open-source libraries sourced from Google’s OSS-Fuzz security project. The system works by translating crash tests into mathematical problems, analyzing compiled machine code, and generating formal proofs. An independent verification program named Rocq then checks these proofs, a process Anthropic describes as nearly free of human review, labeling it "fully autonomous" despite the human-built foundations provided by Theorem.

The approach addresses a long-standing challenge in software development: formal verification has historically been prohibitively expensive. For instance, verifying the small operating system kernel seL4 required twenty person-years of specialist work. By automating this process, the new method makes rigorous safety checks accessible for a broader range of applications. During the verification run, Claude identified hundreds of bugs that conventional testing had overlooked, many of which only manifested under specific conditions such as unusually large input files or particular access sequences.

The impact was particularly evident when verifying a widely used PDF reader, where the team discovered as many security bugs in a single week as are typically reported over an entire year. In another instance, the model caught a logical error that had remained undetected for more than a decade. Anthropic notes that even when a complete proof fails, the attempt still surfaces genuine bugs, adding value to the testing pipeline.

However, the company emphasizes that "formally verified" does not equate to "error-free." The proofs are valid only under strict conditions, including specific processor architectures, single-threaded programs, and inputs under 64 megabytes. Since these libraries often run on diverse hardware configurations, the verification covers a defined subset of operational scenarios. Additionally, the computational cost remains high, with individual proofs consuming hundreds of millions of tokens, though the company has not disclosed the total project expenses.

Looking ahead, the project suggests a broader future for software automation. If AI can both write code and prove its safety, developers may trust AI-generated software more, reducing time spent on manual review. Preliminary data indicates that proofs are reusable and that newer models, such as Opus 5.5, are solving problems more efficiently, using about 40 percent fewer tokens than previous versions. Anthropic expects formal verification with AI to become as standard as routine testing, with Theorem’s roadmap already targeting critical infrastructure components like glibc, OpenSSL, and cURL.