AI software gets a new safety check

The age of manually checking every line of software code for errors may be ending. A new collaboration has shown that artificial intelligence can now take over one of the most difficult and expensive tasks in software engineering: proving that code works correctly under all conditions.

The scale of this achievement is difficult to overstate. In a matter of weeks, the AI system generated approximately 180 million lines of mathematical proof code. This output covered 123 different open-source libraries. Traditionally, this type of rigorous verification requires a team of specialized experts working for years. For example, a previous effort to verify a small operating system kernel required 20 person-years of labor. That is why such methods were usually reserved only for critical systems like airplane avionics.

Anthropic, the company behind the Claude model, worked with a partner named Theorem to make this process automated. The team used Claude to analyze existing software rather than writing new applications from scratch. The AI translated standard crash tests into mathematical problems and then analyzed the compiled machine code. It produced extensive proofs that were then checked by an independent program called Rocq. According to the researchers, this process required very few human reviewers, leading them to describe the proof generation as fully autonomous.

The method has already proven its worth by finding errors that escaped human detection for years. During the verification process, Claude encountered hundreds of bugs and was allowed to fix them automatically. Many of these issues would have slipped past conventional testing because they only appear under very specific conditions, such as unusually large input files or a specific sequence of data access. In one notable instance, the team found as many security bugs in a single week for a popular PDF reader as are typically reported for that software in an entire year. In another case, the AI caught a logical error that had gone undetected for over a decade.

However, the technology is not a magic solution. Anthropic notes that formal verification does not guarantee that software is completely error-free. The proofs are only valid under strict conditions, such as a specific processor architecture and single-threaded programs with inputs under 64 megabytes. In the real world, these libraries often run on different hardware and in various configurations. Additionally, the computing cost is significant, with individual proofs requiring hundreds of millions of tokens. Anthropic has not disclosed the total cost of the project, but the efficiency is improving. A newer model version solves half the proofs with about 40 percent fewer tokens than the previous version.

The stakes for the software industry are high. If AI can verify its own code, developers can trust automated software more and spend less time on manual review. This could save costs across the entire coding process. The project suggests that the question is no longer whether AI can program, but whether it can prove that the code it writes is safe. Anthropic expects this type of verification to become as routine as standard testing in the near future. Theorem’s roadmap already targets critical infrastructure tools like glibc, OpenSSL, and cURL, signaling a major shift in how software safety is managed.

This development marks a significant leap in software automation. By combining AI coding capabilities with mathematical proof, the industry moves closer to a future where software safety is guaranteed by design rather than by chance. The approach tackles a core problem with AI-generated software by providing a layer of mathematical certainty. While practical limits remain, the ability to find hidden bugs at such a scale suggests that the future of software development will be far more rigorous and efficient than it is today.