The Limits of Regulation in Managing Existential AI Risks

• OpenAI recently withdrew a new AI model due to failures in internal safety testing, prompting renewed calls for regulation.
• Current demands for independent oversight often lack specific details on implementation or the evidence required for approval.
• Traditional safety-critical industries use "safety cases" to prove hazard control and low accident probabilities.
• Aviation and nuclear standards require proving that catastrophic events will not occur more than once in 1,000 years with at least 99% probability.
• AI developers claim their systems pose existential risks but have provided no detailed risk analyses or independently assessable safety cases.
• Regulatory policies must rely on rigorous factual analysis rather than speculation to address the unique challenges of AI safety.

The core issue lies in the mismatch between how we regulate physical danger and how we discuss digital existential risk. In engineering disciplines that deal with high-consequence failures, such as controlling aircraft or nuclear power stations, safety is not assumed; it is proven through data. International standards require these industries to submit comprehensive safety cases. These documents undergo intense scrutiny to show that every possible hazard has been identified and mitigated.

The statistical threshold for approval in these fields is unforgiving. Regulators expect developers to demonstrate that the likelihood of an accident causing multiple deaths is less than one in 1,000 years, with a confidence level of at least 99%. This requirement exists because the potential for harm is immediate and physical. Even under these strict conditions, providing such evidence is extremely hard for complex mechanical systems.

Frontier AI developers, however, operate in a different paradigm. They frequently assert that their models could lead to the extinction of humanity. Yet, they have not produced the detailed risk analyses necessary to support such claims. There is no independent assessment of whether these risks are real or manageable. More importantly, there is no evidence that it is even possible to produce safety cases for AI systems that meet the standards used in aviation or nuclear engineering.

This absence of data makes the argument for regulation based solely on oversight insufficient. Regulation works best when there are clear metrics for compliance. In the absence of verifiable safety cases and detailed risk analyses, regulators have no objective basis to determine if an AI system is safe. Therefore, relying on independent oversight without a foundation of rigorous engineering proof is logically unsound.

The withdrawal of OpenAI’s latest model serves as a reminder that technical challenges in AI safety are significant. While the call for independent oversight is understandable, it does not address the fundamental problem: the lack of verifiable evidence regarding existential risks. Traditional industries prove safety through rigorous statistical analysis and detailed hazard identification. The AI sector currently lacks both the data and the methodology to provide similar assurances.

Until developers can produce detailed risk analyses and safety cases that withstand independent assessment, regulatory frameworks will struggle to offer meaningful protection against catastrophic outcomes. Policy must move beyond general calls for oversight and demand the same level of rigorous factual analysis required in other high-stakes engineering fields. Without this shift, arguments about AI safety remain speculative rather than scientific.