OpenAI wants independent auditors for its models. It proposes what they should actually examine

OpenAI is proposing a more detailed framework for independent safety assessment of the most advanced AI. According to the company, external review should not be a one-off test against a few public benchmarks, but longer-term verification of specific safety claims, the methods used and the real effectiveness of safeguards. OpenAI distinguishes between a “safety claim” and a “safety case.” A claim is a specific, verifiable statement, for example about model capabilities or the effectiveness of a particular safeguard. A safety case is a broader evidence-backed argument explaining why risks are acceptably managed for a particular type of use. This distinction matters. A general statement that a model is “safe” cannot be meaningfully audited. It is possible, however, to test a claim that a particular safeguard prevents a model from carrying out a defined class of dangerous tasks under specified conditions. The company says that in some previous external evaluations it provided deeper access to internal information, technical safeguards, confidential data and, in some cases, information needed to analyze the model’s internal process. This addresses a fundamental problem of independent AI audits. An evaluator who sees only the same public interface as an ordinary user cannot verify many claims about training, monitoring or internal safety mechanisms. At the same time, everything cannot simply be made public. Detailed information about safeguards can help attackers, while training data or model weights have high commercial value. Auditing therefore needs a legal, technical and security regime for working with confidential information. OpenAI proposes that some evaluations last several weeks and others several months. They do not necessarily have to be tied only to the launch of a new model. That reflects the reality that system behavior also changes with new tools, updates and deployment methods. For regulators, this approach is interesting as a possible basis for standardization. The European AI Act and other regimes need practical methods for assessing whether a supplier has actually met a safety obligation. The manufacturer’s own documentation is not independent evidence. One risk, of course, is auditor dependence on the company being audited. If the laboratory selects the auditor, pays it and determines the scope of access, rules are needed to protect the independence of the assessment and the ability to publish an unfavorable conclusion. For enterprise customers, the principle can be applied immediately. When buying critical AI, they should ask for specific safety claims, evidence of how they were tested and information on whether anyone independent has verified them. The market may therefore gradually move from the marketing language of “responsible AI” toward auditable technical commitments. That would be a substantial change: safety would become a property that a supplier has to demonstrate, not merely declare. KEY TERMS Safety claim: A specific and verifiable statement about a safety property of a model or system. Safety case: A structured, evidence-backed argument that specific risks are sufficiently controlled. Independent audit: A review performed by an entity that is not part of the manufacturer of the system being assessed.

OpenAI wants independent auditors for its models. It proposes what they should actually examine

The full article is available to users registered at Hard Skills