AI audits need room to test what companies do not volunteer

A proposed White House accord on artificial intelligence audits could face a basic credibility problem: an audit is only as informative as the access and freedom given to the people conducting it. If auditors are limited to the material that AI companies voluntarily show them, important weaknesses may remain outside the review.

The limits of company-controlled reviews

The concern, raised by Samuel Vance-Law, head of research at the Decentralization Research Center, centers on who sets the terms of an audit. When the system’s operator decides which features, records or scenarios can be examined, the process may test a carefully selected version of the technology rather than the full range of ways it operates.

That distinction matters because AI systems can be evaluated through different kinds of tests. A review based only on prepared demonstrations may produce a narrower picture than one in which auditors can choose their own questions and testing conditions. Independent selection of test cases would give auditors more room to investigate issues that are not highlighted by the companies being reviewed.

What meaningful oversight would require

The argument does not suggest that every audit should be unlimited or that companies should lose control over sensitive information. It does suggest that an audit framework needs enough independence to prevent the subject of the review from defining its boundaries entirely.

For readers following the intersection of AI, technology policy and digital assets, the issue is broader than one proposed agreement. Audits are often presented as a way to turn broad assurances about safety and accountability into evidence that can be examined. That role becomes weaker when the audit process is shaped primarily by the party seeking approval or public trust.

The White House accord’s value will therefore depend not only on whether audits are required, but also on how much discretion auditors receive. Allowing them to select what they test would make the process more probing; restricting them to what they are shown would leave the quality of the review closely tied to the company’s own choices.

Source: Coindesk