AI audit field scan catalogs 438 auditors, urges mandates

The arXiv paper counts 438 people and 189 organizations doing algorithmic auditing work, then argues the field has no standards to hold them to.

"AI audits are an increasingly popular mechanism for algorithmic accountability; however, they remain poorly defined," the authors write. Without shared definitions, they add, "claims that an AI product or system has been audited, whether by first-, second-, or third-party auditors, are difficult to verify and may exacerbate, rather than mitigate, bias and harm."

The five authors — Sasha Costanza-Chock, Emma Harvey, Inioluwa Deborah Raji, Martha Czernuszenko and Joy Buolamwini — describe the work as "the first comprehensive field scan of the AI audit ecosystem." Their evidence base: a catalog of 438 individuals and 189 organizations, an anonymous survey of 152 of them, and interviews with 10 industry leaders. The paper is listed for FAccT '22.

Six recommendations follow. Require owners and operators of AI systems to engage in independent algorithmic audits against clearly defined standards. Notify individuals when they are subject to algorithmic decision-making systems. Mandate disclosure of key components of audit findings for peer review. Consider real-world harm in the audit process, including through standardized harm incident reporting and response mechanisms. Directly involve the stakeholders most likely to be harmed by AI systems. And formalize evaluation and, potentially, accreditation of algorithmic auditors themselves.

The abstract flags "proposals with wide support from algorithmic auditors as well as areas of debate" but does not say which of the six sit on which side of that line.