Birhane, Raji: only some AI audits yield accountability

The practical nature of the "AI audit" ecosystem is "muddled and imprecise," a SaTML 2024 paper by Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione and Inioluwa Deborah Raji argues, and only a fraction of the work being done under that label translates into accountability.

The authors set out to "taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, and consulting agencies." Reading across those six groups, they report that "only a subset of AI audit studies translate to desired accountability outcomes."

That gap is the broken bus of the title. The paper's remedy, at abstract level, is to attend to "connections between AI audit design, methodology and institutional context": who is doing the audit, how, and where the findings actually land, rather than treating "audit" as a single, well-understood label.

Neither the arxiv listing nor the abstract puts a number on the effective fraction, or ranks the six stakeholder groups against each other; that reading sits in the body of the paper.