@yonashav @Miles_Brundage There's also a lot of the traditional consulting firms (ie Mckinsey, Deloitte, etc) who have AI audit teams; several law firms etc - we identify what we saw here; https://t.co/xtjMeTt9yW Again, I'm not endorsing anyone in particular but I suspect the …
AI auditing: The Broken Bus on the Road to AI Accountability arxiv.org
AI Weekly's analysis
→
- A SaTML 2024 paper by Birhane, Steed, Ojewale, Vecchione and Raji calls the practical AI audit ecosystem 'muddled and imprecise.'
- The authors report that 'only a subset of AI audit studies translate to desired accountability outcomes' across six stakeholder groups they taxonomize.
- They locate the difference in audit design, methodology and institutional context, not the mere fact that an audit was performed.
Read full analysis →
We also wrote a bit about this here; https://t.co/xm4QhTrail I don't think it's as straightforward as people are imagining to set up a third party audit ecosystem and I really wish there was a more direct reference to the more mature audit regimes we should be learning from!
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance arxiv.org
AI Weekly's analysis
→
- Raji, Xu, Honigsberg and Ho argue existing algorithmic accountability policy has "neglected the lessons from non-algorithmic domains" on third-party participation.
- The paper surveys audit systems in financial, environmental and health regulation and finds their institutional design is "far from monolithic."
- Their conclusion: audits alone are "unlikely to achieve actual algorithmic accountability" without sustained institutional design work.
Read full analysis →
Anyways, here's some of that discourse from 2019; https://t.co/pzBqIF2MEA + Communities like MD4SG, who piloted in 2016; https://t.co/3CQxeNxYiw And folks like @jessicadai_ explicitly drawing bridges to the language of "alignment" in 2023; https://t.co/3etNai5xlr
The Artificiality of Alignment joinreboot.org
I don't want people to underestimate civil society's potential as a meaningful "embedded AI evaluator". There's Data & Society's "AI on the ground" team (https://t.co/QY22xTrgqd), CDT's AI governance lab (https://t.co/L1qlxtSdT5), ACLU's algo justice team, and many more!
AI on the Ground - Data & Society datasociety.net
I don't want people to underestimate civil society's potential as a meaningful "embedded AI evaluator". There's Data & Society's "AI on the ground" team (https://t.co/QY22xTrgqd), CDT's AI governance lab (https://t.co/L1qlxtSdT5), ACLU's algo justice team, and many more!
AI Governance Lab - Center for Democracy and Technology cdt.org
Who are these "embedded evaluators" and how can we guarantee actual independence & consequential judgement? NTIA once wrote extensively about the risk of "audit-washing" in the AI industry & I'm increasingly convinced this is a risk worth taking seriously; https://t.co…
AI Accountability Policy Report | National Telecommunications and Information Administration ntia.gov
