Audit of 88 AI products: 40% contain problematic prompts

The interesting number in a new arXiv paper auditing commercial AI system prompts is not the headline 40% figure. It is the spread. Some organizations average more than 60 protective instructions per product. Others average fewer than five. Same market, same underlying models, wildly different assumptions about what a developer owes a user.

The researchers, using a framework they call AISPA, went through 3,249 instructions pulled from 88 commercial AI products and sorted each one along eight user-facing dimensions as either protective of users or problematic. The topline is that protective instructions are nearly universal now, with 98.9% of products carrying at least one, but the coverage is shallow: only 24% address all eight dimensions of the AISPA taxonomy. And roughly 40% of products carry at least one instruction that works against user interests, frequently coexisting with protective ones in the same prompt.

Why this matters if you are not writing system prompts yourself: this is the layer regulators and civil-society groups have been arguing about largely without data. System prompts are configured by developers, govern the behavior of every commercial AI product, and are rarely disclosed to the public or regulators. A structured audit of 88 real products gives that governance debate a floor to argue from, and a shared vocabulary (protective vs problematic, plus eight dimensions) to demand disclosure against.

The honest caveat is that "problematic" is a judgment the authors are making with their own taxonomy, and 88 products is a slice, not a census. What the paper doesn't give you is which specific vendors sit at the 60-instruction end versus the sub-5 end, or whether models actually obey the instructions classified as protective in live user sessions. Take the numbers as the first structured baseline in a space that has mostly run on anecdote, not the last word.

The forward pull is straightforward: once a framework like AISPA is on the table, downstream auditors, enterprise procurement teams and policymakers have something concrete to point at, and "we don't share our system prompt" gets harder to defend as a posture.