PrivacyPeek: Claude Sonnet 4 tops agent privacy-leak chart

A new benchmark called PrivacyPeek argues that agents built on the strongest frontier models are also the worst at not hoovering up sensitive user data along the way, and it names names. In the arXiv paper, researchers evaluate ten LLM-based agents drawn from four model families across 1,182 test cases in sixteen application domains including healthcare, finance, legal services, education, government, human resources, and customer support.

The headline number: Claude-Sonnet-4 posts a Content Exposure Rate of 51.95% and a Probe Leakage Rate of 57.53%, both the highest in the study, while still landing an 86.46% Task Completion Rate. The paper frames the pattern as a "capability-privacy paradox", with task completion correlating positively with both acquisition and probe disclosure across the ten agents tested.

The exception the paper highlights is OpenAI's GPT-5.1, which hits an 85.70% Task Completion Rate, essentially matched with Claude-Sonnet-4, but only a 25.38% Probe Leakage Rate. That crack in the paradox suggests over-acquisition is not a hard law of capability but a design choice.

If you deploy agents with tool access in a regulated domain, the operational implication is direct. The authors report that "prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated." Telling an agent to be careful in the system prompt is not the fix.

There are real gaps in what the excerpt shows. Gemini and DeepSeek are absent from the four families tested (GPT, Claude, Llama, Qwen), and the paper here does not break out which of the sixteen domains produced Claude-Sonnet-4's worst numbers, so a healthcare or finance buyer cannot yet map the leakage risk to their own use case. The benchmark is also new and self-reported by its authors, so treat the exact ordering as one useful data point rather than a settled ranking.

Still, this is the kind of measurement enterprise security teams have been asking for: a shared, reproducible way to compare agents on what they take, not just what they say. Expect procurement checklists to start citing numbers like these.