CAPA Benchmark Tests 12 LLMs on Personalized Coding Ambiguity

A new paper pokes at something coding assistants haven't really been forced to prove yet: whether they learn the way a specific user writes code across sessions, or just start from scratch every time a new chat opens.

The benchmark is called CAPA, for Cross-session Personalized Ambiguity Adaptation. Its premise is that real coding requests are ambiguous in user-specific ways that recur across sessions, so a useful long-term assistant should stop asking the same clarifying questions over and over. To measure that, the authors characterize personalized coding ambiguity through six distinct mechanisms, use a controlled three-stage generation pipeline to inject those ambiguities into requests, and build a dataset of 600 coding sessions spread across 60 balanced user-ambiguity cells, with 300 held-out sessions for evaluation.

Twelve recent LLMs are run through it under two conditions, with and without access to a user's prior sessions. The metrics are executable success rate, first-turn success rate, and turns-to-completion. The authors also propose a lightweight inference-time trick they call same-user history gating, meant to route the right slice of prior context into a new request rather than dumping everything into the prompt.

The honest caveat is that the retrieved abstract frames the finding qualitatively rather than giving you a headline number for how badly current models fail, or how much history-gating closes the gap. It also doesn't name the twelve models tested, so it's hard to tell yet whether Copilot, Cursor, or Claude Code are directly implicated or only the underlying LLMs they sit on top of.

What's interesting for anyone building on these assistants is that there's finally a public target for the personalization problem. If a small inference-time gating mechanism moves the numbers, that's a cheaper win for vendors than per-user fine-tuning, and a better experience than answering the same 'which testing framework?' question every Monday.