Blind-Spots-Bench Exposes ~10% Closed-vs-Open Frontier Gap

A new arXiv preprint argues that the tasks humans find trivially easy are exactly where the gap between closed frontier models and open-weight alternatives shows up, and standard leaderboards are missing it. The authors introduce a diagnostic benchmark called blind-spots-bench, 235 samples built specifically to look simple to a person while still tripping up modern systems, and use it to compare closed-source and open-weight language, vision-language, and image-generation models.

The headline number, as the paper reports it, is a roughly 10% gap. Closed-source frontier models can substantially outperform open-weight models on these samples even when the two look comparable on established benchmarks. That framing is the part worth sitting with. The teams shipping agents in the wild have mostly been reassured by leaderboard parity between open and closed. If a diagnostic slice of tasks that read as easy to a human can pull a 10-point gap out of that same pair, the leaderboards are compressing something real.

The other finding is that no single model dominates across task types, and some samples remain challenging for every model the authors evaluated, spanning LLMs, VLMs and image generators together. The categories the paper flags in this range include things like string manipulation and drawing animals with specific features, the kind of instruction a user would assume any competent assistant could handle and which quietly break in production.

The honest caveat is that this is a preprint, the abstract does not disclose the specific vendors or the full model list behind the 10% figure, and a 235-sample benchmark is a diagnostic slice rather than a workload. Take the specifics as reported, not settled. What the reporting also does not give you is a modality-by-modality breakdown of where the gap concentrates, or a named list of the samples that stayed unsolved across every model.

Still, the useful move for anyone selecting models is the one this kind of benchmark supports. If your standard eval mix says open-weight has caught up, run a blind-spots-style probe before committing to a deployment. The delta the leaderboards are hiding is exactly the delta users will notice first.