arxiv.org web signal

VisualArtifactSpec benchmarks nine models on 211 visual tasks

TL;DR

  • The benchmark spans 211 creation and modification tasks across presentations, static graphics, animations, videos, and web interfaces, scored on two separate axes.
  • Nine model-harness configurations scored between 42.12 and 70.52 overall, with Artifact Performance driving 73.4% of the gap between top and bottom.
  • Typography and legibility drew negative ratings 52.31% of the time, followed by traceability and handoff at 26.72% and graphical craftsmanship at 25.06%.

Nine model-harness configurations run through 211 visual authoring tasks. Overall scores land between 42.12 and 70.52. The split under the hood is what the paper on arXiv wants readers to notice: "Of the 28.40-point total gap between GPT-6 Astra and Claude Sonnet 5, 20.85 points (73.4%) come from AP." AP stands for Artifact Performance, which rates Content, Organization, Visuals, and Delivery against task-specific anchors, separately from Task Fulfillment's check on explicit requirements.

The benchmark covers "presentations, static graphics, animations and videos, and web interfaces," and every task "specifies authoring objectives, source materials, constraints, and deliverables."

Where did the models fall over? The paper reports that "Typography and legibility (V3, 52.31%), traceability and handoff (D3, 26.72%), graphical craftsmanship (V6, 25.06%)" drew the most negative ratings across configurations. The gap is on the craft side, not the compliance side.

Two researchers we track flagged the preprint.

Shared on Bluesky by 2 AI experts