Figma benchmark scores 32 LLMs: pass@1 spans 6.0% to 67.7%

A new benchmark grades 32 language models on animating static SVG files from motion prompts, and the gap between best and worst is nearly tenfold. In MotionGraphicsBench, the authors report pass@1 rates "from 6.0% to 67.7%" across 47 configurations from 32 LLMs and 12 providers.

The benchmark drops the usual VLM-judge setup. Each of its 100 curated instances pairs a motion edit prompt with a "deterministic, human-readable verification program that accepts different valid implementations and reports which prompt requirements pass or fail, rather than comparing against a reference animation or using a VLM judge." On a balanced 180-animation test set, the paper reports the programs "achieve 93.3% raw agreement with human judgments, compared with 83.3% for a GPT-6 Astra VLM baseline."

Where the models trip is specific. "Generated animations often fail to match reference poses, follow requested motion trajectories, or coordinate related events," the paper states. More attempts help but do not close the gap: across 15 models evaluated at their highest available reasoning effort with 32 attempts per instance, pass@32 still ranges from 38.0% to 95.0%.

The 100 instances are derived from real-world Lottie motion designs, and the dataset is posted at huggingface.co/datasets/figma-ai/motion-graphics-bench. The abstract names neither which model lands at 67.7% nor which sits at 6.0%.