paper web signal

Squoosh UX-1 Predicts 84% of A/B Winners in 318-Test Bench

TL;DR

  • UX-1 classified 267 of 318 historical A/B outcomes correctly (84.0%), against 61.6% for Qwen3 Next 80B on the same set.
  • Of the 126 experiments where UX-1 favored a variant, 118 variants won the live test, four lost, and four were inconclusive.
  • The benchmark covered 318 e-commerce tests that ran longer than 21 days, with UX-1 simulating 1,000 shoppers per arm.

A new benchmark from Squoosh reports that UX-1, its model fine-tuned on human browsing data, correctly predicted the outcome of 267 of 318 historical e-commerce A/B tests, or 84.0%, with a 95% Wilson interval of 79.5 to 87.6%. On the same set of experiments, the strongest general-purpose system the authors tested, Qwen3 Next 80B, reached 61.6%, according to the preprint posted on arXiv.

The precision inside UX-1's variant picks is the sharper number. The authors write that UX-1 "favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four had inconclusive results." The remaining 192 experiments contained 19 variant wins the model did not flag.

The team, Rohith Pasula, Johari Dramiga, Viraj Shah, Suzan Halim and Tyler Dooskin, screened 412 historical tests down to the 318 that ran longer than 21 days, simulated 1,000 shoppers per arm, and masked page roles and recorded outcomes during inference.