LLM shopping agents gain confidence from reasoning, not accuracy
TL;DR
- Across three product-choice benchmarks, more reasoning effort did not raise decision quality in a statistically meaningful way across ten frontier models.
- The same reasoning effort did raise stated confidence at a statistically significant level, buying 'confidence rather than correctness.'
- Errors were strongly shared across models: when one erred, several others tended to err on the same task, often picking the same wrong product.
Asking a shopping LLM to think longer reliably increases how confident it says it is, without making its purchases any better, a new arXiv paper reports.
The authors benchmarked ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS. Across all three, "increasing the thinking effort does not increase decision quality in a statistically meaningful way." The same change in thinking effort moved stated confidence at a statistically significant level, meaning "extra thinking buys confidence rather than correctness."
Calibration tells a related story. "Most models are not well-calibrated at every effort level," the paper says, with reported confidence percentages drifting from actual correctness percentages. The errors are not independent either: "when one model errs in a task, several others tend to err on that same task," and on those shared-error tasks "a large number of them pick the same wrong product."
The paper's abstract names no specific models and publishes no per-model calibration figures.
Originally reported by paper
Read the original article →Original headline: More Reasoning Raises LLM Shopping Confidence, Not Accuracy, Across 10 Frontier Models