Alibaba MerchantBench: LLMs hit 27.3% of human net assets

There is a whole category of AI benchmarks where a model gets one question, gives one answer, and gets scored. MerchantBench, from a team led by Qiming Shi in a new paper on arXiv, is not that. It drops language model agents into a 365-day order-level simulation of running a wholesale e-commerce shop, grounded in 98,843 real product records from 36,576 suppliers on Alibaba's 1688 marketplace, with 26 tools to work with.

The results are the interesting part. Across eight frontier LLMs, GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max and Plus, GLM-5.2, DeepSeek-V4-Pro and Flash, and Kimi K2.6, run under two agent frameworks (ReAct and Hermes) across 48 full-year runs, the best configuration reached only 27.3% of the mean final net assets of the human baseline. That baseline was, notably, three participants with no prior e-commerce operating experience. A year of compounding simulated decisions is enough to separate today's frontier models from novice humans by roughly a factor of four in end-state cash.

Why that gap is more interesting than the headline number: the benchmark's four sub-tasks are product sourcing, listing and pricing control, cash-flow management, and what the authors call mixed-latency feedback adaptation. That last one is the crux. In this world some actions have immediate effects (an upstream supplier event) and some take days to resolve (a downstream order outcome). A model that is great at single-turn tool use can still bleed cash if it fails to carry state across weeks. The gap is not raw reasoning, it is coherence over time when today's action constrains tomorrow's option set.

The honest caveats worth naming. The human baseline is three people, not thirty, and they were novices rather than experienced merchants, so the 27.3% is a signal, not a settled scoreboard. The eight-model roster is a snapshot and the paper does not tell you exactly which failure modes drove the gap, or how much a purpose-built merchant-agent scaffold would close it beyond generic ReAct or Hermes. Nothing in the reporting says the same ordering would hold on Amazon or Shopify data rather than 1688 wholesale.

What to watch for is whoever ships an agent that can actually run a year of a small business without a human unwinding its mistakes each week. That is the capability MerchantBench is trying to measure, and it is the one that turns AI agents into a real operations tool rather than a demo.