MPIE-Bench Exposes Multi-Person Editing Anatomy Failures

Text-to-image models can drop your friend's face into a beach scene now, but ask two named people to embrace, carry a child, or grapple and the result gets weird in a specific way: hands fuse into one another, extra feet appear along the contact line, and one body occupies the volume of the next. A team from the University of Science and Technology of China and Metastone Technology argues every current benchmark misses this failure mode, because the standard grader is a vision-language model reading a checklist, and that checklist saturates near ceiling on images a human would flag on sight.

Their MPIE-Bench paper on Hugging Face puts numbers behind the gap. Across ten editors, seven open source plus GPT-Image-2, Gemini 3 Pro Image, and Seedream 5 Pro at vendor defaults from 2026-07, the VLM-as-judge Interaction score sits at 0.98 to 0.99, while the paper's mesh-based Interaction score for the same images spans 0.45 to 0.72. No single editor leads on both new axes: Gemini tops mesh Anatomy at 0.65, Seedream tops mesh Interaction at 0.72. Identity, meanwhile, is where closed models still walk away from open ones, roughly 0.49 to 0.58 versus 0.02 to 0.21.

The evaluation trick is to reconstruct every person in the generated image with a frozen Multi-HMR checkpoint, then read two geometric questions off the resulting meshes: is every human-like blob explained by a complete body, and does the penetration and surface distance between bodies match the contact the prompt asked for. The dataset itself is 2,500 triplets over 405 scenes, mined from Pexels stock footage, Harmony4D, and CHI3D, and stratified across 14 interaction categories and four contact densities from C0 to C3. A five-rater study on 170 hard contact units found the mesh score tracked people more closely than a zero-shot VLM judge on 9 of 10 checklist items.

The honest caveat is that the mesh score depends on a public reconstruction model with its own error budget, and the closed API results are frozen to July 2026 defaults that will drift. Absolute human agreement was only moderate, and the authors themselves frame the mesh axes as ranking observables rather than a per-image oracle. What the paper does not give you is a clean read on how much of the Anatomy gap is editor failure versus Multi-HMR noise on the same generations.

The practical takeaway for anyone shipping or buying a multi-person image editor is straightforward: the score in the sales deck almost certainly came from a VLM checklist that will not surface fused limbs, and there is now a public, geometry-based grader that will. Open-source labs get a track where competing on Anatomy does not require also solving identity binding, and integrators finally get an audit lever the vendors do not control.