J-Zero co-evolves a Judge, gains 8 points on unverifiable tasks

A new arxiv preprint from KAIST reports that adding the reward model itself as a third co-training player lets a self-play loop keep improving in domains where the answer cannot be checked. The authors — Gyouk Chu, Myeongho Jeon and Eunho Yang — call the system J-Zero, and claim an 8.0-point average gain on unverifiable benchmarks where prior zero-data systems flatten out after two rounds.

The setup runs three roles against each other. A Challenger generates harder tasks, a Solver answers them, and a Judge "co-adapts using preference pairs whose ordering is known in advance from how each response was produced." That last piece is the departure from R-Zero and similar frameworks, which lean on a fixed reward model or a verifiable checker; here the Judge is trained inside the loop from structural asymmetries in how each candidate was generated, with no human labels.

On Qwen3-4B-Base and Qwen3-8B-Base, using Skywork-Reward-V2-Llama-3.1-8B as the starting Judge, the paper reports AlpacaEval 2.0 moving from 6.22 to 28.56 on the 4B model and from 12.93 to 33.53 on the 8B, alongside gains on Arena-Hard-v2.0 and EQ-Bench Creative Writing v3. On the verifiable side — GSM8K, MATH500, Minerva, OlympiadBench, AMC23, AIME24 and AIME25 — J-Zero improves by 4.2 points on average and "outperforms R-Zero by 4.74 and 3.56 points on Qwen3-4B and 8B respectively."

The authors flag Judge co-evolution as the reason gains persist: J-Zero "continues to improve through at least ten iterations, whereas the baselines degrade after two." The abstract does not publish per-iteration Judge accuracy numbers, and the reported unverifiable scores rely on the same family of automated judges the community has been debating for a year.