AltRA-Test checks when LLM judges can replace human raters
TL;DR
- AltRA-Test is a bootstrap-based statistical test for whether an LLM can validly replace human annotators, without requiring shared annotators on every item.
- Across 18 LLMs and 20 datasets, LLMs pass in 35.3% of setups under default chain-of-thought prompting and 47.3% with the best prompting technique.
- About 100 items typically suffice; Gemini-3.1-Pro clears 16 of 20 datasets, while some tasks admit no valid LLM replacement at all.
A new statistical test proposes to settle when large language models can validly stand in for human annotators, and reports they pass "in about half of the setups" under favorable prompting. Under default chain-of-thought prompting, that rate falls to about one in three.
The method, called AltRA-Test for Alternative Annotator Test with Resampled Annotations, is laid out in a preprint on Hugging Face by researchers at the Technion, the University of Haifa, and Google Research. It extends a prior Alt-Test that required every item to be labeled by the same annotators. The new version "does not require that all items are annotated by the same annotators," which is what makes it workable for the messy annotator subsets typical of large crowdsourced datasets.
The procedure runs a bootstrap that resamples items, draws a panel of human annotations, then substitutes one with an LLM annotation to create paired all-human and mixed human-LLM panels. Replacement is supported "when the LLM preserves the signal that the human annotators share." Across 18 LLMs and 5 prompting techniques on 20 datasets spanning text, image, audio and video, "about 100 items often suffice for the test."
Pass rates vary sharply by task. The paper reports 35.3% under default chain-of-thought prompting and 47.3% under the best prompting technique. Gemini-3.1-Pro clears the test on 16 of 20 datasets, while every tested model fails on datasets including Persuasion and Measuring-Hate-Speech.
"Since replacement is not always justified, we encourage researchers to test it for their task," the authors write.
Originally reported by huggingface.co
Read the original article →Original headline: Google Research Ships AltRA-Test to Decide When LLM Judges Can Validly Replace Human Annotators