IdeaTaste finds LLMs match 18% of human research ideas

LLM-generated research proposals matched what academic communities actually published in 2026 only 18% of the time at the level of ideas, and 26% of the time on citations. That is the headline result of IdeaTaste, a benchmark from Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar and Chenyan Xiong, which ran six language models across four research areas and scored more than 90,000 proposals in total.

The setup is the point. "We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026," the authors write. Instead of asking human judges whether a machine-written proposal sounds plausible, the researchers withheld a year of real publications and used them as the ground truth.

The models were not uniformly weak. The paper reports that "the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue." Expert reviewers backed that up: "AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst."

The paper names neither the six models nor the four research areas in its abstract. The authors say they are maintaining an open, continuously updated leaderboard so future publication cycles can keep scoring new models.