IG-Bench: Top LLM Scientist Reaches Just 27.3% on Idea Lineage

A new benchmark called IG-Bench, introduced in the paper Ideas Have Genomes on Hugging Face, tries to answer a question that most 'AI scientist' demos quietly skip: can a model actually trace how one research idea inherits from, mutates, or repairs an earlier one, and then generate a new proposal that fits that lineage. The headline result is sobering. Across 14 LLM-based scientist systems, the strongest reaches only 27.3% exact accuracy on lineage reasoning.

The framing is worth pausing on because it changes what 'good' looks like. Instead of grading a model on whether a single idea sounds novel, the authors model scientific works as Idea Genome objects with typed, evidence-grounded parts, and record how they inherit, mutate, drop features, import from outside, or recombine. On top of that they build IG-Exam, a closed-form test with 42 task types over 1,029 instances covering abstraction, inheritance tracing, evolutionary reasoning, and lineage verification, and IG-Arena, which uses a Population-Evolution Score to grade whether a proposed idea can plausibly serve as a descendant of a given parent set. The evaluation spans 10 scientific domains and 1,961 golden lineage traces.

The second finding is arguably more useful than the top-line number. The authors report that feeding models structured lineage context 'reshuffles system rankings rather than helping every participant uniformly.' In practical terms that means agent stacks tuned around one retrieval or scaffolding format can look great on a leaderboard and then regress when the context pipeline changes. Anyone shipping a research-assistant product on top of these agents should be running their own before-and-after checks rather than trusting a single benchmark row.

The honest caveat is that the paper page does not, at least on this Hugging Face view, break out which specific frontier models occupy the top of the 27.3% tier or how curation of the 10 domains and 1,961 traces was validated, so take the ordering as reported and treat the absolute score as a signal, not a verdict. Nor does the summary tell us how well the Population-Evolution Score tracks with expert human judgment.

The upside is that the field now has a shared, lineage-aware yardstick. Retrieval and context vendors have a clean target to aim at, evaluation teams have a way to pressure-test 'autonomous scientist' claims before signing procurement contracts, and groups training smaller open models around a Population-Evolution objective have something to beat that is not just another novelty rubric.