paper web signal

ResearchStudio-Idea mines 1,947 papers into 15 ideation cards

TL;DR

  • IdeaSpark was built from 1,947 ICLR, ICML, and NeurIPS papers between 2021 and 2025, including Orals, a high-citation subset, and rejected submissions.
  • Analysis surfaced 31 recurring ideation sub-patterns, consolidated into 15 reusable pattern cards with contexts, bottleneck types, differentiation strategies, precedents, and failure modes.
  • Blind automated-judge evaluations report IdeaSpark produces stronger proposals than no-skill and generic-skill baselines while maintaining competitive novelty.

There is a version of LLM-driven research ideation most people who have tried it will recognise: ask a model for directions in your field, get back plausible-sounding proposals that could have been written five years ago, and struggle to tell the interesting ones from the recycled ones. A new paper on arXiv from Qihao Zhao and colleagues tries to fix the front end of that problem by mining real conference outcomes instead of trusting a model to know what a good idea looks like.

The construction is worth pausing on. The authors collected 1,947 machine learning conference papers from ICLR, ICML, and NeurIPS between 2021 and 2025, including Oral papers, a separately tracked high-citation subset, and rejected submissions. From that corpus they identified 31 recurring ideation sub-patterns and consolidated them into 15 reusable ideation patterns. Each pattern is operationalized as a structured card containing research contexts, bottleneck types, differentiation strategies, supporting precedents, and common failure modes. The suite itself ships as three skills: Paper-Search for multi-source literature search, Scoop-Check as a standalone prior-art collision checker for novelty claims, and IdeaSpark, the end-to-end workflow that composes evidence grounding, pattern-guided generation, collision retrieval, audit, and idea-card rendering.

Why this matters if you are not writing conference papers: the shape here, a small library of pattern cards with named failure modes and precedents, applied as an audited workflow instead of a single prompt, is a genuinely useful template for any setting where an LLM is being asked to do work that senior humans normally scaffold. Graduate students without a senior advisor and small teams trying to industrialise research ideation are the natural beneficiaries.

The honest caveat is that the reported wins are blind automated-judge evaluations against no-skill and generic-skill baselines, and the paper's own claim is competitive novelty, not superior novelty. That is not the same as a working researcher preferring IdeaSpark's proposals in their own field, and an LLM judge can be sympathetic to output that reads like the papers it was trained on. What the arXiv writeup does not give you is which base model powers IdeaSpark, how the 31 sub-patterns were coded out of the corpus, or whether the same card structure transfers outside machine learning.

The interesting downstream move, if the format holds up under human review, is not more ideation bots but porting the same auditable card workflow into research fields where strategy is even harder to hand down.