I saw a bunch of people dragging this study based on the recruitment emails back in the spring. But it seems to have resulted in empirical evidence for exactly the critiques those people wanted to make! arxiv.org/abs/2606.08251
AI Weekly's analysis
→
- No model class tested — reasoning or non-reasoning — spontaneously proposed null hypotheses, a move scientists make more freely.
- Non-reasoning LLMs converge into a 'hivemind' of similar ideas; LLMs struggle most in pluralistic fields like the social sciences.
- The study drew 25,139 rating sets from 6,749 scientists across biology, medicine, chemistry, and the social sciences.
Read full analysis →