HF Paper 'CausalDS': 953-Scene Causal-Reasoning Benchmark for Data-Science Agents — Claude Opus 4.8 Leads at 0.278 Overall Score / 82.4% Pass Rate, Frontier Models Beat Open-Weight Ones Mostly on Abstention
Summary
CausalDS is a new benchmark of 953 synthetic scenes with hidden structural causal models paired with natural-language stories and observational data, covering all three rungs of Pearl's causal inference plus abstention-aware scoring. Results show frontier models (Claude Opus 4.8, Gemini 3.1 Pro) beat open-weight models chiefly on uncertainty quantification and abstention; Claude Opus 4.8 leads with 0.278 overall score and an 82.4% pass rate.
Originally reported by huggingface.co
Read the original article →Original headline: HF Paper 'CausalDS': 953-Scene Causal-Reasoning Benchmark for Data-Science Agents — Claude Opus 4.8 Leads at 0.278 Overall Score / 82.4% Pass Rate, Frontier Models Beat Open-Weight Ones Mostly on Abstention