Salganik team: ML poorly predicts 484 life outcomes in survey

Even with up to 15,160 machine learning models per outcome, most life outcomes in a long-running U.S. child wellbeing survey came out stubbornly hard to predict. In a new arXiv preprint, Emily M. Cantrell, Pranay Anchuri, Hanzhang Ren, and Matthew J. Salganik report a median cross-validated R² of 0.03 across 484 outcomes drawn from the Future of Families and Child Wellbeing Study. Two researchers we track had the preprint circulating within days of its posting.

The team worked from the Future of Families and Child Wellbeing Study, which the paper describes as 'a probability sample of children born in large U.S. cities between 1998 and 2000.' To guard against overfitting, they ran all infrastructure development and reported results on just 2,121 participants, half of the full 4,242.

Where the models did work, the ceiling was still modest. The best-predicted outcome, whether a biological primary caregiver was married to the biological parent, reached an R²CV of 0.63; youth height in centimeters hit 0.62. And even there, the authors write that 'even for the most predictable outcomes, our complex prediction procedure typically exhibited little skill relative to a simple procedure.'

'Most outcomes were not very predictable, but some were much more predictable than others,' they write, framing the three findings the paper lays out. The third was 'modest patterns in how predictability varied across outcomes.'