Be.FM-1.5 leads BehaviorBench on population-level alignment

A new benchmark out this month from Jin Huang and co-authors including Matthew O. Jackson and Qiaozhu Mei tries to answer a question the behavioral science crowd has been asking for a while: are general-purpose foundation models actually good at the work behavioral scientists hire them for, or do they just look good on the cherry-picked tasks the demos lean on.

The answer in the BehaviorBench paper on arXiv is a careful split. The authors set up four competencies meant to cover the real uses: behavior prediction and simulation, strategic decision-making, subject-trait inference, and behavioral knowledge application. What makes the benchmark different from a typical LLM eval is that it scores models at two levels. Per-subject accuracy is the familiar one. The newer one, which the paper calls "an essential requirement for behavioral validity," is population-level alignment, whether the distribution of model responses matches the distribution of real people's responses in aggregate.

That second metric is where the headline finding sits. The authors report that "proprietary general-purpose models excel at individual-level prediction and knowledge-intensive tasks, whereas behavioral foundation models, fine-tuned on behavioral data, achieve substantially stronger distributional alignment." To make the point concretely they ship Be.FM-1.5, an extension of the existing Be.FM family, and, per the paper, "fine-tuned on data that is held-out from BehaviorBench." Be.FM-1.5 leads on the distributional metrics while staying competitive on the individual ones, which the authors read as evidence that "proper behavioral adaptation can close the gap."

The honest caveat is that the abstract is the abstract. It does not name the specific proprietary models that lost the distributional comparison, it does not put numbers on "substantially stronger," and it acknowledges its own ceiling with the line "no single model family dominates the full benchmark." Distributional alignment is also only as meaningful as the populations it is measured against, and the paper, in what was retrieved, does not litigate which populations those are.

Still, the framing matters more than the leaderboard. If you are building anything that uses a model to stand in for a population, agent-based market simulations, synthetic survey panels, policy preference models, the metric that should keep you up at night is not whether each answer looks right but whether the spread of answers matches reality. BehaviorBench is the first benchmark I have seen that treats that as the headline rather than a footnote.