Nathan Lambert Maps the Three Eras of Post-Training Evals

Nathan Lambert's twelfth lecture in the RLHF & Post-Training Course is a stock-take on the thing everyone in the field quietly worries about, whether the benchmarks we cite still measure anything useful. The answer, drawn from Chapter 16 of the accompanying book, is that they measure something, but the something has moved three times.

Lambert sketches three eras. The early chat phase leaned on MT-Bench, AlpacaEval and Arena-Hard, using models like GPT-4 as judges of conversation quality. That gave way to a multi-skill era, where suites like the one used for Tülu covered MMLU, TruthfulQA, GSM8K, HumanEval and the rest, treating knowledge, reasoning, math, coding and safety as separate axes. The current era, in his framing, is defined by reasoning and tool use, and the benchmarks that get quoted have shifted accordingly to GPQA Diamond, Humanity's Last Exam, SWE-Bench+ and LiveCodeBench.

The part of the lecture worth pausing on is the honest section on noise. Lambert notes that post-training evaluations in the reasoning era show between 0.25 and 1.5 point standard deviations even when the evaluation setup is held constant, because labs use different prompting strategies and sampling parameters. That is a lot of room for a leaderboard shuffle. Add contamination, with documented cases of UltraFeedback contaminated with TruthfulQA, Evol-CodeAlpaca with HumanEval and NuminaMath with MATH, and the confidence you should put in any single-point comparison drops further.

The agentic turn changes the evaluation stack itself. A model is no longer a text-in-text-out box, it needs, in his description, a harness that manages context, tools and credentials, plus a sandbox with the files and rules that make a task reproducible. That means CPUs alongside the GPUs, which is not a detail leaderboard tables usually surface. Terminal-Bench is called out as the most popular agentic standard so far.

The honest caveat is that this is a lecture summary rather than a report of new results, so the specifics you should carry away are the framing and the noise numbers, not fresh rankings. What the lecture does not give you is a prescription for what any one team should adopt next, or a good answer for how to evaluate outputs too long or complex for a human to reliably grade. Anyone building an evaluation pipeline, or buying models on the basis of one, has to sit with those two gaps.