Nathan Lambert calls preference data post-training's black box

Every open-weights model release comes with weights, sometimes a paper, sometimes a data card. The one thing it almost never comes with is the human preference data used to shape the model's behavior, or the instructions that told the annotators what 'better' meant. Nathan Lambert's eighth lecture in the RLHF & Post-Training Course, published on YouTube and paired with the corresponding chapters of his RLHF book, walks through why. His flat claim, as of 2026: there are no open models with fully open human preference data released with the methods used to collect it. The largest public preference datasets he can point to are NVIDIA's HelpSteer2-Preference and HelpSteer3-Preference from the Nemotron team, and that is roughly the ceiling of transparency in the field.

The reason is partly economic and partly cultural. Lambert describes a typical smaller preference-data contract as running around $500K over roughly six weeks, with data arriving in weekly batches and quality expectations only settling in by weeks four or five. Bigger programs run into the millions. Access to vendors like Scale AI is, in his description, a who-you-know game where budget and brand get you to the front of the queue. Default terms on those contracts often prohibit open-sourcing the resulting artifacts, which is one plain reason the labs that pay for the data cannot share it even when they want to.

That opacity is not a rounding error, because the annotator instructions are effectively the model's rubric for what counts as a good answer. OpenAI's InstructGPT guidelines are the closest thing to a public reference, and Lambert treats them as the template everyone else emulates in private. He also documents ugly parts of the vendor market that rarely make it into papers: contracts that failed to deliver until the customer threatened legal or financial action, and vendors listing companies as customers for PR without actually working with them.

The honest caveat is that this is one practitioner's read of a market he is embedded in, and the specifics of any given contract are second-hand by design. What the lecture does not give you is a named list of who is buying from whom, or a look inside a frontier lab's live instruction document. What it does give you is a clear frame: if you are building on open weights and expecting reproducibility, the preference layer is the part you cannot see into, and that gap is where the model's real behavior was actually shaped.