Goldberg: supervised fine-tuning teaches LLMs to hallucinate

Yoav Goldberg argues in an April 2023 gist that supervised fine-tuning on human question-answer pairs is not merely slower than reinforcement learning for training language models. It actively teaches models to fabricate. The post draws on a talk by OpenAI's John Schulman.

The core move in the gist is a two-case split. When a model already "knows" an answer, supervised training on a demonstration reinforces that knowledge. When it does not, the same training pushes it to memorize or, worse, to generalize, learning to produce plausible-sounding false information. Trainers cannot inspect what the model knows, so the second case is unavoidable. "We want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains," Goldberg writes.

RL dodges this by scoring outputs rather than imitating them. Wrong answers get penalized, and a model can learn to abstain instead of guess, though Goldberg flags that preventing over-abstention remains an open research question. He splits interactions into three modes, text-grounded, knowledge-seeking, and creative, and argues the knowledge-seeking case is where supervised-only pipelines fail hardest.

The implication he draws for distillation is pointed. Training public models on GPT outputs via supervised methods inherits the fabrication failure mode without the RL scaffolding that disciplines it.