InstructGPT: 1.3B RLHF model preferred over 175B GPT-3
TL;DR
- Human evaluators preferred the 1.3B-parameter InstructGPT over the 175B GPT-3, despite the smaller model having 100x fewer parameters.
- The method combines supervised fine-tuning on labeler-written demonstrations with reinforcement learning from rankings of model outputs.
- The authors report improvements in truthfulness and reductions in toxic output, with minimal regressions on public NLP datasets.
OpenAI researchers reported that a 1.3-billion-parameter language model trained with human feedback was preferred to the 175-billion-parameter GPT-3 in human evaluations on their prompt distribution. That is a roughly 100x gap in parameters.
The two-stage recipe is laid out in the March 2022 arxiv preprint. Labelers first wrote demonstrations of the desired model behavior, which the team used to fine-tune GPT-3 with supervised learning. They then collected rankings of model outputs and used reinforcement learning from human feedback to fine-tune the supervised model further. They called the resulting models InstructGPT.
"Making language models bigger does not inherently make them better at following a user's intent," the paper opens. The authors report "improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets," while adding that "InstructGPT still makes simple mistakes." The abstract publishes no per-task numbers and does not state how many labelers were involved.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Training language models to follow instructions with human feedback