Rules beat few-shot examples for LLM in-context learning
TL;DR
- Across five tasks spanning games, arithmetic and linguistic inference, LLMs learned more reliably from written rules than from input-output examples.
- Stacking more examples on top of rules produced no consistent gains, while instruction tuning amplified the rule-based learning advantage.
- The rule advantage was largest on tasks recruiting algebraic abstractions and smallest on tasks requiring distributional sensitivity or parametric knowledge.
Large language models learn more reliably from written rules than from input-output examples, according to a paper by Xiang Fu, Seungmin Cho, Yukyung Lee and Najoung Kim covering five in-context learning tasks across games, arithmetic and linguistic inference.
"Models generally learn more reliably from rules than from examples alone," the authors write, and stacking more examples on top of rules did not consistently improve performance.
The size of the gap depends on task shape. "The comparative efficacy of rules over examples is larger when the task recruits algebraic abstractions and computations, and smaller when the task requires distributional sensitivity," the paper reports. Instruction tuning "amplifies the benefit of rule-based learning while keeping example-based learning capacities intact," and base models showed no built-in preference for examples either.
Two of the researchers we follow shared the paper's link.
Shared on Bluesky by 2 AI experts
-
Najoung Kim @najoung.bsky.social: support presenting rule- vs. example-based in LLMs: arxiv.org/abs/2609.03213 →
Originally reported by arxiv.org
Read the original article →Original headline: LLMs Learn Better In-Context from Rules than from Examples