JHU paper lifts LLM long-horizon retention from 1.2% to 34.9%
TL;DR
- On 100 query-answer tasks learned in sequence, naive fine-tuning retains just 1.2% by the end; the best composed method reaches 34.9%.
- The recipe combines three anchor types (data, function, weight) with merged LoRA and lands in the top three methods on every dataset tested.
- Data anchor and merged LoRA interact super-additively across all three 100-task datasets, per the authors' factorial experiment.
A Johns Hopkins paper on arXiv puts a hard number on catastrophic forgetting. Learn 100 query-answer tasks in a row by continual supervised fine-tuning, with no task identifiers at inference and no earlier training examples retained, and average final retention lands at 1.2%. Compose the right mechanisms and it climbs to 34.9%.
The authors, Zheyuan Zhang, Alvin Zhang, Daniel Khashabi and Tianmin Shu, organize continual-learning tricks along two axes: data, function and weight anchors for what prior information to preserve, and low-rank allocation rules for where successive updates live. They search the combinations with task-level successive halving and a factorial experiment across three 100-task memorization datasets. Their best method combines all three anchors with merged LoRA and "raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement."
The punchline is interaction, not any single trick. "No single continual learning mechanism we evaluate maintains strong retention at this horizon," the paper reports, and the data anchor and merged LoRA "interact super-additively on all three datasets." The abstract offers no per-dataset breakdown beyond the 34.9% average and does not name the base models used.
Originally reported by paper
Read the original article →Original headline: JHU Raises LLM Sequential Retention 28x by Composing Three CL Mechanisms