UpgradeBench tests when to port, refresh or retrain LLM adapters
TL;DR
- A fixed decision policy over 33 upgrade episodes yielded 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining.
- On OLMo, direct adapter copying retention fell from 0.88-0.99 at 46B tokens of continued pretraining to zero at 2.9T tokens.
- A lightweight CKA probe over 256 prompts predicted cross-version adapter portability at Spearman 0.74 across eight model pairs.
"On OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens." That is the finding at the heart of UpgradeBench, a decision-centric benchmark from Ye Chen and Weining Zhang that asks a plain operational question: when a base model ships a new checkpoint, what should organizations do with the specialists they fine-tuned on the old one?
The benchmark spans four consecutive Qwen releases, one continuation checkpoint, six tasks and two model sizes, augmented by OLMo checkpoints with known training lineage. Durability varies sharply by task. Text-to-SQL specialists last "under one release interval." Intent classification holds "over fourteen months."
The authors then simulate a fixed decision policy across 33 upgrade episodes. The paper reports "0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining." A lightweight CKA probe over 256 prompts predicted cross-version adapter portability at Spearman 0.74 across eight model pairs.
The abstract hedges on the recovery route. Teacher relabeling can rebuild target-base specialists without fresh gold annotations, "though compute savings are not guaranteed." Two of the researchers we follow surfaced the paper in our tracker.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists