'Context Language Models' let LLMs edit their own context file
TL;DR
- Context Language Models (CLMs) treat context as a file the model itself updates, replacing externally-coded context-pruning strategies.
- Zero-shot, the authors report 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus against SOTA context-management baselines.
- On a 24-hour multi-repository agent-swarm task, CLMs show 65% greater improvement at matched compute versus existing approaches.
A team led by Rulin Shao has posted a preprint proposing that language models manage their own context by treating it as a file and making unrestricted updates to it. Zero-shot against existing models, the abstract reports 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on a 12-hour EdgeBench task, and 65% greater improvement at matched compute on a 24-hour multi-repository agent-swarm run.
"This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files," the paper states. Co-authors include Mike Lewis, Luke Zettlemoyer, Wen-tau Yih, Hamish Ivison, Nathan Lambert and Pang Wei Koh; code is published at a facebookresearch repository. Two of the researchers we follow in our Who's Who directory passed the link around on submission day.
Baseline numbers for the SOTA context-management strategies these deltas are measured against are not included in the abstract.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Context Language Models