FiC self-distillation lifts LLM multi-turn accuracy to 92-100%
TL;DR
- A Stanford team's FiC framework has small-to-mid LLMs teach themselves to recover from the 'Lost-in-Conversation' accuracy drop on sharded multi-turn prompts.
- Across Llama, Qwen, Phi and OLMo backbones from 3B to 14B parameters, FiC recovers 92-100% of single-turn performance and hits 100% on two Llama models.
- Training is cheap: 0.7 to 6.0 A100-hours per model, with up to 33% fewer tokens needed to resolve the conversation.
Large language models can recover 92-100% of their single-turn accuracy on the same information split across multiple conversation turns, according to a new arXiv paper from Stanford researchers Tianlang Chen, Shirley Wu, and Jure Leskovec. Their framework, Found in Conversation (FiC), targets what recent work has dubbed the 'Lost-in-Conversation' problem.
The core move is what the authors call View-Asymmetric Self-Distillation: the same model runs twice on information-equivalent inputs, once in a single-turn concatenated view (the teacher) and once in a sharded multi-turn view (the student). 'The teacher–student gap is exactly the multi-turn-vs-single-turn gap we want to close, with no other distributional difference,' the paper notes.
Results are reported on math (GSM8K), database queries (Spider), and function calling (Berkeley Function Calling Leaderboard), across Llama-3.2-3B, Llama-3.1-8B, Qwen2.5-7B, OLMo-2-13B, and Phi-4-14B. FiC reaches 100% of the single-turn baseline on two Llama backbones and cuts conversation length by up to 33%. The authors put the compute overhead at 0.7 to 6.0 A100-hours per model.
The paper is slated for NeurIPS 2026, with a revised version posted on November 12. Two researchers we track flagged it the same week.
Shared on Bluesky by 2 AI experts
Originally reported by arxiv.org
Read the original article →Original headline: Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap