MAPD Distills Claude, GPT, Gemini Into Qwen3 for $1,454

A team spanning Beijing Institute of Technology, Tsinghua, East China Normal, USTC and Chinese Academy of Sciences labs posted a paper to Hugging Face that squeezes the reasoning of Claude-Opus-4.6, GPT-5.5 and Gemini-3.1-Pro into Qwen3-1.7B and Qwen3-4B, and does the whole distillation for roughly $1,454 of teacher API spend.

The trick is not more reinforcement learning, it is an intermediate representation. Rather than force the small student to imitate the verbose free-form trajectories of a proprietary teacher, which the authors say induces severe style drift and hallucinations, Multi-Agent Protocol Distillation runs an offline multi-agent system powered by the closed model, decomposes each query, retrieves evidence, repairs failed searches, and converts the whole exploration trace into a style-normalized JSON protocol with fields for task type, reasoning plan, and extractive grounding facts. During training that protocol is fed only to a privileged branch of the student, and the two branches are aligned via token-level self-distillation on top of a GRPO reinforcement-learning objective.

Across seven knowledge-intensive QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle) the paper reports average success rates of 39.4% on Qwen3-1.7B and 44.4% on Qwen3-4B, which the authors call relative average improvements of 4.8% and 3.3% over the SDAR baseline. The multi-hop gains are noticeably bigger than the single-hop ones, 7.9% versus 2.3% on the 1.7B model, which fits the pitch that decomposition and evidence aggregation is where the closed teachers earn their keep. Using Gemini-3.1-Pro as the reference, the cost works out to $0.057 per training instance and roughly 6.3 teacher calls consuming about 12.5K tokens per instance, for 25,584 valid protocols out of 25,600 attempted.

The honest caveats are stacked in the paper itself. Pure online self-distillation without the external protocol collapses to 5.9% on the 1.7B student and 20.9% on the 4B, and pushing the distillation weight too hard forces the model into a retrieve-without-reasoning shortcut that tanks multi-hop scores even as single-hop numbers tick up. The benchmarks are all wiki-retrieval QA, and the agent is capped at four interaction turns and 512 response tokens, so nothing here proves the recipe survives longer horizons or noisier retrieval than a local Wikipedia dump.

What is interesting for anyone shipping small open-source agents is the transferability claim. Swap Claude for GPT for Gemini and the numbers move by under two points, so a team can arbitrage API cost and latency without retuning the pipeline, and the resulting student needs no proprietary API at inference. If it holds up outside these seven benchmarks, the price of a competitive open-source search agent just dropped to about the cost of a mid-range laptop.