CrossProfile paper: memory heuristics fail LLM group calls

Memory filtering strategies central to multi-agent LLM pipelines fail at the task they were designed for, according to a paper from Anouar Imel, Imad Charradi, Abderrahman Sghir and Khalil El Khammar. In the hidden-profile configuration of the authors' CrossProfile benchmark, picking memory by relevance loses "23 to 31 points to rarest-first selection in the classic hidden-profile configuration," but rarest-first itself does no better than random on average.

The paper, posted on Hugging Face on October 27, introduces CrossProfile as 60 templates and 480 tasks with automatically verified answer keys. Ownership of supporting, decoy and irrelevant facts is crossed while text remains fixed. The authors run Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models through the pipeline, placing the result alongside other recent memory-and-retrieval work we've tracked such as a LASAGNA study that trimmed advertised RAG gains.

Why relevance fails is simple. "Selecting by relevance readmits decoys, because decoys are on topic," the paper notes. The decision model also inherits "a form of social proof: it gives extra weight to facts attributed to many agents." Removing the attribution tags, or reassigning them at random with line order fixed, "raises accuracy in the classic configuration by 13 to 24 points."

The counter-intuitive finding sits on full-transcript memory: "the complete transcript yields lower accuracy than no notes at all." The policy that kept the evidence mattered less than what the memory happened to keep. "Accuracy follows the balance of evidence the memory keeps, not the policy that kept it," the paper concludes.