Orange's DeepStress Finds Search Agents Trust Fake Sources

Several open-weight search agents answer confidently when their retrieved documents come from domains sampled off a public list of misleading and outright fake websites. That is the headline finding of DeepStress, a stress-testing framework from Ismaël Rousseau at Orange Research in Lannion and Aix-Marseille, which replaces a search agent's retrieval module with a synthetic document generator the experimenter controls along three axes: trustworthiness, relevance and factuality.

Search-R1, a 7B Qwen2.5 agent trained with reinforcement learning, barely reacts. "Search-R1 remains between 78% and 80% correct across all trustworthiness levels, with no clear change in its abstention rate," the authors write. SpiqaDR, AutoRefine and ASearcher show the same pattern. The closed GPT-5 systems behave differently: GPT-5-mini (low) raises abstention from 14.0% to 22.5% as trust falls, while GPT-5-mini (high) hits its tool-call budget 62.7% of the time, up from 32.0%. DR-Tulu, an 8B open deep-research model, drops from 59.8% to 40.9% correct and its no-answer runs climb from 10.8% to 34.6%.

The HotpotQA result is stranger. At 0% factuality, where every returned document carries factual errors, gpt-5-mini (medium) still answers 60% of HotpotQA questions correctly, and every system clears 10%. On BrowseComp-Plus, built from a corpus too recent to have been pretrained on, the same setting drops performance to near zero. "This suggests that many HotpotQA questions can be answered from parametric knowledge," the authors note, and then say it plainly: "HotpotQA should not be used to evaluate search agents without a direct-inference baseline."

To rank systems under stress the paper introduces RAS, a Reliability-Aware Score that gives full credit for correct answers, none for confidently wrong ones, and partial credit for abstention when evidence is unreliable. Across 33,600 traces on 12 systems, two datasets and seven scenarios, average correctness at 0% document quality falls to 0.30 while average RAS stays at 0.61, with GPT-5-nano (medium) keeping a stable RAS even as its correctness slides from 0.83 to 0.41. Search-R1 drops in both, 0.80 to 0.35 correct and 0.80 to 0.37 RAS.

DeepStress joins a growing shelf of robustness probes — it's one of 26 hallucination-focused papers we've tracked here in the last 90 days, and lands alongside this week's Ai2 StateBench result. The code, the authors say, will be released after peer review.