LLM Agents Pick Weaker Items From Favored Sources, Paper Finds

Across 12 agent models tested on product shopping, hotel booking, and paper citation, an LLM agent picked an item satisfying one requirement fewer about two-thirds of the time if the weaker item came from a source the model preferred, according to a preprint posted to arXiv.

"An item satisfying one requirement fewer is selected about two-thirds of the time when it comes from a preferred source and the better one from a dispreferred source, but almost never in the reverse case," the authors write. Across the models they tested, each "prefers some sources and avoids others in every domain, largely agreeing on which."

Source identity drives the pattern on its own. Hiding it weakens the preference. Relabeling an item with a preferred-source tag raises its selection rate. The paper points to two routes: training that rewards better items can make a source "a shortcut for requirement satisfaction," and missing information can trigger preconceptions about where an item comes from. Supplying the missing details, or a prompt that counters those preconceptions, reduces the effect.

The abstract does not name the 12 models, the preferred and dispreferred sources, or the per-domain breakdown.