Taste-Bench Puts Best LLM Agent at 59.7% on Decision Forks

The best frontier model tested on Taste-Bench answered only 59.7% of the questions correctly. That is the headline number from "The Tasteful Agent", a paper posted to arXiv and surfaced on Hugging Face's daily papers list, which sets out to measure something long-horizon agent benchmarks have mostly skipped: whether the agent picks the right branch when it hits a fork.

The authors call this quality "the taste of an agent." They build the benchmark by mining trajectories that agents actually produced on engineering and research tasks. "Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork," the abstract states. The forks come "from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation."

Two findings sting. "Forks whose deciding evidence appears later in the trajectory are much harder for every model," the paper reports, and "a larger reasoning budget does not improve the accuracy." Thinking longer, on the current frontier, does not buy better judgment.

The more constructive claim is that taste can be trained. The authors distill a teacher "that has seen the outcome" into a student, and report the student "improves end-to-end success on held-out SWE-bench Pro tasks." The abstract names no specific frontier model behind the 59.7% figure and gives no teacher or student sizes.