SAP's EdgeGen Reports 2-42% Progress Lift on tau2bench Airline
TL;DR
- SAP's EdgeGen extracts compliance rules from an agent's specification, then synthesizes database-grounded tasks designed to violate them.
- Finetuning on EdgeGen data yields a 2-42% mean-progress improvement on tau2bench airline, while baseline methods degrade some models.
- On Gemma-4-e4b, EdgeGen reports 10% mean-progress gain over human-curated harnesses and 30% over base harnesses.
Finetuning on data generated by a new SAP framework called EdgeGen produces "a consistent mean progress improvement of 2 percent to 42 percent on tau2bench airline domain," according to the paper posted on Hugging Face. Baseline methods, the authors write, "show degradation for some models."
The pitch is that current synthetic task generators over-index on happy paths. EdgeGen instead "extracts compliance rules from an agent's specification and uses them to generate database-grounded edge-case tasks designed to violate these rules." The pipeline, the abstract adds, "forms a fully automated closed-loop system that requires no human annotation."
The second number the paper leans on is harness optimization on the Gemma-4-e4b model: "a mean progress improvement of 10 percent and 30 percent over the human-curated and base harnesses, respectively."
The abstract does not name the other models tested, break out per-rule accuracy, or report results beyond the airline domain. It arrives in a busy week for fine-tuning research on our tracker, where harness-delta framing has become the default way to score a small model.
Originally reported by huggingface.co
Read the original article →Original headline: SAP's EDGEGEN Auto-Generates Tool-Calling Agent Edge Cases, Reports 30% Progress Gain Over Base Harness