UserIDA splits user intent from wording, hits 86.6% on LMSYS-USP

User simulators are the load-bearing infrastructure behind most modern assistant training and evaluation, and they mostly work by imitating what a plausible next user turn would look like. That is where the arXiv paper behind UserIDA locates the problem: a simulator that sounds like a user can still steer the conversation the wrong way, for example by accepting an assistant answer that violates a stated constraint instead of pushing back to repair it. The authors, Bo Wang, Ruixing Zhang and colleagues, argue that separating which local action a user turn should perform from how it is worded is a missing per-turn control variable, and that assistant weaknesses in clarification and error recovery get masked when the simulator itself does not surface that choice.

Their fix is UserIDA (User Intent-Directive Alignment), which defines a six-way directive taxonomy covering Initiate, Amend, Supply, Repair, SetRegister and GroundAccept, described as how a user creates, updates, repairs, regulates, or closes the local interaction state. The simulator is first trained via supervised fine-tuning conditioned on the target directive, then run through what the authors call intent-calibrated policy optimization, a group-based RL step whose reward places intent-violating candidates below all compliant alternatives while retaining quality discrimination among the compliant ones.

On LMSYS-USP, the reported turn-level intent accuracy is 86.6%, which the paper says exceeds the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, where the evaluator forces a specific intent mid-dialogue, UserIDA realizes at least four of the six target intents in 91.7% of evaluated dialogue states, against 22.9% for the strongest external baseline, and all-turn intent success in controlled-prefix trajectories reportedly rises from 13% to 58%. Code is on the team's GitHub repo.

The paper is careful to describe its six intents as a primary control interface rather than a complete multidimensional dialogue-act ontology, and the sections here do not name the base model the simulator was fine-tuned from or the compute cost of the RL step. Evaluation also does not extend beyond LMSYS-USP conversations to task-oriented settings like booking or coding, which sit alongside a broader run of similar simulator work in our research feed.

For assistant teams whose evaluation harnesses only measure whether the simulated user sounds right, a controllable intent knob is the more useful primitive: you can finally check whether your assistant reliably handles a Repair turn, not just an average one.