ToolArtist trains one agent policy for reasoning, tools, images

Most agentic image systems today are hand-wired pipelines: a planner sends a prompt to a text-to-image model, maybe calls a search API for a reference, maybe hands off to an editor for a fix-up pass. ToolArtist, described in a new arxiv paper from Jiahao Zhao and collaborators, tries to fold all of that into a single trained agent policy that decides for itself when to reason, when to call an external tool, and when to draw.

The recipe has two stages. First, supervised fine-tuning on trajectories from a teacher agent that already has access to search and image-generation tools. Then a reinforcement-learning stage the authors call Reason-Act-Draw GRPO, or RAD-GRPO, which uses what they describe as complementary intent and quality rewards to jointly optimize the model. The tool-heavy trajectories are then converted into a Unified Multimodal Model format, with the image-generation tool concealed so the same model can produce images natively rather than always calling out.

The headline claim, as the paper puts it, is that placing the entire open-world image-generation process under an agent policy 'consistently outperforms approaches with fixed pipelines or only partially agent-controlled components.' The authors also commit to releasing the training data and the complete post-training infrastructure, which if it actually lands would give smaller labs a reproducible baseline for tool-augmented multimodal RL rather than another closed result.

The honest caveat is that the wording is qualitative. What the retrieved page does not spell out is which benchmarks were used, the exact margins, the model size, or how much latency the agent loop adds per image, so take the 'consistently outperforms' framing as the authors' claim rather than a settled result. If the open release ships and the numbers hold, the group most exposed is the one currently maintaining stitched-together image-agent pipelines, whose orchestration layer starts to look like something a single fine-tuned model could subsume.