Google DeepMind launches 200-task Multimodal Knowledge Agent benchmark

Google DeepMind on November 10 released the Multimodal Knowledge Agent benchmark, built with Artificial Analysis, and used it to place Gemini 4 Carbon second in quality at six times lower cost on average than the top-scoring model.

The pitch is that existing agent tests are too text-heavy. "Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout," authors Rohan Doshi and Ben Caine write on the Google blog. "Yet, everyday professional work looks very different." MKA packages 200 tasks across 10 verticals and 83 occupations, with domain experts ranging from architects and biologists to underwriters and data scientists. "Each task reflects a multi-step project that takes a seasoned professional three to eight hours to complete, pairing a realistic prompt and a bundle of multimodal input files with an expert-written rubric of roughly 70 yes/no checks," the post says. Scoring is weighted 75% factual accuracy and 25% visual design.

Every model runs inside the same open-source agent harness, called Stirrup, with a shared toolset and turn budget. On that harness the tested models "have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task." An agentic autorater grades outputs; when domain experts checked its work, they agreed 96% of the time.

The full 200-task set stays private with Artificial Analysis. Ten sample tasks are posted on Hugging Face. The post does not name the model that beat Gemini 4 Carbon on quality, and it is the third Google alert on our tracker today, following the wider Gemini 4 Carbon API rollout.