Google DeepMind launches 200-task Multimodal Knowledge Agent benchmark
TL;DR
- Artificial Analysis's independent leaderboard puts Claude Fable 5.2 first on quality at 57% Climb WSR, with Gemini 4 Carbon 6x cheaper per task.
- All 200 tasks are private with no public answer key, a contamination-prevention design that distinguishes MKA from most open evals.
- Visual design quality accounts for roughly 25% of total scoring, penalizing models that produce correct but poorly formatted professional deliverables.
Google DeepMind on November 10 released the Multimodal Knowledge Agent benchmark, built with Artificial Analysis, and used it to place Gemini 4 Carbon second in quality at six times lower cost on average than the top-scoring model.
The pitch is that existing agent tests are too text-heavy. "Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout," authors Rohan Doshi and Ben Caine write on the Google blog. "Yet, everyday professional work looks very different." MKA packages 200 tasks across 10 verticals and 83 occupations, with domain experts ranging from architects and biologists to underwriters and data scientists. "Each task reflects a multi-step project that takes a seasoned professional three to eight hours to complete, pairing a realistic prompt and a bundle of multimodal input files with an expert-written rubric of roughly 70 yes/no checks," the post says. Scoring is weighted 75% factual accuracy and 25% visual design.
Every model runs inside the same open-source agent harness, called Stirrup, with a shared toolset and turn budget. On that harness the tested models "have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task." An agentic autorater grades outputs; when domain experts checked its work, they agreed 96% of the time.
The full 200-task set stays private with Artificial Analysis. Ten sample tasks are posted on Hugging Face. The post does not name the model that beat Gemini 4 Carbon on quality, and it is the third Google alert on our tracker today, following the wider Gemini 4 Carbon API rollout.
What others are reporting
-
Artificial Analysis Read →
The co-creator's live leaderboard tests 21 models with granular breakdowns by vertical, output file type, and cost; the most complete performance dataset on MKA available.
It tests whether agents can read visually dense project files, such as architectural drawings and engineering schematics, and produce the deliverables a professional would.
-
Crypto Briefing Read →
Names the conflict of interest in DeepMind co-building a benchmark where its own model competes, and examines how design quality scoring raises the bar for professional output.
By requiring both visual comprehension and polished output, MKA tests the full loop of knowledge work: take in complex material, reason about it, and hand back a usable product.
-
Superpower Daily Read →
Highlights the choice to keep all 200 tasks private to prevent contamination, and flags visual design as 25% of scoring, a governance detail absent from the primary announcement.
Professional work often requires something else: cross-referencing architectural blueprints, following engineering schematics, or turning findings into decks.
Shared on Bluesky by 1 AI expert
Originally reported by blog.google
Read the original article →Original headline: Google DeepMind opens Multimodal Knowledge Agent benchmark with 200 tasks across 10 verticals