Google launches Multimodal Knowledge Agent benchmark with Gemini 4 Carbon

• Google and Artificial Analysis launched the Multimodal Knowledge Agent (MKA) benchmark to evaluate how agents reason across visually complex files for real-world knowledge work.
• Gemini 4 Carbon achieved the #2 quality score among tested models while completing tasks at 6x lower cost on average compared to the top-scoring model.
• The benchmark consists of 200 tasks created by domain experts across 10 verticals and 83 occupations, each representing a multi-step project taking 3 to 8 hours to complete.
• An agentic autorater built on a non-Google model grades outputs, achieving 96% agreement with human domain experts.

Benchmark Structure

• Tasks pair realistic prompts with multimodal input files and expert-written rubrics of roughly 70 yes/no checks.
• Rubrics are weighted 75% on factual accuracy and 25% on visual design.
• A separate set of 10 public tasks is available on Hugging Face for format inspection and testing.

Evaluation Methodology

• All models run in the same open-source agent harness, Stirrup, with identical tools, sandbox, and turn budget.
• The 200-task evaluation set is kept private by Artificial Analysis to prevent leakage into training data or search results.
• Tested frontier models showed extreme differences in efficiency, using between 30 and 130 turns on average to complete each task.