On November 11th, according to IT, on November 10th local time, Google announced a collaboration with AI and the model evaluation institution Artificial Analysis to launch the Multimodal Knowledge Agent Benchmark Test MKA, which is used to assess the ability of AI agents to process visually complex documents and complete real-world knowledge-based tasks.

Google stated that its newly released Gemini 4 Carbon achieved second place among the models tested in this experiment, with an average cost of completing tasks that was only about one-sixth of that of the model that ranked first.

Google's Senior Product Manager DeepMind and Chief Engineer Ben Caine stated that many existing knowledge-based work benchmarks can be completed with just text, but real-world work often involves a large amount of visual information. For example, architects need to cross-check building blueprints, engineers need to analyze engineering schematics, and professionals also need to organize information from different sources into presentations, reports, and spreadsheets.

MKA The benchmark test is aimed at evaluating the performance of AI intelligents in these complex work scenarios. Google collaborated with experts from 10 industries and 83 professions to establish a test set comprising 200 tasks, covering areas such as business, construction, education, engineering, finance, healthcare, law and government, logistics and mapping, science, as well as software and data.

Each task simulates a multi-step professional project, including real work-related keywords as well as a set of multimodal input files. Google states that these tasks typically require experienced professionals to complete in 3 to 8 hours. IT notes that each task is also accompanied by scoring criteria written by experts, which on average include about 70 non-judgmental check items, of which factual accuracy accounts for 75% and visual design accounts for 25%.

In terms of the evaluation mechanism, all the models participating in the test ran on the same open-source agent framework Stirrup, using the same tools, sandbox environment, and interaction round budget. The final deliverables were scored by an agent evaluation system built on a non-Google model, which opened all files, rendered pages, checked spreadsheet formulas, and verified the results according to expert scoring criteria. Google stated that when domain experts manually scored the model outputs, the consistency rate with the results of the automatic evaluation system reached 96%.

To prevent test questions from being leaked and entering the model training data, Artificial Analysis has set 200 formal evaluation tasks as private. At the same time, the project has also made another 10 tasks public on Hugging Face for developers to understand the test format and evaluate their own agents.

Google's test results show that Gemini and Carbon achieved second-place quality rankings among the tested models, but their average task cost was only one-sixth of the model that ranked first. Google also noted that there were significant differences in efficiency among the different models when completing the same set of tasks, with an average of 30 to 130 interaction rounds consumed per task.

Currently, the MKA benchmark tests have been passed, and Artificial Analysis has been opened. Developers can use this platform to evaluate their models. Gemini and Carbon have also passed the Gemini API tests and are available in Google AI Studio.

Related Reading:

• Google's Most Powerful Model to Date: Gemini 4 Carbon Debuts, Focusing on Complex Programming, Scientific Research, and Cybersecurity
• Google's Most Powerful AI Model: Gemini Ranks First in Global Intelligence Index, "Dedicated" to Cybersecurity