OSWorld-MCP Benchmark Leaderboard

OSWorld-MCP: Benchmarking MCP Tool Invocation in Computer-Use Agents

Authors: Hongrui Jia^2,†, Jitong Liao^1,†, Xi Zhang^1,†, Haiyang Xu1^1,*, Tianbao Xie¹, Chaoya Jiang², Ming Yan^1,*, Si Liu³, Wei Ye², Fei Huang¹

Affiliations: ¹ Tongyi Lab, Alibaba Group · ² Peking University · ³ Beijing Zhongguancun Academy

📢 2025-10-28: We released our paper and code! Read the Paper | View on GitHub

Abstract

With advances in decision-making and reasoning capabilities, multimodal agents have shown strong potential in computer application scenarios. Past evaluations mainly assessed GUI interaction skills, while tool invocation abilities enabled by the Model Context Protocol (MCP) have been largely overlooked. We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents’ tool invocation, GUI operation, and decision-making abilities in a real-world environment.

Figure 1: Task execution via GUI vs MCP Tool

Highlights

First benchmark integrating GUI + MCP evaluation in realistic OS environments.
158 curated high-quality MCP tools covering 7 common applications.
New metrics: Accuracy, Tool Invocation Rate (TIR), Average Completion Steps (ACS).
MCP tools improved success rate up to +15.7% accuracy in some cases.
Positive relationship between TIR and accuracy validates benchmark challenge.

Figure 2: Tool distribution across software ecosystems

Leaderboard (GUI + MCP)

Model	Steps	Accuracy	TIR	ACS
Agent-S2.5	15	42.1	30.0	10.0
Claude-4-Sonnet	15	35.3	30.0	10.4
Seed1.5-VL	15	32.0	25.1	10.2
Gemini-2.5-Pro	15	20.5	16.8	11.4
OpenAI o3	15	20.4	16.7	11.6
Qwen3-VL	15	31.3	24.5	10.5
Qwen2.5-VL	15	15.8	13.1	13.5
Agent-S2.5	50	49.5	35.3	17.0
Claude-4-Sonnet	50	43.3	36.6	20.1
Seed1.5-VL	50	38.4	29.0	23.0
Gemini-2.5-Pro	50	27.2	21.5	29.7
OpenAI o3	50	25.2	21.0	32.1
Qwen3-VL	50	39.1	29.5	21.1
Qwen2.5-VL	50	14.8	10.9	37.2

Click headers to sort, use buttons to filter results.