WindTunnel: Benchmarking WebMCP Against Browser Agents
TL;DR
- WindTunnel is the first public benchmark to compare WebMCP with screenshot- and page-structure-based browser agents on the same websites, tasks, and scoring conditions.
- We evaluated 15 browser-agent configurations on 49 tasks across eight real, open-source web applications.
- Across the models tested, WebMCP passed 93–98% of attempts, compared with 78–90% for computer use and page-structure approaches.
- WebMCP was also 3–5× faster, 5–10× cheaper, and used 5–11× fewer tokens per task.
- When available, WebMCP should be the default interaction layer, with screenshots and page structure as fallbacks. Its structured tool calls also give website owners clearer visibility into how agents use their products.
- The benchmark is fully open and reproducible, with the complete setup, tasks, implementations, scoring logic, and raw results published on GitHub.
The browser is becoming an increasingly important interface for the agentic web, across a growing range of tasks and applications. We are already seeing several interaction models emerge. With co-browsing, a person uses the browser while an AI assistant works alongside them in a side panel, as with Gemini in Chrome and Claude in Chrome. With delegated browsing, the agent can navigate websites and complete tasks on the user’s behalf, as ChatGPT’s built-in browser can do. These modes are also starting to overlap: users can watch the agent work, intervene, or take over when needed.
As AI agents take on more work in the browser, it becomes increasingly important to measure how well they actually perform. That comes down to three practical questions: Can the agent complete the task correctly? How long does it take? And what does it cost? WindTunnel measures these outcomes directly, while also tracking token usage to help explain the differences in cost and efficiency.
WindTunnel groups the ways browser agents interact with websites into three main interfaces:
- WebMCP: The website exposes structured tools that the agent can discover and invoke directly.
- Screenshots (computer use): The agent observes rendered images of the page and interacts through mouse and keyboard actions, including clicks at screen coordinates.
- Page structure: The agent reads the page’s DOM, accessibility tree, or both to identify elements and interact with them.
There are already several well-known browser-agent benchmarks—such as WebArena, VisualWebArena, and WorkArena—that evaluate agents relying on screenshots, page structure, or a combination of both. Yet, to our knowledge, very little research has compared all three interfaces on the same websites, using the same tasks and conditions.
To make this comparison possible, we assembled a benchmark suite around eight real, open-source web applications. Across these sites, we defined 49 benchmark tasks, ranging from simple information retrieval to multi-step actions and transactions. For each application, we built a reference—or “golden”—WebMCP implementation, together with seeded data and outcome-based scoring checks. We then ran every method against the same locally hosted applications, starting states, tasks, and scoring rules.
Sites in the benchmark
| Application | Type | What the benchmark exercises | Tasks in reference run |
|---|---|---|---|
| Next.js Starter Medusa | Online store | Product discovery, cart management, and guest checkout | 9 |
| Hi.Events | Events and ticketing | Event discovery, ticket selection, and checkout | 10 |
| Easy!Appointments | Appointment booking | Availability, booking, and authenticated administration | 8 |
| LearnHouse | Course platform | Course discovery and authenticated course authoring | 7 |
| IDURAR ERP/CRM | B2B ERP/CRM | Authenticated creation, retrieval, updating, and deletion of records | 8 |
| Directory | Business directory | Search, filtering, and retrieving business details | 3 |
| Tailwind Next.js Blog | Read-only control | Finding and reading published content | 2 |
| Bulletproof React | Authentication control | Public content and authenticated access boundaries | 2 |
Methodology:
WindTunnel now covers 15 configurations across three interfaces. Each configuration combines an interface, implementation, and model; one shared harness runs and scores all of them against the same tasks and outcome checks.
| Interface | Configurations | Models | How it operates |
|---|---|---|---|
| WebMCP | 7 | Sonnet 4.6, GPT-5.5, Luna, SOL, Gemini 3.6 Flash, Opus 5 | Discovers and invokes structured tools exposed by the page |
| Computer use | 6 | Sonnet 4.6, GPT-5.5, Luna, SOL, Gemini 3.6 Flash, Opus 5 | Observes screenshots and performs mouse and keyboard actions |
| Page structure | 2 | Sonnet 4.6 | Uses the DOM, accessibility tree, or both with vision |
Everything needed to reproduce the benchmark—including pinned site versions, seeded data, task definitions, WebMCP implementations, agent arms, scoring logic, and raw results—is available in the WindTunnel GitHub repository.
Results
The reference run and subsequent model runs now form one 15-configuration leaderboard. WebMCP occupies the top seven positions, with higher attempt success and substantially lower median cost and agent time than the computer-use and page-structure configurations.
| Rank | Configuration | Interface | Final score | Tasks solved | Attempt success | Median cost / task | Median tokens / task | Median time / task |
|---|---|---|---|---|---|---|---|---|
| 1 | Sonnet 4.6 · Stagehand v4 | WebMCP | 91.9 | 48/49 | 144/147 (98.0%) | $0.008 | 3,596 | 7.3s |
| 2 | Gemini 3.6 Flash | WebMCP | 89.9 | 47/49 | 142/147 (96.6%) | $0.004 | 4,460 | 7.8s |
| 3 | Sonnet 4.6 · native | WebMCP | 88.3 | 48/49 | 142/147 (96.6%) | $0.007 | 3,626 | 7.2s |
| 4 | GPT-5.6 Luna | WebMCP | 85.5 | 46/49 | 137/147 (93.2%) | $0.002 | 2,596 | 5.7s |
| 5 | Claude Opus 5 | WebMCP | 81.9 | 48/49 | 142/147 (96.6%) | $0.014 | 4,871 | 9.9s |
| 6 | GPT-5.5 | WebMCP | 76.2 | 47/49 | 137/147 (93.2%) | $0.015 | 2,546 | 5.8s |
| 7 | GPT-5.6 SOL | WebMCP | 74.7 | 46/49 | 138/147 (93.9%) | $0.013 | 2,573 | 9.4s |
| 8 | GPT-5.5 | Computer use | 47.3 | 44/49 | 131/145 (90.3%) | $0.090 | 18,659 | 20.4s |
| 9 | Sonnet 4.6 | Accessibility tree | 35.8 | 42/49 | 128/147 (87.1%) | $0.043 | 11,784 | 35.6s |
| 10 | Sonnet 4.6 | DOM + vision | 34.7 | 43/49 | 130/147 (88.4%) | $0.112 | 33,365 | 37.8s |
| 11 | GPT-5.6 Luna | Computer use | 33.9 | 41/49 | 122/147 (83.0%) | $0.017 | 20,914 | 18.3s |
| 12 | GPT-5.6 SOL | Computer use | 26.8 | 42/49 | 123/147 (83.7%) | $0.063 | 16,312 | 25.1s |
| 13 | Claude Opus 5 | Computer use | 24.8 | 43/49 | 127/147 (86.4%) | $0.139 | 47,141 | 50.4s |
| 14 | Gemini 3.6 Flash | Computer use | 15.2 | 38/49 | 116/147 (78.9%) | $0.020 | 23,658 | 33.7s |
| 15 | Sonnet 4.6 | Computer use | 9.7 | 39/49 | 115/147 (78.2%) | $0.047 | 38,856 | 31.1s |
Final score: The final score combines attempt success rate (60%), median cost per task (20%), and median agent time per task (20%). Cost and time are log-normalized (large differences are compressed so extreme values do not dominate); higher is better. The interactive benchmark page includes the harness details and full run artifacts.
Across the tested models, WebMCP attempt success ranged from 93.2–96.6%, compared with 78.9–90.3% for computer use. Luna was the lowest-cost WebMCP configuration at a $0.002 median attempt; Opus solved 48/49 WebMCP tasks but was the most expensive computer-use configuration.
Conclusions
Exposing WebMCP tools substantially improves a website’s agent experience (AX)—how effectively agents can understand and use it. Agents should not have to reconstruct available actions from pixels or page structure. WebMCP gives them a direct, structured way to act, and in our benchmark that made them more reliable, faster, and cheaper—often by an order of magnitude on longer tasks.
Screenshots and page structure still matter as fallbacks, especially when a site does not expose the required tools. But when WebMCP is available, the results suggest that it should be the default way for an agent to operate a website through the browser.
Final notes:
WebMCP offers another benefit that is easy to overlook: it makes agent activity visible to the website itself. When an agent operates through screenshots or page structure, its behavior usually appears as ordinary clicks, keystrokes, and page loads. Website owners may capture those events through existing analytics, but they often cannot tell that an agent produced them, which operation it was trying to perform, or where the workflow failed. They are left trying to reconstruct the story from a clickstream designed around human browsing.
With WebMCP, every interaction is a named tool call with structured parameters and a defined result. Because the website owns the tool implementation, it can instrument those calls as first-class application events—measuring which capabilities agents use, where calls fail, and which workflows are missing.
That means the benefits flow both ways. Users and their agents get a faster, cheaper, and more reliable way to complete tasks. Website owners get a clearer way to understand, improve, and govern how agents use their products. WebMCP does not just help agents use websites better; it gives websites a better way to work with agents.