TL;DR. OpenAI and Anthropic now ship agent teams out of the box: a lead agent breaks a task down and hands the parts to subagents. On October 9, the evaluation company Vals AI measured what that buys: 50 complete web applications, built by GPT-6 Sol and Claude Opus 5.5, each working alone and as a team. The teams cost 1.8 to 5.1 times as much. Only one of four comparisons showed a measurably better result. Anthropic's own figures for teams of up to 100 agents show the same pattern. An agent team mostly buys time, rarely quality. And it only buys time when the work splits cleanly.
What was measured
Vals AI used its Vibe Code Bench. An agent gets a product spec and builds a complete web application from it, with database, API, interface and deployment. Testing agents then click through every workflow in a browser. The score is the share of tests that pass. In a team, the lead agent may keep up to five subagents busy at once. Both models ran at medium and at maximum reasoning effort, which makes eight setups.
| Setup | Score alone → team | Cost per app alone → team | Runtime (median) alone → team |
|---|---|---|---|
| GPT-6 Sol medium effort | 77.6% → 84.9% | $1.22 → $3.07 | 12.4 → 11.9 min |
| GPT-6 Sol maximum effort | 89.0% → 90.4% | $4.82 → $8.54 | 38.4 → 27.6 min |
| Claude Opus 5.5 medium effort | 91.5% → 91.2% | $4.08 → $9.10 | 20.3 → 26.6 min |
| Claude Opus 5.5 maximum effort | 89.8% → 93.2% | $23.77 → $122 | 74.7 → 174.1 min |
What the team costs and what it delivers
Extra cost of the teamGain statistically robustDifference within the range of chance
Only one result is statistically robust: as a team, Sol at medium effort gains 7.3 points. The other three differences range from minus 0.3 to plus 3.4 points and may be chance. For Sol, more reasoning effort helped even more than a team, adding 11.4 points for the single agent. For Opus, neither option changed anything robust. The most expensive setup, Opus at maximum effort as a team, did reach the highest score in the study. But it cost about 30 times as much as the single Opus agent at medium effort, for 1.6 points that are also within the range of chance.
Where the gains came from
It is telling where Sol's team gained. On the nine applications where it was ahead by 20 points or more, the single agent had reported that its tests passed. Yet features had never been built and whole workflows had never been tried. The team built them and checked more carefully: the lead agent compared the separately built parts against each other and found errors at the seams, such as a form that did not match the database. So the gain came less from building in parallel than from cross-checking.
Opus took a different route. Its lead agent usually wrote a shared contract first, describing how the parts fit together, in 42 of 50 runs at medium effort. Then it handed out the work in waves. At maximum effort the team therefore took up to 2.3 times as long as the single agent, and one run hit the time limit of 5.25 hours. Most of the money went into inputs that were read again and again: every subagent carries its own context with the task and its working state and resends it with every call. The Opus subagents read a median of 224 million cached tokens per application, the single agent 55 million.
What Anthropic measures with 100 agents
Anthropic examined teams in its system card for Claude Opus 5.5 of September 22, with up to 100 agents working on one task together for 24 hours. One task: write a knowledge base from a company's documents that helps answer questions. The other: complete a difficult mathematical proof.
Score after 24 hours by number of agents (Claude Opus 5.5)
knowledge base from company documentsmathematical proof
Gain from 1 to 10 agents: +0.17 and +0.27. From 10 to 100 agents: +0.04 and +0.02.
Anthropic stresses the second effect: larger teams reach a given level much earlier. In a programming test, a team of five agents reached the same score 2.7 times faster than a single agent, using correspondingly more tokens. On shorter research tasks, however, teams without time pressure were slower, because coordination takes time. Only a tight time budget turned this around: with half the time, the team of five at least matched the single agent's result. The system card also shows how large teams organize themselves. The 100-agent team working on the proof formed twelve sub-leads, while the knowledge base team stayed flat.
The fine print
- One run, one kind of task. Each of the eight setups ran exactly once at Vals AI, and only on web applications. The study explicitly leaves open whether the result carries over to research or document work.
- Two different tools. Sol ran through OpenAI's Agents API, Opus through Claude Code. Comparisons between the two models therefore mix the model with its tooling.
- Anthropic measures itself. The figures on large teams come from its own system card and from a slightly different model version.
- There are counterexamples. In a study by Sakana AI and UC Berkeley of October 8, external judges preferred the work of agent teams on open research tasks in 85.1% of comparisons. The teams produced about 2.4 times as many tokens for it. At comparable length, an 80% preference remained.
What this means for a mid-sized company
Agent teams are arriving in standard products right now. On October 8, Google presented a work agent that handles multi-step tasks with subagents, in parallel or in sequence, currently in a private preview. The measurements say: the number of agents is not a quality dial. It is a speed dial with a price tag. Vals AI itself names work made of many independent parts as the next candidates, such as broad research or analyses across many documents. That fits a task like checking 300 supplier contracts for one clause. A single quote calculation, on the other hand, does not get better with five agents, only more expensive.
What you should do now
- Start with one agent. A single agent at medium reasoning effort is the benchmark. A team has to beat it measurably on your own cases, otherwise you stay with one.
- Reasoning effort first, then the team. For Sol, more reasoning effort helped more than a team. Try the simpler dial first and compare the cost per result.
- Teams only for work that splits. Many independent documents, broad research, large codebases: there a team saves time. Where everything hangs on one thread, coordination eats the advantage.
- Add a check instead of more builders. The biggest gain in the test came from cross-checking. A separate review step that holds the result against the requirements helps a single agent too. A "tests passed" that the agent reports about itself is not yet proof.
- Cap cost and runtime. Teams multiply the tokens read. Set a ceiling for budget and time per run, and measure the cost per finished result instead of per token.
More agents do not make a task smarter, at best faster. Knowing that before you buy saves expensive experiments. If you would like to check whether one of your tasks suits an agent team, we are happy to work through the numbers with you.
