TL;DR. Two researchers at the University of Toronto analysed more than a thousand runs of a coding agent, token by token. The result: about 80 percent of what the agent generates goes into reading, searching and testing, and only a small remainder into the actual change. That is mostly not waste but the price of the agent having to understand the project first. The more important news is in the second part of the study: a test suite that already wobbles on clean code lets the agent's defects through. Your tests decide quality more than the model does.
What the study measured
On November 17, Alán Aspuru-Guzik and Varinia Bernales published the study "Where the Tokens Go". They borrow terms from thermodynamics: tokens that bring the project closer to the requested change are called work. All others are called heat, meaning calls after which the code stands exactly as before.
The task was deliberately simple. In a Python project with several hundred modules, the agent had to rename one module and update every reference to it, and the tests had to pass afterwards. Of 1,200 runs, 1,110 reached the goal. A typical run cost about 2,300 tokens. About 1,900 of them went into reading files, searching the project and testing, about 400 into the change itself. Across all landed runs, 80 percent was heat, between 73 and 85 percent in every test series. A run took 9 to 17 steps, and only about two of them changed the code.
Output tokens per successful run, split into change and searching/reading/testing
Study, University of Toronto
Own test, November 20
01,2002,400 tokens
Change ("work")Reading, searching, testing ("heat")Minimum cost according to the study
The authors also estimate what the change should cost at minimum: about 350 tokens. That is 14 to 22 percent of the actual cost. They call this the efficiency of an agent, a number no benchmark reports so far. Benchmarks count whether a task was solved, not what the path there cost.
Is it waste?
The authors say: mostly not. In these steps the agent finds out what it needs to change. Savings are more likely, they write, from telling the agent what it would otherwise have to find out itself. They explicitly did not measure this.
We tried it on a small scale. Claude Sonnet 5.5 had to rename a module in a test project with twelve modules; the module is used in five places, one of them only as text in a list. Ten runs got just the task, ten also got a note listing all five places. All 20 runs reached the goal. Without the note the agent needed 947 output tokens on average, with the note 1,802. It still searched the project itself in every run, additionally read the files named on the note and wrote a longer report at the end. So a hint only saves tokens if it truly replaces the search, for example in large projects where searching is expensive. We did not measure that.
Our test showed one more thing: for every output token there were eight to nine input tokens, i.e. project content the model had to read. The study only counts output. Your bill shows both.
The real weak spot is the gate
An agent considers its work done when the tests pass. The study therefore checks how reliable this gate is. In 60 trials on three machines, the researchers planted defects in the code themselves and let them be checked: not at all, only with the tests of the changed file, or with the whole test suite.
Share of code states let through that carried a planted defect, in percent
0%25%50%
The whole suite was no better than the tests of a single file. The reason: some tests failed even on clean code, sometimes because of the network, sometimes because of the order, sometimes because of the machine. One machine had 11 failures on clean code and therefore let up to 33 through. A planted defect caused 8 to 12, so every one passed. One morning the three machines counted 17, 11 and 1 failures, the next morning 2, 1 and 4. Careful calibration did not help either. The authors conclude: the remedy lies in the tests themselves.
In a second experiment the researchers deliberately falsified the verdict of the tests. Result: the share of broken changes that passed as done followed the gate's error rate almost one to one.
Injected error rate of the tests against broken changes let through, in percent
0%15%30%
injected error rate of the testsbroken changes among those let through
A gate that is wrong once in ten cases therefore lets roughly every tenth broken change pass as done. If a task is hard and more than half of the attempts fail, it gets worse. In a purpose-built toy project, 52 percent of the finished changes were broken at a 30 percent gate error. The authors write: if models improve faster than tests do, the tests will decide the quality of what ships.
The catch
- One model, one project, two renames. Only DeepSeek V4.1 Flash with its reasoning mode switched off was measured. The authors say themselves that larger changes may look different.
- Output tokens only. What the model reads is not counted in the study. In our test it was by far the larger part.
- Small sample for the gate. According to the authors, 60 trials are not enough to tell the two kinds of gate apart with certainty.
- An analogy, not a law of nature. The physics terms are a tool for ordering. The study is a preprint and not yet peer-reviewed.
- Our test is even smaller. One model, one task, 20 runs. It shows a direction, not an average.
One side finding, by contrast, is fairly clear: if you set the model very strictly, i.e. to a low temperature, it repeats itself. At 0.25 and below, 12 to 35 percent of runs ended in a loop in which the same search command ran again and again. Below 0.5, a successful change cost 2 to 14 times as much as above it, counting all runs.
What this means for mid-sized companies
If you let agents work on your own software, on the ERP system, on supplier interfaces or on internal tools, you should know: the biggest cost item is understanding the project, not writing code. A cluttered project costs extra on every single task.
The gate matters more. If your test suite has tests that are "just red sometimes", a person on the team knows which ones to ignore. An agent does not, and a generous threshold then lets through exactly the defects you are looking for.
What you should do now
- Fix or remove flaky tests. Run your test suite ten times on unchanged code. Every test that turns red even once needs fixing before an agent relies on it.
- No threshold for "a few failures are normal". The agent should only be done at zero failures.
- Measure the cost per run, including failed runs. Have your tool log input and output tokens per task. Failed runs are the most expensive.
- Break loops. If an agent repeats the same command with the same output, stop and restart. Set a step limit based on successful runs.
- Test hints instead of assuming them. A project description for the agent can help. Measure with and without before rolling it out to everyone.
Better models get a lot of attention. This study shows that tests deserve at least as much. If you want to know how reliable your tests are as a gate for agents, we are happy to look at it together.