Generation Got Cheap. Verification Did Not.

This is a report on other people's evidence rather than our own. We have opinions about what machine assistance does to a delivery pipeline, but opinions are what everybody has, and the published record is now good enough that nobody needs another vendor's feelings on the subject.

What follows is what the two best controlled experiments found, why their opposite results are both correct, what the largest industry survey measured either side of them, and the piece of arithmetic that explains all of it. Then, since we are asking you to trust us with work, what we actually do about it.

1. Two experiments that point opposite ways

In early 2023, Peng, Kalliamvakou, Cihon and Demirer ran a controlled experiment with seventy professional developers. Each was asked to implement an HTTP server in JavaScript as quickly as they could. Half had an AI assistant, half did not. The assisted group finished in 71 minutes against 161 for the control group, which is 55.8 percent faster.

In mid 2025, METR ran a randomised trial with sixteen experienced open source developers across 246 real tasks in repositories they had worked on for an average of five years. When they were allowed AI tools, they took 19 percent longer.

Neither study is weak. Both randomised. Both used real developers on real tasks. They point in opposite directions with the same class of tool.

2. They do not actually disagree

Look at what was being built rather than at what was helping.

The first task was a fresh HTTP server in a language with a million public examples, written by people with no prior context to respect. Nothing existed to be understood before typing could start. Generation was essentially the whole job, so making generation cheap made the whole job cheap.

The second was a set of changes inside mature codebases that the developers already knew intimately, where the hard part was never producing the characters. The hard part was knowing which twenty lines of a large system to touch, and being certain that touching them broke nothing. In that setting the assistant produced candidate work quickly, and then the developer had to read it, distrust it, and check it against knowledge they already held. That checking was pure addition, and it is where the 19 percent went.

So the honest one line summary of the current evidence is this. Machine assistance moves quickly when the context is small and the correctness bar is soft. It moves slowly, sometimes backwards, when the context is large and the correctness bar is hard.

Which is to say it is fastest exactly where the stakes are lowest.

3. What happened at industry scale

The DORA programme surveys thousands of practitioners a year and models the relationship between what teams do and how their delivery performs. It looked directly at AI adoption two years running, and the two answers are worth putting side by side.

In 2024, a 25 percent increase in AI adoption was associated with a 1.5 percent decrease in delivery throughput and a 7.2 percent decrease in delivery stability. Both directions were negative, which was not what anyone had put on their slides.

In 2025, with adoption up around 90 percent of respondents, the throughput relationship reversed and turned positive. The stability relationship did not. AI adoption was still associated with more instability, and the report's own explanation is the one this paper is about: the time saved in creating code is being spent again on auditing and verifying it.

Read the pair as a story rather than as two data points. The industry learned, over one year, how to get the speed. It has not yet learned how to keep the safety. The gap between those two curves is where projects break.

4. The arithmetic of a moved bottleneck

There is a two hundred year old idea that settles this, and it takes one line.

If generation is a fraction of your total cycle, making generation infinitely fast saves you that fraction and nothing more. Everything else is a floor you cannot get under by generating faster.

Suppose generation is 35 percent of the time between a decision and a working release. Make it ten times faster and the cycle drops by about 31 percent. Make it a thousand times faster and it drops by 35 percent, and then it stops forever, because the other 65 percent is understanding the problem, deciding what should exist, reviewing, integrating, testing, and getting it safely into production.

Now add the second effect, the one the DORA stability number is measuring. Cheap generation does not just make the generation step faster. It increases the volume of change flowing into every step downstream. More code, more diffs, more surface, all of it produced faster than a human read it. So the review and integration portion does not stay constant. It grows.

That is the dashed line in the figure, and it is why a team can adopt these tools, genuinely produce more, and end up slower and less stable than it was. Nobody did anything wrong. The bottleneck moved and the system did not.

If generation is free, the constraint is whatever you do to convince yourself the generated thing is correct. That is now the job.

5. The part that should worry you most

METR asked its developers to estimate their own effect. Before the trial they expected AI to make them 24 percent faster. Afterwards, having been measured at 19 percent slower, they estimated it had made them 20 percent faster.

That is a roughly 39 point gap between what competent professionals experienced and what actually happened, measured on themselves, on their own code.

DORA's 2025 numbers rhyme with it: around 90 percent of respondents use AI at work and over 80 percent believe it has increased their productivity, in the same dataset that shows stability going the wrong way.

We think the mechanism is that assistance removes the sensation of difficulty. Staring at a blank file is unpleasant and memorable. Reading a plausible looking diff is pleasant and forgettable, even when it takes longer and even when it is where the defect gets in. Effort is what people can feel. Elapsed time is not.

The practical consequence is that self reported productivity is now worthless as a management signal, including our own. If we tell you our team is more productive because they say so, do not believe us. Measure the cycle.

6. What we let machines do

Our position is that the tool is excellent and that the pipeline around it has to change to earn the benefit. Concretely, the split we run:

• Generation, always. First drafts, boilerplate, migrations, test scaffolding, one off scripts, documentation drafts. Cheap to produce, cheap to check, low blast radius.
• Search and explanation, always. Finding the four places a behaviour is implemented, explaining an unfamiliar module, summarising a change set. This is where assistance is quietly at its best and nobody writes headlines about it.
• Mechanical review, always. Types, linters, formatters, dependency and licence checks, secret scanning, generated test runs. Every check a machine can make is a check no human should be making, and it happens before a person is asked to look at anything.
• Judgement, never alone. Architecture, data modelling, security decisions, anything touching money, anything touching personal data, and the final read of a change before it merges. A person signs, and that person is named.

The design rule behind that list is simple to state and unpleasant to follow: the volume of change a pipeline accepts must be limited by the rate at which it can be verified, not by the rate at which it can be produced. If verification is the constraint, then spending on verification is the only spending that increases capacity. Everything else just makes the queue longer.

So the investment goes into the checks: tests that mean something, environments that reproduce, review that is scheduled rather than squeezed, and a hard cap on the size of a change that a human is asked to approve. Small diffs are not a style preference. They are the mechanism that keeps human judgement in the loop at machine speed.

7. What we still do not know

Three honest gaps, because an R&D report that reports no uncertainty is an advertisement.

We have not published our own measurements. We hold cycle time and change failure rate internally, but a single firm's numbers over a small number of projects cannot separate the tool from the team from the client, and we are not going to dress up an anecdote as a finding. When we have enough projects under one measurement regime to say something defensible, we will publish it here, including if it is unflattering.

The published evidence ages fast. Both trials in section one measured tools that have since changed materially. Any conclusion here should be read with its date attached, and the honest posture is that the effect size is unstable, not that it is settled in either direction.

And the long term question is genuinely open. If assistance keeps improving, does verification also become automatable, or does it stay stubbornly human because it is really the act of deciding what should be true? We do not know. We are betting, structurally and financially, that some irreducible part of it stays human, and we would rather say that is a bet than pretend it is a fact.

References

1. 01Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer, The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, 2023, seventy developers, arxiv.org
2. 02METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 2025, sixteen developers, 246 tasks, metr.org
3. 03DORA, Accelerate State of DevOps Report 2024, dora.dev
4. 04DORA, State of AI-assisted Software Development 2025, dora.dev
5. 05Gene M. Amdahl, Validity of the single processor approach to achieving large scale computing capabilities, 1967, for the arithmetic in section four