It is the end of the quarter and the development team is holding its retrospective. It has used coding agents since last quarter. Everyone says the same thing: we are faster. Is it worth measuring?
Yes, because what the team feels does not measure speed. METR, a lab that evaluates AI models, asked 349 technical workers in 2026 (exhibit 102). The median answer was 3 times faster with AI. METR put a warning next to the result: “Importantly, survey results are not necessarily grounded in reality.” It backs the warning with its own data. In its 2025 randomized study, people overestimated their gain by about 40 points.
3 times
Median self-reported speed change (value: 1.4 to 2 times), in 349 people. Exhibit 102
Almost every agent plan rests on productivity. That is why it matters what was measured, on which task and with which tool. And whether the number counts finished work or only produced work.
The cleanest evidence is a randomized study: a draw decides who uses the tool. A 2026 experiment randomly assigned 1,174 adults (exhibit 49). Each one solved a business problem with or without an AI assistant. Everyone improved with the assistant. But it is a chat assistant, not an agent, and the task was online, outside any company.
For coding agents, the largest dataset comes from Microsoft (exhibit 116). Tens of thousands of its engineers started to use two terminal agents, Claude Code and GitHub Copilot CLI. Over four months, those who adopted them merged 24.0% more pull requests per day. A pull request is a proposed change to the code. The comparison is against a synthetic control group, built from data on those who did not adopt them.
+24.0% (+14.5% to +33.7%)
Pull requests merged per engineer per day versus control, over a 4-month window (95% interval). Exhibit 116
The study has three limits. It is observational: it looks at what happened, with no draw. Its authors work at Microsoft. And it does not measure quality. Cursor published a similar 39% about its own tool (exhibit 18). The Cursor study could not be opened.
Copilot’s coding agent worked for ten months in dotnet/runtime, the repository for the .NET runtime (exhibit 68). It opened 878 pull requests, with a 67.9% success rate. Of the merged changes, 0.6% were reverted, against 0.8% for everything else. It did better on mechanical tasks, such as writing tests, than on tasks that need judgment.
One simple step nearly doubled its success. With an instructions file and the build set up, it went from 38.1% to 69%. The build is the process that compiles and tests the code. A Microsoft engineer wrote it, about a Microsoft tool.
Forums have cases outside code. An IT lead built a triage workflow with read-only access (exhibit 117). Triage means sorting tickets by type and urgency. He said on Reddit that it saves him 30 minutes per complex ticket. In another team, an agent cut the median incident time from about 40 to 12 minutes (exhibit 142). These are accounts with no independent measurement.
In a Hacker News thread, the people who reported success name the same conditions (exhibit 43): tidy code, broad tests, bounded tasks and reviewing the plan first.
The limit shows up when the task gets longer. A benchmark is a fixed set of tasks used to compare models. In SlopCodeBench, a 2026 benchmark, the agent extends its own code while the requirements change (exhibit 100). The best of 15 agents passed only 14.8% of the checkpoints. A note from the Agentic AI Foundation tells the story of a loosely supervised “software factory” (exhibit 131). It ran for three or four months, and the company ended up archiving the code.
Coding agents already do real work in serious repositories.
Verdict: Yes, on bounded tasks and with expert review. In dotnet/runtime, the agent opened 878 pull requests with a 67.9% success rate. Almost none were reverted. At Microsoft, engineers who adopted terminal coding agents merged 24.0% more pull requests than a control group. That study did not measure quality. But when it has to extend its own code, the best of 15 agents passes only 14.8% of the checkpoints.
Rule
Give the agent bounded tasks that have tests, write an instructions file, and leave the build ready for it to run. Have someone who knows the code review every change.
Why medium confidence. The supporting data come from companies writing about their own tools or their group's tools. Microsoft writes about Copilot and about its internal rollout. Cursor writes about a study that could not be opened. None is randomized and almost none measures quality. Against it there is a 2026 benchmark (a standard test) and one anecdote: a software factory that ended up archived. Practitioner accounts agree on the conditions: tidy code, broad tests, bounded tasks and someone who reviews everything.
Show the 7 exhibits in detailHide the detail3 for, 2 against, 2 qualify
Ten months of data on the Copilot coding agent in the dotnet/runtime repository. More than half of its pull requests were merged and almost none were reverted, because each one went through expert review. It did best on mechanical, well-scoped tasks (cleanup, tests) and worst on those that need judgment (performance). An instructions file and build setup almost doubled success. The bottleneck moved from writing code to reviewing it.
Numbers
878 PRs, 67.9% successAgent pull requests and success rate (87.1% for PRs by Microsoft staff; populations not comparable).
0.6% vs 0.8%Merged PRs later reverted: the agent's vs the rest.
from 38.1% to 69%Success before and after adding instructions and setting up the build.
86.2% vs 55.1%Success with and without commits from a person in the PR.
5 to 9 hoursReview work created by nine PRs the author opened in a short time.
Limitation Written by a Microsoft engineer about a Microsoft tool. It is a repository with very expert reviewers. The author notes that the agent and the people do not do the same kind of tasks.
“we cannot compromise on review quality, on expert and experienced eyes validating the changes”
Murphy-Hill, E., Butler, J., Savelieva, A. arXiv (Microsoft)
It studies the rollout of two terminal coding agents among tens of thousands of Microsoft engineers. Those who adopted them merged more pull requests than a synthetic control group. The most junior engineers (IC2) had the largest increase in pull requests created: +34%, against +21% for the reference level (IC4). By tenure, the gain is C-shaped: people with short and long tenure gain most. The adoption study covers only Copilot CLI: there, junior engineers tried it less at first. Use spread mostly through networks of colleagues. It does not measure quality.
Numbers
+24.0% (+14.5% to +33.7%)Pull requests merged per engineer per day versus control, over a 4-month window (95% interval).
+34% versus +21%Increase in pull requests created with 3 days of use per week, versus the same person's weeks without use. It compares level IC2 versus IC4, the reference (IC3: +21%, IC5: +25%, IC6: +22%). Read from figure 9, with Claude Code and Copilot CLI.
-13% and -14%Odds of trying Copilot CLI early for levels IC2 and IC3 versus IC4 (IC5: +22%). Copilot CLI only: the adoption study does not include Claude Code.
Limitation Observational, not randomized. The authors are from Microsoft, which owns GitHub, and they say so. A pull request is not delivered value. The gain by level compares each person's weeks with and without use: it is an association, not a cause. The adoption study includes only Copilot CLI.
“first use spread primarily through social networks”
It summarizes a University of Chicago study on organizations that use Cursor. After the agent became the default mode, they merged more pull requests (requests to change code) than a baseline group. Junior developers accept more autocomplete, and senior developers accept more agent edits. The most experienced also plan more before they generate code.
Numbers
+39 %Merged pull requests versus the baseline group trend.
+6 %Acceptance rate of agent edits for each additional standard deviation of years of experience (relative to the mean).
Limitation Vendor post about a study (Sarkar, SSRN 5713646) that could not be opened (403). Not randomized. No significant change in the revert rate.
“more experienced developers are more likely to plan an action before generating code”
Orlanski, G., Roy, D., et al. arXiv (UW-Madison and others)
The agent has to extend its own code as the requirements change. No agent solves a full problem end to end. The agent's code gets longer and more tangled with each iteration, while human code stays stable. Asking for quality in the prompt improves the start, but does not stop the decline.
Numbers
14,8 %Checkpoints passed by the best of 15 agents, across 36 problems and 196 checkpoints.
77% and 75.5%Trajectories in which structural erosion rises and in which code verbosity rises.
2.3 and 2.0 timesHow much more verbose and more eroded the agent code is than that of 473 open source Python repositories.
Limitation The figures are from version 2 (May 7, 2026), which expands the benchmark from 20 to 36 problems and changes the results. Version 1 is from March 25, 2026. The abstract was read. It measures code quality, not per-step errors.
An article about Dex Horthy, author of the 12-factor agents guide. A year later he holds the same view: a big agent loop does not hold up for serious work without structure and verification. His company let a lightly supervised "software factory" run for three or four months and then archived all the code. He says the missing piece is better verifiers of code quality.
Numbers
30,000 to 40,000 linesSize the factory code reached before it was archived.
10 of 30 (33.3%)Checkpoints the best models passed on the SlopCodeBench subset he cites; the differences between vendors are within the margin of error.
Limitation It is an article about one person, with his own anecdote; it is not a study.
An engineer with 11 years of experience says he automated about 80% of his job with a coding agent. The agent picks up issues and opens pull requests. He says quality holds because he reviews every change. He spends 2 to 3 hours a day reviewing and testing. One commenter says that when review was removed, quality degraded within a week. Others discuss the ethics of doing it without telling the employer.
Numbers
about 80%Share of the job the author says he automated.
2 to 3 hours a dayTime he spends reviewing and testing.
1,200 points and 276 commentsSize of the thread according to the archive.
Limitation One person's story, with one week of use at the time of the post. Commenters point out it works because the work is simple. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
The author sees a push to review little or no code ("if the tests pass, ship it"). On his existing code, half his time went to fixing subtle errors. The success stories agree on the conditions: code already organized by people, static types, broad tests, narrow tasks and reviewing the plan first. One says the agent wrote tests that tested nothing.
Numbers
461 points and 455 commentsSize of the thread.
Limitation Practitioner opinions. It was read through the Algolia API (post and top-level replies with the most replies).
The first objection: the early speed does not last. A Carnegie Mellon study compared 806 GitHub projects that adopted Cursor with 1,380 that did not (exhibit 17). In the first month, lines added rose 281.3%. After two months, the gain was gone. It left technical debt, which is code that is costly to maintain later. Code complexity rose 41.64%, and that complexity slows work down later. These are open-source projects, not companies.
Then there is METR itself. In 2025 it measured that its developers took 19% longer with AI. In 2026 it repeated the study with late-2025 tools (exhibit 60). There were 57 developers, 143 repositories and more than 800 tasks. This time, time with AI fell 18% for returning developers.
But the confidence intervals cross zero. A confidence interval is the range where the real effect probably is. If it includes zero, the effect can be zero. Also, 30% to 50% of the developers held back some tasks so they would not have to do them without AI. That is selection bias: the comparison loses exactly those tasks. METR concludes that AI probably speeds work up more today than in 2025. But its data is very weak evidence of how much.
Verdict: It probably speeds up output, but the size of the effect is uncertain. In a randomized experiment with an assistant, everyone improved on a business task. In open-source projects that adopted Cursor, lines added rose 281.3% in the first month. But the gain faded after two months. METR repeated its study in 2026. METR believes AI probably speeds work up more than in 2025, although it does not trust its own numbers.
Rule
Do not assume the gain: measure it on your own work, against doing the task without the tool. Measure quality again a few months later, because the early speed can leave debt behind.
Why medium confidence. The randomized experiment uses a chat assistant, not agents. The data on coding agents are observational. One is from Microsoft on its own rollout and one is on open-source projects. METR's repeat study has intervals that cross zero. METR distrusts its own figure because of selection bias. A vendor that estimates productivity from its own data shrinks it after adjusting for success.
Show the 5 exhibits in detailHide the detail2 for, 1 against, 2 qualify
Cruces, G., Fernández Meijide, D., Galiani, S., Gálvez, R. H., Lombardi, M. NBER (Working Paper 34851)
It is a randomized experiment (groups assigned at random) with a generative AI assistant, on a task to solve a business problem. Everyone improves, and people with less education improve more. The gap shrinks a lot but does not disappear, and when the AI is removed it largely returns. People with more education use the tool better.
Numbers
from 0.548 to 0.139 standard deviationsPerformance gap by education without and with AI (about three quarters closes), in 1,174 adults aged 25 to 45.
Limitation The NBER page with the abstract was read. It measures education, not experience; it is a chat assistant, not an agent, and the task is online, outside a company. It has revisions from May and October 2026.
Murphy-Hill, E., Butler, J., Savelieva, A. arXiv (Microsoft)
It studies the rollout of two terminal coding agents among tens of thousands of Microsoft engineers. Those who adopted them merged more pull requests than a synthetic control group. The most junior engineers (IC2) had the largest increase in pull requests created: +34%, against +21% for the reference level (IC4). By tenure, the gain is C-shaped: people with short and long tenure gain most. The adoption study covers only Copilot CLI: there, junior engineers tried it less at first. Use spread mostly through networks of colleagues. It does not measure quality.
Numbers
+24.0% (+14.5% to +33.7%)Pull requests merged per engineer per day versus control, over a 4-month window (95% interval).
+34% versus +21%Increase in pull requests created with 3 days of use per week, versus the same person's weeks without use. It compares level IC2 versus IC4, the reference (IC3: +21%, IC5: +25%, IC6: +22%). Read from figure 9, with Claude Code and Copilot CLI.
-13% and -14%Odds of trying Copilot CLI early for levels IC2 and IC3 versus IC4 (IC5: +22%). Copilot CLI only: the adoption study does not include Claude Code.
Limitation Observational, not randomized. The authors are from Microsoft, which owns GitHub, and they say so. A pull request is not delivered value. The gain by level compares each person's weeks with and without use: it is an association, not a cause. The adoption study includes only Copilot CLI.
“first use spread primarily through social networks”
GitHub projects that adopted Cursor speed up a lot at first, but the gain fades after two months. More complexity and more static analysis warnings (automatic code checks) remain, and that slows velocity later. Cursor speeds things up for a while and leaves technical debt (problems to fix later).
Numbers
+281.3% and +48.4%Lines added in month 1 and month 2 after adoption, in 806 repositories versus 1,380 controls.
+30.26% and +41.64%Static analysis warnings and code complexity after adoption.
Limitation Difference-in-differences on open source projects, not companies. The figures come from version 3 (January 26, 2026); the title changed between versions and it was not checked whether the results changed. The discussion mentions 25.1% complexity, which does not match the table.
“a statistically significant, large, but transient increase in project-level development velocity”
Becker, J., Rush, N., Cunningham, T., Rein, D., Mahamud, K. METR
METR repeated its study with more developers and with late-2025 tools. The numbers suggest AI now speeds up the work, but METR does not trust them. Many developers did not want to work without AI and left out tasks to avoid doing them without it. It concludes AI probably speeds up work more today than in 2025, but the data is very weak evidence of the gain's size. It also changed the experiment design.
Numbers
57 developers, 143 repositories, over 800 tasksSize of the new study (10 returning and 47 new).
-18% and -4%Change in time with AI: returning developers (interval -38% to +9%) and new ones (-15% to +9%).
+19%Change in time with AI in the original 2025 study (interval +2% to +39%).
between 30% and 50%Developers who did not submit some tasks to avoid doing them without AI.
Limitation Selection bias that METR acknowledges. Pay also went down, and people who run several agents in parallel measure their time poorly. The intervals cross zero: they do not show a certain gain.
With November 2025 data, collaboration again surpassed automation in the Claude chat. Conversations in which the person only gives orders went down. In the API, used mostly by companies, automation is still about three quarters. Task success is lower in the API, and adjusting for success shrinks the estimated productivity gain. The report says the first net effect may be deskilling, which means jobs lose the tasks that need the most skill.
Numbers
52% and 45%Collaboration and automation conversations in the chat. Directive ones fell to 32% (39% in August 2025).
67% and 49%Task success in the chat and in the API.
from 1.8 to 1.0 to 1.2 pointsEstimated annual contribution to US labor productivity growth, before and after adjusting for success.
Limitation Published by the vendor with its own data: 1 million conversations and 1 million API transcripts (November 13 to 20, 2025).
“the productivity benefits will be smaller than raw speedups suggest”
Quote checked against the original
Read at the source. Checked on Nov 2, 2026.
The second objection is about method. If people overestimate their gain by about 40 points, a team survey does not measure productivity. A METR note estimated time saved from 5,305 real usage transcripts (exhibit 58). They came from 7 of its staff. It found time-saving factors of 1.5 to 13 times. But a language model estimated the time without AI. So the author presents the result as an upper bound, that is, a maximum, not a measured effect. The 30 minutes per ticket from the IT forum is also a self-reported saving.
If the team feels it works faster with agents, it is because it does.
Verdict: Feeling it is not enough. In a METR survey, technical workers say they go 3 times faster with AI. METR itself recalls its randomized study. There, people overestimated their gain by about 40 points. Estimates made from transcripts show savings of 1.5 to 13 times. But their author presents them as an upper bound, that is, a maximum.
Rule
Measure with a control group or a before-and-after on the same kind of task. Do not use perception surveys or self-reported savings as proof of productivity.
Why medium confidence. The roughly 40-point overestimate is recalled by METR from its 2025 study. Its 2026 repeat neither confirms nor rules it out. The survey is a convenience sample, with about a 2% response rate. The supporting side is estimates from 7 people and a self-reported saving on Reddit.
Show the 4 exhibits in detailHide the detail2 for, 1 against, 1 qualifies
An exploratory note that estimates time saved with coding agents. It uses transcripts of real use by METR staff. It gives high saving factors, but the author presents them as a soft upper bound (a rough ceiling), not as a measured effect.
Numbers
5,305 transcripts from 7 peopleBasis of the analysis, from January 2026.
1.5 to 13 timesEstimated time saving factor (per person, 2.11 to 11.62 times).
Limitation The time it would have taken without AI is estimated by a language model, validated with only 34 samples. It covers 7 people and a single tool. It is an upper bound, not an effect measured against a control group.
A thread asks which IT tasks were really automated with AI, and the most upvoted comment answers “none”. On the plus side, one lead built a triage flow (sorting of requests) with read-only access. The flow reads device management, the directory, two-factor authentication, the network and ticket history. The next step is to automate with a person approving from a chat. He says the goal is to grow, not to cut people.
Numbers
30 minutes per ticketSaving on complex tickets, according to the person who built the triage.
67 points and 146 commentsSize of the thread according to the archive. The quoted comment has 47 points and the “none” comment, 134.
Limitation Self-reported saving, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“On more complicated stuff this is saving EASILY 30 minutes per ticket.”
A survey of technical workers on how much value and speed AI adds for them. They say a lot. METR warns that surveys tend to show higher results than experiments. It recalls that in its randomized study (with groups assigned by chance) people overestimated their gain by about 40 points.
Numbers
3 timesMedian self-reported speed change (value: 1.4 to 2 times), in 349 people.
Limitation Convenience sample, with about a 2% response rate and about 70% paid participants. It is self-report, not measurement.
“Importantly, survey results are not necessarily grounded in reality.”
Becker, J., Rush, N., Cunningham, T., Rein, D., Mahamud, K. METR
METR repeated its study with more developers and with late-2025 tools. The numbers suggest AI now speeds up the work, but METR does not trust them. Many developers did not want to work without AI and left out tasks to avoid doing them without it. It concludes AI probably speeds up work more today than in 2025, but the data is very weak evidence of the gain's size. It also changed the experiment design.
Numbers
57 developers, 143 repositories, over 800 tasksSize of the new study (10 returning and 47 new).
-18% and -4%Change in time with AI: returning developers (interval -38% to +9%) and new ones (-15% to +9%).
+19%Change in time with AI in the original 2025 study (interval +2% to +39%).
between 30% and 50%Developers who did not submit some tasks to avoid doing them without AI.
Limitation Selection bias that METR acknowledges. Pay also went down, and people who run several agents in parallel measure their time poorly. The intervals cross zero: they do not show a certain gain.
Read at the source. Checked on Nov 1, 2026.
The third objection: a delivered change is not an accepted change. METR asked four maintainers to review 296 agent pull requests (exhibit 67). A maintainer is the person who decides what goes into a project. They came from scikit-learn, Sphinx and pytest. All the changes already passed the SWE-bench Verified automated grader, a benchmark of real coding tasks. About half would not have been merged. In newer models, the failures were more about code quality than broken functionality. The authors note one limit: the agent had a single attempt, with no chance to iterate with the reviewer.
Simon Willison summed it up in a widely discussed essay on Hacker News (exhibit 35). Whoever submits code has to show that it works. Tests that fail if someone reverts the change do that job. An agent is held to the same standard. In the earlier Hacker News thread, someone said the agent wrote tests that tested nothing.
If the agent's pull request passes the tests, it is ready to merge.
Verdict: No. Maintainers of three projects reviewed agent pull requests that had already passed the SWE-bench Verified grader. About half would not have been merged. In newer models, the failures were more about code quality than broken functionality. In open-source projects that adopted Cursor, code complexity rose 41.64%. Static analysis warnings rose 30.26%.
Rule
Before merging, have someone who knows the code read the change. Check complexity and static analysis warnings, as well as the tests. Ask the agent for tests that fail if the change is reverted.
Why medium confidence. The METR note is a research note, with four maintainers and three repositories. The agent had a single attempt, with no chance to iterate with the reviewer. The quality study looks at open source, not companies. The supporting figure is published by a vendor about a study that could not be opened. It shows more merged pull requests, with no significant change in the revert rate. The Hacker News accounts agree with the note. One says the agent wrote tests that tested nothing.
Show the 5 exhibits in detailHide the detail1 for, 2 against, 2 qualify
It summarizes a University of Chicago study on organizations that use Cursor. After the agent became the default mode, they merged more pull requests (requests to change code) than a baseline group. Junior developers accept more autocomplete, and senior developers accept more agent edits. The most experienced also plan more before they generate code.
Numbers
+39 %Merged pull requests versus the baseline group trend.
+6 %Acceptance rate of agent edits for each additional standard deviation of years of experience (relative to the mean).
Limitation Vendor post about a study (Sarkar, SSRN 5713646) that could not be opened (403). Not randomized. No significant change in the revert rate.
“more experienced developers are more likely to plan an action before generating code”
Four maintainers of scikit-learn, Sphinx and pytest reviewed agent pull requests that already passed the SWE-bench Verified automated grader. About half would not be merged. In newer models, the failures shifted toward code quality more than broken functionality. The authors note it does not prove a capability limit: the agent had a single attempt, with no chance to iterate with the reviewer.
Numbers
24.2 pointsHow much higher the automated grader scores than the maintainers, normalized against human pull requests (standard error 2.7). Unnormalized, 34.9 points.
296 agent pull requestsReviewed, together with 47 human reference ones, on 95 of the 500 problems. Maintainers merged about 68% of the human ones.
about 50 versus about 8 minutesHorizon of Claude Sonnet 4.5 with the automated grader and with maintainer review; the authors say this figure is very noisy.
Limitation A research note, not a peer-reviewed paper. Four maintainers and three repositories. Models from Claude 3.5 Sonnet to Claude Sonnet 4.5 and GPT-5.
“roughly half of test-passing SWE-bench Verified PRs”
GitHub projects that adopted Cursor speed up a lot at first, but the gain fades after two months. More complexity and more static analysis warnings (automatic code checks) remain, and that slows velocity later. Cursor speeds things up for a while and leaves technical debt (problems to fix later).
Numbers
+281.3% and +48.4%Lines added in month 1 and month 2 after adoption, in 806 repositories versus 1,380 controls.
+30.26% and +41.64%Static analysis warnings and code complexity after adoption.
Limitation Difference-in-differences on open source projects, not companies. The figures come from version 3 (January 26, 2026); the title changed between versions and it was not checked whether the results changed. The discussion mentions 25.1% complexity, which does not match the table.
“a statistically significant, large, but transient increase in project-level development velocity”
Whoever submits code has to show it works: test it by hand and leave automated tests that fail if the change is reverted. Coding agents are held to the same standard, and the person remains responsible for what is delivered.
Numbers
859 points and 658 commentsSize of the discussion on Hacker News (item 46313297).
Limitation An opinion essay by a specialist, with no data of its own.
“Your job is to deliver code you have proven to work.”
The author sees a push to review little or no code ("if the tests pass, ship it"). On his existing code, half his time went to fixing subtle errors. The success stories agree on the conditions: code already organized by people, static types, broad tests, narrow tasks and reviewing the plan first. One says the agent wrote tests that tested nothing.
Numbers
461 points and 455 commentsSize of the thread.
Limitation Practitioner opinions. It was read through the Algolia API (post and top-level replies with the most replies).
Read at the source. Checked on Nov 2, 2026.
The fourth objection follows from the third. If every change needs a serious review, review sets the limit. In dotnet/runtime, the bottleneck moved from writing code to reviewing it (exhibit 68). The author opened nine pull requests with the agent in a short while. That created 5 to 9 hours of review for him. With commits from someone on the team, changes succeeded 86.2% of the time. Without those commits, 55.1%. The author puts it this way: “we cannot compromise on review quality, on expert and experienced eyes validating the changes”.
Success with and without commits from a person in the PR.Exhibit 68
Agent pull requests that succeed, in %
With commits from a person86.2%
Without commits from a person55.1%
0100%
An engineer with 11 years of experience automated about 80% of his job (exhibit 79). He said on Reddit that he spends 2 to 3 hours a day reviewing and testing. A commenter told the other side: he dropped the review and quality fell within a week. BCG surveyed about 12,000 frontline workers (exhibit 109). A good share say they spend more time reviewing and correcting AI.
Verdict: Output comes sooner, but finishing depends on review. In dotnet/runtime, the author opened nine pull requests with the agent in a short while. Those changes created 5 to 9 hours of review. One person says they automated about 80% of their job. They still spend 2 to 3 hours a day reviewing and testing.
Rule
Measure the time until the change is accepted, not until the agent delivers. Before adding agents, secure the review capacity.
Why medium confidence. The review figure comes from a single repository and from a Microsoft engineer. The BCG survey also finds a good share of frontline workers spending more time reviewing and correcting AI. But it is self-reported. On the supporting side, one team says an agent cut its median incident time from about 40 to 12 minutes. The Reddit accounts come from one person or one team, with no independent measurement.
Show the 4 exhibits in detailHide the detail1 for, 2 against, 1 qualifies
An agent that handles alerts brought median incident time down from about 40 to 12 minutes. But then an old service that the tools cannot access failed. The 3 people on call spent the first hour working out how it even worked, because nobody had touched it by hand in months. The author proposes handling a few incidents a month by hand, on purpose. He compares this with London cab drivers, who still learn the streets even though they have GPS.
Numbers
from about 40 to 12 minutesMedian incident time with the agent.
105 points and 35 commentsSize of the thread according to the archive. The quoted comment has 81 points.
Limitation One team's story, without independent measurement. A commenter says it looks like an ad, because the post mentions a commercial tool. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“All the Star Trek engineers still actually knew how things worked.”
Ten months of data on the Copilot coding agent in the dotnet/runtime repository. More than half of its pull requests were merged and almost none were reverted, because each one went through expert review. It did best on mechanical, well-scoped tasks (cleanup, tests) and worst on those that need judgment (performance). An instructions file and build setup almost doubled success. The bottleneck moved from writing code to reviewing it.
Numbers
878 PRs, 67.9% successAgent pull requests and success rate (87.1% for PRs by Microsoft staff; populations not comparable).
0.6% vs 0.8%Merged PRs later reverted: the agent's vs the rest.
from 38.1% to 69%Success before and after adding instructions and setting up the build.
86.2% vs 55.1%Success with and without commits from a person in the PR.
5 to 9 hoursReview work created by nine PRs the author opened in a short time.
Limitation Written by a Microsoft engineer about a Microsoft tool. It is a repository with very expert reviewers. The author notes that the agent and the people do not do the same kind of tasks.
“we cannot compromise on review quality, on expert and experienced eyes validating the changes”
Beauchene, Duranton, Martin, Lyon, Walters. Boston Consulting Group
Frontline AI use grew a lot and people say they save time. But most are not told what to do with that time. A good share spend more time reviewing, correcting or directing the AI. Organizations with agents built into their workflows doubled in a year.
Numbers
74% (+23 points)Frontline people who use AI daily or several times a week, out of about 12,000 respondents.
66 %Get little or no guidance on what to do with the time saved.
30% (13% in 2025)Organizations that built agents into their workflows.
Limitation Self-report; BCG sells consulting. Automated download now returns 403, so it carries no verbatim quote.
An engineer with 11 years of experience says he automated about 80% of his job with a coding agent. The agent picks up issues and opens pull requests. He says quality holds because he reviews every change. He spends 2 to 3 hours a day reviewing and testing. One commenter says that when review was removed, quality degraded within a week. Others discuss the ethics of doing it without telling the employer.
Numbers
about 80%Share of the job the author says he automated.
2 to 3 hours a dayTime he spends reviewing and testing.
1,200 points and 276 commentsSize of the thread according to the archive.
Limitation One person's story, with one week of use at the time of the post. Commenters point out it works because the work is simple. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“output quality degraded within a week”
Quote checked against the original
Read at the source. Checked on Nov 2, 2026.
The fifth objection: if AI sped up work that much, it would show up in the economy. In the BCG survey, 66% get little or no guidance on the time they save. A study from Denmark linked chatbot use to earnings and hours worked (exhibit 62). It used administrative data, not surveys. Two years after ChatGPT arrived, it rules out effects larger than 2%.
Anthropic adjusted its estimate with data from its own product (exhibit 42). The adjustment counts only the tasks that succeed. With that adjustment, AI’s annual contribution to US labor productivity fell. It went from 1.8 to between 1.0 and 1.2 points of growth.
AI has already changed how much people earn and how much they work.
Verdict: It is not visible yet. In Denmark, using administrative data, the effects on earnings and hours stay below 2%. This was measured two years after ChatGPT arrived. 66% of frontline workers get little or no guidance. They do not know what to do with the time they save. One vendor adjusts its US productivity estimate for success. With that, the annual contribution to labor productivity growth drops from 1.8 to between 1.0 and 1.2 points.
Rule
Do not put a company-wide saving in the business plan based on studies of individual tasks. Measure the outcome of your process end to end.
Why medium confidence. The Danish study uses solid administrative data, but on chatbots in 2023 and 2024, not agents. The BCG survey is self-reported and comes from a consultancy. The productivity estimate is made by a vendor with its own data. No source in the base shows a measured change in earnings or hours.
Show the 3 exhibits in detailHide the detail0 for, 1 against, 2 qualify
Humlum, A., Vestergaard, E. NBER (working paper 33777)
Administrative data from Denmark that links chatbot use at work with earnings and hours. It finds null effects, measured precisely. It rules out effects larger than 2% on earnings and recorded hours two years after ChatGPT arrived.
Numbers
under 2%Largest effect on earnings and hours that the data allows.
Limitation Data from 2023 and 2024, on chatbots and not agents. The first version, from May 2025, was titled “Large Language Models, Small Labor Market Effects” and ruled out effects larger than 1%. The March 2026 revision changes the results. The bound moves to 2% two years after ChatGPT, and it adds the reorganization of tasks and moves into better-paid occupations. That is why the record uses the revision date. An average time saving of about 3% comes from the first version, was not verified in the revision and is not used.
Beauchene, Duranton, Martin, Lyon, Walters. Boston Consulting Group
Frontline AI use grew a lot and people say they save time. But most are not told what to do with that time. A good share spend more time reviewing, correcting or directing the AI. Organizations with agents built into their workflows doubled in a year.
Numbers
74% (+23 points)Frontline people who use AI daily or several times a week, out of about 12,000 respondents.
66 %Get little or no guidance on what to do with the time saved.
30% (13% in 2025)Organizations that built agents into their workflows.
Limitation Self-report; BCG sells consulting. Automated download now returns 403, so it carries no verbatim quote.
With November 2025 data, collaboration again surpassed automation in the Claude chat. Conversations in which the person only gives orders went down. In the API, used mostly by companies, automation is still about three quarters. Task success is lower in the API, and adjusting for success shrinks the estimated productivity gain. The report says the first net effect may be deskilling, which means jobs lose the tasks that need the most skill.
Numbers
52% and 45%Collaboration and automation conversations in the chat. Directive ones fell to 32% (39% in August 2025).
67% and 49%Task success in the chat and in the API.
from 1.8 to 1.0 to 1.2 pointsEstimated annual contribution to US labor productivity growth, before and after adjusting for success.
Limitation Published by the vendor with its own data: 1 million conversations and 1 million API transcripts (November 13 to 20, 2025).
“the productivity benefits will be smaller than raw speedups suggest”
AI probably speeds up output on bounded tasks, with tests and with someone who reviews.The size of the gain is uncertain. All six claims carry medium confidence, and they say the same thing.
The big numbers count merged pull requests, and almost none measure quality. The only study that measures quality after adoption found technical debt.
The rule of method holds firm. What people feel measures nothing. The time that matters ends when the change is accepted, not when the agent delivers or the tests pass. That time has a ceiling: the hours of whoever reviews.
The economy does not show the jump yet. That does not prove it will not come. But a business plan cannot take the 24.0% from one rollout and apply it to the whole company.
Measure the gain on your own work, end to end, against doing it without the tool. Count until the change is accepted. Measure quality again a few months later.
Now:
Pick a type of bounded task that has an automated check, such as tests or a build.
Write the agent’s instructions file. Leave the build ready for the agent to run.
Measure with a control group or a before-and-after on the same kind of task. If you can, pick the control group, because it gives stronger evidence. Do not use perception surveys as proof.
Before merging, have someone who knows the code read the change. Check complexity and static analysis warnings (automatic review of the code without running it). Ask the agent for tests that fail if the change is reverted.
Log review hours alongside the agent’s time. If review cannot keep up, do not add more agents.
The evaluation plan helps you build the tasks and set the exit criterion. The cost per successful task calculator adds review hours to the model cost. The next chapter covers junior staff when the agent does their tasks.