Skip to content
Agents on Trial

The trial

Thirteen chapters that test every claim about AI agents in the enterprise. Each chapter puts one question on trial. They all follow the same order:

  1. The thesis.
  2. The evidence for.
  3. The evidence against.
  4. The verdict.
  5. The design rule that follows from the verdict.

The chapters

Section titled “The chapters”
  1. The promise and the gap. Does the fact that almost every company uses AI prove that agents already pay off?
  2. Agent or fixed workflow? When should the model choose the path, and when should you write it in code?
  3. The error that multiplies. Is it enough for each step to be right almost every time for the agent to be reliable?
  4. One agent or several? Does splitting the work across several agents make it better, or just more expensive?
  5. Security by design. Is a well-aligned model with a good filter enough to make an agent safe?
  6. Connecting systems: MCP and A2A. Does a connection standard make the connection safe?
  7. People in the loop. Does putting a person in the loop control an agent’s risk?
  8. Evaluate before you promise. Does a benchmark score tell you how the agent will perform at your company?
  9. Cost per successful task. How much does it really cost for an agent to do a task well?
  10. Real productivity. Do agents get work finished sooner, or do they only make it feel faster?
  11. Work, ethics and learning. If the agent does the work, what happens to the people who did it, to what they knew and to accountability?
  12. Regulation. Are there rules an agent has to follow today, or is there still time?
  13. The decision framework. With everything that was tested, how do you decide whether, where and how to deploy an agent?

How to read a claim

Section titled “How to read a claim”

Every claim in the text appears as a card like the one below.

  • The confidence label says how much weight the evidence behind the verdict carries: high, medium or low. The three marks show the same thing.
  • The verdict sums up what the evidence shows, for and against.
  • The rule is the design rule you take away.
  • The exhibits open below, face to face. The evidence for goes on one side and the evidence against on the other. The exhibits that qualify the claim go below, on the axis.
  • Each exhibit has a number, the type of source, the date and what it found. It also gives its limits and whether it could be read in the original.

AI speeds up people's real work.

Medium confidence

Chapter 10Real productivity

For

Against

Qualifies

Verdict

Verdict: It probably speeds up output, but the size of the effect is uncertain. In a randomized experiment with an assistant, everyone improved on a business task. In open-source projects that adopted Cursor, lines added rose 281.3% in the first month. But the gain faded after two months. METR repeated its study in 2026. METR believes AI probably speeds work up more than in 2025, although it does not trust its own numbers.

Rule

Do not assume the gain: measure it on your own work, against doing the task without the tool. Measure quality again a few months later, because the early speed can leave debt behind.

Why medium confidence. The randomized experiment uses a chat assistant, not agents. The data on coding agents are observational. One is from Microsoft on its own rollout and one is on open-source projects. METR's repeat study has intervals that cross zero. METR distrusts its own figure because of selection bias. A vendor that estimates productivity from its own data shrinks it after adjusting for success.

Show the 5 exhibits in detailHide the detail2 for, 1 against, 2 qualify

For

  1. Exhibit049Paper

    Does Generative AI Narrow Education-Based Productivity Gaps? Evidence from a Randomized Experiment

    Cruces, G., Fernández Meijide, D., Galiani, S., Gálvez, R. H., Lombardi, M. NBER (Working Paper 34851)

    It is a randomized experiment (groups assigned at random) with a generative AI assistant, on a task to solve a business problem. Everyone improves, and people with less education improve more. The gap shrinks a lot but does not disappear, and when the AI is removed it largely returns. People with more education use the tool better.

    Numbers

    • from 0.548 to 0.139 standard deviationsPerformance gap by education without and with AI (about three quarters closes), in 1,174 adults aged 25 to 45.

    Limitation The NBER page with the abstract was read. It measures education, not experience; it is a chat assistant, not an agent, and the task is online, outside a company. It has revisions from May and October 2026.

    Read at the source. Checked on Nov 2, 2026.

  2. Exhibit116Paper

    Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

    Murphy-Hill, E., Butler, J., Savelieva, A. arXiv (Microsoft)

    It studies the rollout of two terminal coding agents among tens of thousands of Microsoft engineers. Those who adopted them merged more pull requests than a synthetic control group. The most junior engineers (IC2) had the largest increase in pull requests created: +34%, against +21% for the reference level (IC4). By tenure, the gain is C-shaped: people with short and long tenure gain most. The adoption study covers only Copilot CLI: there, junior engineers tried it less at first. Use spread mostly through networks of colleagues. It does not measure quality.

    Numbers

    • +24.0% (+14.5% to +33.7%)Pull requests merged per engineer per day versus control, over a 4-month window (95% interval).
    • +34% versus +21%Increase in pull requests created with 3 days of use per week, versus the same person's weeks without use. It compares level IC2 versus IC4, the reference (IC3: +21%, IC5: +25%, IC6: +22%). Read from figure 9, with Claude Code and Copilot CLI.
    • -13% and -14%Odds of trying Copilot CLI early for levels IC2 and IC3 versus IC4 (IC5: +22%). Copilot CLI only: the adoption study does not include Claude Code.

    Limitation Observational, not randomized. The authors are from Microsoft, which owns GitHub, and they say so. A pull request is not delivered value. The gain by level compares each person's weeks with and without use: it is an association, not a cause. The adoption study includes only Copilot CLI.

    “first use spread primarily through social networks”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

Against

  1. Exhibit017Paper

    Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects

    He, H., Miller, C., Agarwal, S., Kästner, C., Vasilescu, B. arXiv (Carnegie Mellon)

    GitHub projects that adopted Cursor speed up a lot at first, but the gain fades after two months. More complexity and more static analysis warnings (automatic code checks) remain, and that slows velocity later. Cursor speeds things up for a while and leaves technical debt (problems to fix later).

    Numbers

    • +281.3% and +48.4%Lines added in month 1 and month 2 after adoption, in 806 repositories versus 1,380 controls.
    • +30.26% and +41.64%Static analysis warnings and code complexity after adoption.

    Limitation Difference-in-differences on open source projects, not companies. The figures come from version 3 (January 26, 2026); the title changed between versions and it was not checked whether the results changed. The discussion mentions 25.1% complexity, which does not match the table.

    “a statistically significant, large, but transient increase in project-level development velocity”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

Qualifies

  1. Exhibit060Technical post

    We are Changing our Developer Productivity Experiment Design

    Becker, J., Rush, N., Cunningham, T., Rein, D., Mahamud, K. METR

    METR repeated its study with more developers and with late-2025 tools. The numbers suggest AI now speeds up the work, but METR does not trust them. Many developers did not want to work without AI and left out tasks to avoid doing them without it. It concludes AI probably speeds up work more today than in 2025, but the data is very weak evidence of the gain's size. It also changed the experiment design.

    Numbers

    • 57 developers, 143 repositories, over 800 tasksSize of the new study (10 returning and 47 new).
    • -18% and -4%Change in time with AI: returning developers (interval -38% to +9%) and new ones (-15% to +9%).
    • +19%Change in time with AI in the original 2025 study (interval +2% to +39%).
    • between 30% and 50%Developers who did not submit some tasks to avoid doing them without AI.

    Limitation Selection bias that METR acknowledges. Pay also went down, and people who run several agents in parallel measure their time poorly. The intervals cross zero: they do not show a certain gain.

    Read at the source. Checked on Nov 1, 2026.

  2. Exhibit042Paper

    Anthropic Economic Index report: Economic primitives

    Anthropic

    With November 2025 data, collaboration again surpassed automation in the Claude chat. Conversations in which the person only gives orders went down. In the API, used mostly by companies, automation is still about three quarters. Task success is lower in the API, and adjusting for success shrinks the estimated productivity gain. The report says the first net effect may be deskilling, which means jobs lose the tasks that need the most skill.

    Numbers

    • 52% and 45%Collaboration and automation conversations in the chat. Directive ones fell to 32% (39% in August 2025).
    • 67% and 49%Task success in the chat and in the API.
    • from 1.8 to 1.0 to 1.2 pointsEstimated annual contribution to US labor productivity growth, before and after adjusting for success.

    Limitation Published by the vendor with its own data: 1 million conversations and 1 million API transcripts (November 13 to 20, 2025).

    “the productivity benefits will be smaller than raw speedups suggest”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

Every source, with filters by type and by stance, is in the evidence.