Skip to content
Agents on Trial

Agents on Trial

13 chapters65 claims150 exhibitsChecked on

When is an AI agent worth it in a company, and when is it not?

Every claim about agents has evidence for, evidence against and a verdict. The verdict turns into a design rule.

Read the trialSee the 150 exhibits

The 150 exhibits, by chapter

  • 25for
  • 70qualify
  • 55against
  1. Chapter 1. The promise and the gap, cites 20 exhibits in total
  2. Chapter 2. Agent or fixed workflow?, cites 17 exhibits in total
  3. Chapter 3. The error that multiplies, cites 17 exhibits in total
  4. Chapter 4. One agent or several?, cites 10 exhibits in total
  5. Chapter 5. Security by design, cites 28 exhibits in total
  6. Chapter 6. Connecting systems: MCP and A2A, cites 21 exhibits in total
  7. Chapter 7. People in the loop, cites 17 exhibits in total
  8. Chapter 8. Evaluate before you promise, cites 24 exhibits in total
  9. Chapter 9. Cost per successful task, cites 18 exhibits in total
  10. Chapter 10. Real productivity, cites 19 exhibits in total
  11. Chapter 11. Work, ethics and learning, cites 22 exhibits in total
  12. Chapter 12. Regulation, cites 15 exhibits in total
  13. Chapter 13. The decision framework, cites 23 exhibits in total

Each bar is a source with a number. Above supports, below contradicts and on the line qualifies: it adds a condition or a limit. Each exhibit appears once, in the first chapter it weighs on, with its overall position.

That is why a cluster can be small: the number below says how many exhibits the chapter cites in total.

With the keyboard: Tab to the axis, the arrow keys move through the exhibits and Enter opens the card. Without JavaScript, each bar links to its exhibit in the evidence.

Height shows what kind of source it is.

  1. 56 documents with a method or official textspaper, regulation, specification, record, ruling
  2. 11 reported datasurvey, pricing
  3. 83 accountstechnical post, Reddit, Hacker News, press

The 13 chapters

Each one tests an idea about agents and ends in a design rule: what to do. The one that cites the most exhibits is 05, Security by design: 28.

  1. Chapter 1. The promise and the gap. 5 claims and 20 exhibits.Does the fact that almost every company uses AI prove that agents already pay off?
  2. Chapter 2. Agent or fixed workflow?. 4 claims and 17 exhibits.When should the model choose the path, and when should you write it in code?
  3. Chapter 3. The error that multiplies. 4 claims and 17 exhibits.Is it enough for each step to be right almost every time for the agent to be reliable?
  4. Chapter 4. One agent or several?. 4 claims and 10 exhibits.Does splitting the work across several agents make it better, or just more expensive?
  5. Chapter 5. Security by design. 6 claims and 28 exhibits.Is a well-aligned model with a good filter enough to make an agent safe?
  6. Chapter 6. Connecting systems: MCP and A2A. 5 claims and 21 exhibits.Does a connection standard make the connection safe?
  7. Chapter 7. People in the loop. 5 claims and 17 exhibits.Does putting a person in the loop control an agent's risk?
  8. Chapter 8. Evaluate before you promise. 6 claims and 24 exhibits.Does a benchmark score tell you how the agent will perform at your company?
  9. Chapter 9. Cost per successful task. 5 claims and 18 exhibits.How much does it really cost for an agent to do a task well?
  10. Chapter 10. Real productivity. 6 claims and 19 exhibits.Do agents get work finished sooner, or do they only make it feel faster?
  11. Chapter 11. Work, ethics and learning. 5 claims and 22 exhibits.If the agent does the work, what happens to the people who did it, to what they knew and to accountability?
  12. Chapter 12. Regulation. 5 claims and 15 exhibits.Are there rules an agent has to follow today, or is there still time?
  13. Chapter 13. The decision framework. 5 claims and 23 exhibits.With everything that was tested, how do you decide whether, where and how to deploy an agent?

One claim, face to face

Section titled “One claim, face to face”

Face to face means that its exhibits stand against each other. This is what every claim in the trial looks like.

At the top: the claim, the confidence of the verdict and the rule: what to do with what the exhibits show. Below: the exhibits. The ones for and the ones against face each other. The ones that qualify, that is, add a condition or a limit, sit on the axis.

Open each exhibit to see what the source found, what kind of source it is, when it is from and whether we read it in the original.

AI speeds up people's real work.

Medium confidence

Chapter 10Real productivity

For

Against

Qualifies

Verdict

Verdict: It probably speeds up output, but the size of the effect is uncertain. In a randomized experiment with an assistant, everyone improved on a business task. In open-source projects that adopted Cursor, lines added rose 281.3% in the first month. But the gain faded after two months. METR repeated its study in 2026. METR believes AI probably speeds work up more than in 2025, although it does not trust its own numbers.

Rule

Do not assume the gain: measure it on your own work, against doing the task without the tool. Measure quality again a few months later, because the early speed can leave debt behind.

Why medium confidence. The randomized experiment uses a chat assistant, not agents. The data on coding agents are observational. One is from Microsoft on its own rollout and one is on open-source projects. METR's repeat study has intervals that cross zero. METR distrusts its own figure because of selection bias. A vendor that estimates productivity from its own data shrinks it after adjusting for success.

Show the 5 exhibits in detailHide the detail2 for, 1 against, 2 qualify

For

  1. Exhibit049Paper

    Does Generative AI Narrow Education-Based Productivity Gaps? Evidence from a Randomized Experiment

    Cruces, G., Fernández Meijide, D., Galiani, S., Gálvez, R. H., Lombardi, M. NBER (Working Paper 34851)

    It is a randomized experiment (groups assigned at random) with a generative AI assistant, on a task to solve a business problem. Everyone improves, and people with less education improve more. The gap shrinks a lot but does not disappear, and when the AI is removed it largely returns. People with more education use the tool better.

    Numbers

    • from 0.548 to 0.139 standard deviationsPerformance gap by education without and with AI (about three quarters closes), in 1,174 adults aged 25 to 45.

    Limitation The NBER page with the abstract was read. It measures education, not experience; it is a chat assistant, not an agent, and the task is online, outside a company. It has revisions from May and October 2026.

    Read at the source. Checked on Nov 2, 2026.

  2. Exhibit116Paper

    Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

    Murphy-Hill, E., Butler, J., Savelieva, A. arXiv (Microsoft)

    It studies the rollout of two terminal coding agents among tens of thousands of Microsoft engineers. Those who adopted them merged more pull requests than a synthetic control group. The most junior engineers (IC2) had the largest increase in pull requests created: +34%, against +21% for the reference level (IC4). By tenure, the gain is C-shaped: people with short and long tenure gain most. The adoption study covers only Copilot CLI: there, junior engineers tried it less at first. Use spread mostly through networks of colleagues. It does not measure quality.

    Numbers

    • +24.0% (+14.5% to +33.7%)Pull requests merged per engineer per day versus control, over a 4-month window (95% interval).
    • +34% versus +21%Increase in pull requests created with 3 days of use per week, versus the same person's weeks without use. It compares level IC2 versus IC4, the reference (IC3: +21%, IC5: +25%, IC6: +22%). Read from figure 9, with Claude Code and Copilot CLI.
    • -13% and -14%Odds of trying Copilot CLI early for levels IC2 and IC3 versus IC4 (IC5: +22%). Copilot CLI only: the adoption study does not include Claude Code.

    Limitation Observational, not randomized. The authors are from Microsoft, which owns GitHub, and they say so. A pull request is not delivered value. The gain by level compares each person's weeks with and without use: it is an association, not a cause. The adoption study includes only Copilot CLI.

    “first use spread primarily through social networks”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

Against

  1. Exhibit017Paper

    Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects

    He, H., Miller, C., Agarwal, S., Kästner, C., Vasilescu, B. arXiv (Carnegie Mellon)

    GitHub projects that adopted Cursor speed up a lot at first, but the gain fades after two months. More complexity and more static analysis warnings (automatic code checks) remain, and that slows velocity later. Cursor speeds things up for a while and leaves technical debt (problems to fix later).

    Numbers

    • +281.3% and +48.4%Lines added in month 1 and month 2 after adoption, in 806 repositories versus 1,380 controls.
    • +30.26% and +41.64%Static analysis warnings and code complexity after adoption.

    Limitation Difference-in-differences on open source projects, not companies. The figures come from version 3 (January 26, 2026); the title changed between versions and it was not checked whether the results changed. The discussion mentions 25.1% complexity, which does not match the table.

    “a statistically significant, large, but transient increase in project-level development velocity”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

Qualifies

  1. Exhibit060Technical post

    We are Changing our Developer Productivity Experiment Design

    Becker, J., Rush, N., Cunningham, T., Rein, D., Mahamud, K. METR

    METR repeated its study with more developers and with late-2025 tools. The numbers suggest AI now speeds up the work, but METR does not trust them. Many developers did not want to work without AI and left out tasks to avoid doing them without it. It concludes AI probably speeds up work more today than in 2025, but the data is very weak evidence of the gain's size. It also changed the experiment design.

    Numbers

    • 57 developers, 143 repositories, over 800 tasksSize of the new study (10 returning and 47 new).
    • -18% and -4%Change in time with AI: returning developers (interval -38% to +9%) and new ones (-15% to +9%).
    • +19%Change in time with AI in the original 2025 study (interval +2% to +39%).
    • between 30% and 50%Developers who did not submit some tasks to avoid doing them without AI.

    Limitation Selection bias that METR acknowledges. Pay also went down, and people who run several agents in parallel measure their time poorly. The intervals cross zero: they do not show a certain gain.

    Read at the source. Checked on Nov 1, 2026.

  2. Exhibit042Paper

    Anthropic Economic Index report: Economic primitives

    Anthropic

    With November 2025 data, collaboration again surpassed automation in the Claude chat. Conversations in which the person only gives orders went down. In the API, used mostly by companies, automation is still about three quarters. Task success is lower in the API, and adjusting for success shrinks the estimated productivity gain. The report says the first net effect may be deskilling, which means jobs lose the tasks that need the most skill.

    Numbers

    • 52% and 45%Collaboration and automation conversations in the chat. Directive ones fell to 32% (39% in August 2025).
    • 67% and 49%Task success in the chat and in the API.
    • from 1.8 to 1.0 to 1.2 pointsEstimated annual contribution to US labor productivity growth, before and after adjusting for success.

    Limitation Published by the vendor with its own data: 1 million conversations and 1 million API transcripts (November 13 to 20, 2025).

    “the productivity benefits will be smaller than raw speedups suggest”

    Quote checked against the original

    Read at the source. Checked on Nov 2, 2026.

To decide, tools and templates

Tools3

They calculate and decide with the repository’s tested logic. The result stays in the link, ready to share.

Templates7

In Markdown, ready to copy: decide on, design, evaluate and run an agent.

How the evidence was built: which sources get in, which rules they follow and how we decide the side and the confidence.

Set in Libre Franklin, Source Serif 4 and Red Hat Mono, served from the site itself. Evidence base checked on Nov 2, 2026.