A team builds an agent that works well in the pilot. In production it starts failing more often, and nobody changed anything. Someone opened a thread about it on an agents forum (exhibit 101).
A commenter did the math. If each step succeeds 90% of the time, a 5-step workflow fails about 41%. The reason is that 0.9 to the fifth power is about 0.59.
90% per step, 41% failure over 5 steps
A commenter's math (0.9 to the power of 5 is about 0.59). Exhibit 101
Another commenter added that it is a system problem, and more code alone will not fix it.
The thesis on trial is the intuition that this math breaks: if each step is right 90%, the agent is reliable. In a demo you see the per-step number. In production you pay for the whole process.
Reliability here means the agent behaves correctly over time, not once.
Behind it sits a bigger bet: the next model, more context or longer tasks will fix the problem. The bet has data behind it.
Models complete longer and longer tasks, and that supports the bet.
METR measures the time horizon of models: how long an expert takes on the tasks that AI completes with 50% success. According to its data page (exhibit 55), that horizon has doubled every 128.7 days since 2023, about 4.2 months.
Claude Opus 4.5 reaches 293 minutes, with an interval of 162 to 624. The interval is the range where the real value probably sits.
Claude Opus 4.6, which came out later, reaches 719 minutes. These are software, machine learning and cybersecurity tasks.
With a larger set of 228 tasks (exhibit 47), the overall trend barely changes. Measured from 2024, the doubling is even faster: 88.6 days. Read this way, per-step success is not fixed: it rises with each generation of models, and so does the length of the tasks.
The other half of the bet is giving the model more material: if it sees everything, it decides better. No source in the base backs that with data, but the intuition is strong.
The success of the chain is the per-step success multiplied by itself, once per step. In another thread (exhibit 111), the math is done with 95% per step:
A 5-step process succeeds 77% of the time.
A 10-step one succeeds 60% of the time.
A 20-step one succeeds less than 36% of the time.
The author of that post concluded that serious work cannot be automated profitably. A commenter suggested measuring the cost per successful outcome, the topic of chapter 9.
The math assumes independent errors, meaning each step fails regardless of what happened before. Both threads are small and bring no field data.
Even so, no source in the base argues that 90% per step is enough. You can try your own numbers in the compound reliability calculator.
The math also shows the way out. A paper on a million-step task (exhibit 19) splits the work into minimal steps. At each step several answers vote until one leads by 3 votes.
That way, a small model without reasoning completed the Towers of Hanoi with 20 disks. It took 1,048,575 steps without a single error. Without reasoning means without the mode that thinks before it answers.
Without the method, the authors write, “the process inevitably becomes derailed after at most a few hundred steps”. It is a preprint, a paper that other experts have not reviewed yet. Also, the steps were defined in advance: it is pure execution, not discovery.
If each step is right 90% of the time, the agent is reliable.
Verdict: No: errors multiply. At 90% per step, a 5-step workflow fails about 41% of the time. At 95% per step, a 20-step one succeeds less than 36% of the time. That math assumes independent errors. The flip side: a small model split the task into minimal steps and voted on each one. That way it completed 1,048,575 steps without a single error.
Rule
Compute the reliability of the whole process: per-step success raised to the number of steps. If it falls short of what the task needs, shorten the chain or add a check between steps.
Why medium confidence. The math is arithmetic and assumes independent errors. The two Reddit threads that do it are small. The million-step paper is a preprint (a paper without peer review). Its steps were defined in advance: it is pure execution, not discovery. No source argues that 90% per step is enough.
Show the 3 exhibits in detailHide the detail0 for, 2 against, 1 qualifies
A thread about agents that work in the pilot and degrade in production. One comment does the math: with 90% success per step, a 5-step workflow fails about 41% of the time. This happens because errors multiply. Another says it is a system problem, and more code alone does not fix it.
Numbers
90% per step, 41% failure over 5 stepsA commenter's math (0.9 to the power of 5 is about 0.59).
Limitation Small thread (6 points and 30 comments in the archive). The math is arithmetic, without field data, and assumes independent errors. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“compounding errors is a systematic problem that you cannot code your way out of”
The post argues that price per token is the wrong metric, because you pay the same whether the result is right or wrong. Reliability falls as the chain of steps gets longer, so the cost of checking has to be added. The author concludes that serious work cannot be automated cost-effectively. A comment proposes something more useful: measuring cost per successful outcome.
Numbers
77%, 60% and under 36%Reliability of a 5, 10 and 20 step process with 95% success per step, if errors were independent.
Limitation A very small thread (2 points and 6 comments in the archive) and the post promotes an article by the author. The math is arithmetic, not a measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Cost per successful outcome seems like a far more useful metric than cost per token”
Meyerson, E., Paolo, G., Dailey, R., et al. arXiv (Cognizant AI Lab, UT Austin)
The other side of compound reliability: they split the task into minimal steps, each handled by a microagent. At each step they vote among several answers until one leads by 3 votes, and they discard answers that show signs of confusion. That way they complete the Towers of Hanoi with 20 disks without a single error, using a small model without reasoning. The fix is not a bigger model, but decomposing and verifying each step.
Numbers
1,048,575 stepsLength of the task solved without errors, with gpt-4.1-mini.
about 0.22%Per-step error of the model used.
about 3,500 dollarsProjected cost for 95% success; the paper does not give the actual cost of the run.
Limitation Preprint. The steps have to be defined in advance and the strategy is given to the model in each prompt: it is pure execution, not discovery. The analysis assumes independent errors.
“the process inevitably becomes derailed after at most a few hundred steps”
The 293 minutes do not mean the agent can work alone for almost five hours. The authors of the metric make that clear in a note (exhibit 44). The horizon says how much human work AI completes with 50% success.
On visual computer-use tasks, the horizon is 40 to 100 times shorter. On a real physical task, making coffee, Claude 4.5 Sonnet has a horizon of about 2 minutes.
For work that needs 98% success or more, a 50% horizon is no guide. If you ask for 80% success, Claude Opus 4.5’s horizon drops from 293 to 49 minutes.
Who grades the work also changes the figure. Another METR note (exhibit 67) used agents’ pull requests, that is, requests to change code. Those changes had already passed the SWE-bench Verified automatic grader.
Four maintainers of scikit-learn, Sphinx and pytest reviewed them. About half did not pass that review.
With maintainers as judges, Claude Sonnet 4.5’s horizon drops from about 50 to about 8 minutes. The authors warn that this figure is very noisy.
Horizon of Claude Sonnet 4.5 with the automated grader and with maintainer review; the authors say this figure is very noisy.Exhibit 67
Time horizon, in minutes
Automated graderabout 50 minutes
Maintainersabout 8 minutes
0
If an agent solves hours-long tasks, I can already delegate hours of work to it.
Verdict: No. The horizon measures how much human work the AI solves with 50% success. It does not measure how long it can work alone. For Claude Opus 4.5 it is 293 minutes, with an interval of 162 to 624. But at 80% success it drops to 49. On visual computer-use tasks it is 40 to 100 times shorter. When maintainers judge instead of the automatic grader, Claude Sonnet 4.5's drops from about 50 to about 8 minutes.
Rule
Set the success rate the task needs before delegating it, and measure the agent against that bar, in your domain. Split long work into short stretches, with a check at the end of each one.
Why medium confidence. The METR sources are primary, but they come from the same group. They have very wide intervals and tasks in software, machine learning and cybersecurity. The authors say the maintainer figure is very noisy. The cost near the limit comes from a thread about a 2025 essay.
Show the 5 exhibits in detailHide the detail1 for, 2 against, 2 qualify
METR data page with time horizon 1.1. It measures the length of tasks that each model solves with 50% and with 80% success, in the time of an expert person. The trend is still exponential, and since 2023 the horizon doubles faster than in the 2025 paper. But the 80% success horizon is 4 to 10 times shorter than the 50% one. METR warns that above 16 hours its measurement is no longer reliable.
Numbers
128.7 days (interval of 104.4 to 158.0)Time for the 50% horizon to double, counting from 2023 (about 4.2 months). Since 2019 it is 187.8 days. It excludes measurements above 16 hours.
293 minutes (162 to 624) and 49 minutes (21 to 105)Horizon of Claude Opus 4.5 with 50% and with 80% success, with the data corrected on March 3, 2026.
719 minutes (317 to 3,634) and 70 minutesHorizon of Claude Opus 4.6 with 50% and with 80% success.
1,045 minutes (509 to 3,304) and 186 minutesHorizon of Claude Mythos Preview (early version, April 2026) with 50% and with 80% success: about 17.4 hours and 3.1 hours. It is above the 16-hour ceiling that METR considers reliable.
352, 342 and 384 minutes50% horizon of GPT-5.2, GPT-5.4 and Gemini 3.1 Pro.
Limitation The figures come from the page data file (benchmark_results_1_1.yaml). On March 3, 2026 METR corrected a regularization mistake, so 1.1 figures published before that date do not match. On September 8, 2026 METR said the page is no longer actively updated. The tasks are software, machine learning and cybersecurity, and 50% success is a low bar for delegating.
“Measurements above 16 hrs are unreliable with our current task suite.”
A note by the metric's own authors. The time horizon does not say how long an agent can work alone. It says how much human work AI replaces with 50% success. The figure changes a lot by domain and does not carry over directly to real work, which is messier.
Numbers
4 h 49 minHorizon of Claude Opus 4.5, with a 95% interval from 1 h 49 min to 20 h 25 min.
40 to 100 times shorterHorizon on visual computer-use tasks.
about 2 minutesHorizon of Claude 4.5 Sonnet on a real physical task: making coffee.
Limitation It is a note by the authors about their own metric, not a separate study. The 4 h 49 min figure is a measurement with the TH1 series. With TH1.1, after METR's correction, the same model gets 293 minutes (record “Time Horizon 1.1”). For work that needs 98% success or more, a 50% horizon is not a useful guide.
Four maintainers of scikit-learn, Sphinx and pytest reviewed agent pull requests that already passed the SWE-bench Verified automated grader. About half would not be merged. In newer models, the failures shifted toward code quality more than broken functionality. The authors note it does not prove a capability limit: the agent had a single attempt, with no chance to iterate with the reviewer.
Numbers
24.2 pointsHow much higher the automated grader scores than the maintainers, normalized against human pull requests (standard error 2.7). Unnormalized, 34.9 points.
296 agent pull requestsReviewed, together with 47 human reference ones, on 95 of the 500 problems. Maintainers merged about 68% of the human ones.
about 50 versus about 8 minutesHorizon of Claude Sonnet 4.5 with the automated grader and with maintainer review; the authors say this figure is very noisy.
Limitation A research note, not a peer-reviewed paper. Four maintainers and three repositories. Models from Claude 3.5 Sonnet to Claude Sonnet 4.5 and GPT-5.
“roughly half of test-passing SWE-bench Verified PRs”
METR expanded its task set and re-estimated the time horizon. The horizon is how long a person takes on the tasks that AI solves with 50% success. The overall trend barely changes, but the trend of recent years is faster.
Numbers
228 tasks (previously 170)Size of the task set.
130.8 days and 88.6 daysTime for the horizon to double, measured since 2023 (interval of 107 to 161 days) and since 2024.
293 minutes50% horizon of Claude Opus 4.5 after METR's correction of March 3, 2026, with an interval of 162 to 624 minutes. The January post, written before the correction, gave 320 (170 to 729).
Limitation On March 3, 2026 METR corrected a regularization mistake that changed the measurements. The post still shows the January figures. The Claude Opus 4.5 figure in this entry is the corrected one, taken from METR's time horizons page. 50% success is a low bar for delegating critical work. Confidence intervals are very wide. Of 31 tasks of 8 hours or more, only 5 have a measured human baseline. The tasks are software, machine learning and cybersecurity.
A thread about an essay by Toby Ord that uses METR data to estimate the hourly cost of agents. That cost rises sharply as the task nears the limit of what the model can do, right where it fails most. The strongest counterpoint in the thread is that cheaper open models complete the same set of tasks for much less.
Numbers
about USD 350 per hourCost of o3 at its full horizon, where it still fails about half the time.
306 points and 137 commentsSize of the thread.
Limitation The essay is from 2025 and uses that year's models. The thread was read on news.ycombinator.com and through the public Algolia API, not the full essay. Quotes from the thread were not checked word by word in the source verification, so none is used.
Context is everything the model has in front of it when it decides: instructions, documents, history and tool results. A late 2025 preprint (exhibit 7) tested five models with all the relevant information already retrieved. Even so, performance fell by 13.9% to 85% as the input grew.
The effect held when they replaced the extra text with whitespace. Length alone, with no distraction, already hurts.
Tools take up context too. According to Anthropic (exhibit 23), 58 tools from 5 MCP servers took up about 55,000 tokens before any work started.
MCP is a standard for connecting tools to a model. A token is roughly part of a word.
When the model searched for tools on demand, Opus 4.5’s accuracy rose from 79.5% to 88.1%. It is a product announcement with internal evaluations.
According to the Fly.io post (exhibit 16), past a threshold the system gets worse without warning. The fixes that work take things out of the context.
On long tasks, Anthropic found that compacting the history was not enough (exhibit 26). Compacting means summarizing the conversation so it takes less space. What worked was keeping the state outside, in a progress file and in git commits.
Chroma, which sells search, goes the same way: its subagent works within a fixed token budget (exhibit 72). A subagent is an agent that another agent calls for part of a task.
The more context you give the agent, the better it decides.
Verdict: No: more input does not guarantee better use. Five models retrieved all the relevant information. Even so, they performed between 13.9% and 85% worse as the input grew. According to one vendor, 58 tools took up about 55,000 tokens before starting. In its evaluations, searching for them on demand raised accuracy from 79.5% to 88.1%. On long tasks, what worked was moving state out of the context.
Rule
Treat context as a budget: include only what the decision needs and load tools when they are needed. On long tasks, keep state in files or commits, not in the conversation.
Why medium confidence. The input-length study is a preprint with five models, without peer review, and only the abstract was read. The rest are vendor posts with their own cases and an opinion essay. One of those posts comes from a company that sells search. No source argues that more context is always better.
Show the 5 exhibits in detailHide the detail0 for, 3 against, 2 qualify
Even when it retrieves all the relevant information, the model performs worse as the input grows. This happens within the length it claims to support, and it holds if the extra tokens are whitespace or fully masked. Length alone, with no distraction, already hurts. Asking the model to restate the evidence before solving helps a little.
Numbers
13.9% to 85%Performance drop as the input grows, in 5 open and closed models and math, question and code tasks.
up to 4%GPT-4o improvement on RULER when asked to restate the evidence.
Limitation Preprint; the abstract was read. Five models.
“degrades substantially (13.9%--85%) as input length increases”
Anthropic adds three features: searching tools on demand, calling them from code and adding usage examples to the definition. The code runs in a sandbox, an isolated environment. It shows that many tool definitions use up context and lower accuracy. It proposes loading tools when they are needed, not having few tools.
Numbers
about 55,000 tokensWhat 58 tools from 5 MCP servers take up before starting.
about 8,700 versus 77,000 tokensContext used with and without tool search (85% less).
from 79.5% to 88.1%Opus 4.5 accuracy on MCP evaluations without and with tool search (Opus 4: from 49% to 74%).
Limitation Vendor product announcement; the evaluations are internal.
An agent is a model call inside a loop with tools, and writing one is simple. The hard part is managing context: past a threshold, the system gets worse without warning. Connecting the agent to reality, so it does not believe it solved something it did not, remains an open problem.
Numbers
1,070 points and 395 commentsSize of the discussion on Hacker News (item 45840088).
Limitation The post shows no author or date; the date is that of the Hacker News discussion. Opinion, with no data.
“Past a threshold, the whole system begins getting nondeterministically stupider.”
In long tasks, each agent session starts with no memory, and compacting (summarizing) the context is not enough. What worked was keeping state outside the context. An initial agent sets up the environment, a progress file, git commits and a JSON list of features. At the start, all those features fail. Then each agent advances one feature at a time and leaves things clean. Failures seen: declaring the project done too early and marking features as done without testing them.
Numbers
over 200 featuresSize of the example project, with about 5 test steps per feature (Opus 4.5 model).
Limitation Vendor post with its own case; it gives no context sizes or token counts.
Bashir, H., Hong, K., Jiang, P., Shi, Z. Chroma, UIUC
The team that measured how long context makes models worse trains a search subagent. The subagent can delete parts of its own context and works with a fixed token budget, to avoid that decline. It then passes documents to a frontier model (one of the most advanced), which gives the answer.
Numbers
0.97 versus 0.95 and 0.99Rate at which it finds the final answer with bounded context: Context-1 (4x version) versus gpt-5.2 and opus-4.5. The base model reaches 0.58.
20 billion parametersSize of the subagent.
Limitation Chroma sells search; the figures are its own.
“we bound the context window to a fixed token budget”
Models really are improving, and that is why the length of the tasks they complete keeps growing.Waiting for the next model, however, does not fix the chain.
SlopCodeBench (exhibit 100) shows it where it hurts most. It is a standard test where the agent extends its own code as the requirements change.
None of 15 agents completed a problem end to end. The best one passed 14.8% of the checkpoints.
14,8 %
Checkpoints passed by the best of 15 agents, across 36 problems and 196 checkpoints. Exhibit 100
The agents’ code gets longer and more tangled with each round. Asking for quality in the prompt does not stop the decay.
Long-run reliability came from somewhere else. The million error-free steps came from splitting the task and checking each step, with a small model.
A broad comparison by Google and others found something similar (exhibit 31). Without a central checkpoint, errors spread more.
A study of 86 practitioners with agents in production (exhibit 29) gives the same diagnosis. Reliability is still the main challenge. Those teams tackle it through design: 68% let the agent run at most 10 steps before a person steps in.
A better model changes which checks you need, but it does not make them unnecessary. With Opus 4.6, Anthropic removed pieces of the harness it had built for Opus 4.5 (exhibit 70). A harness is the code that wraps the model and controls the path.
Each piece encoded something the earlier model could not do on its own. It is the vendor’s case, but it leaves a task: when you change models, you review the harness.
A check in code between steps cuts the chain into segments. If a segment fails, the code catches it there.
The next segment starts from a verified state. That way, the multiplication applies to each short segment, not to the whole chain.
Failures on long tasks will be fixed by the next model, which will be bigger.
Verdict: Partly. The 50% success horizon went from 293 minutes for Claude Opus 4.5 to 719 for Opus 4.6. But none of 15 agents completed a problem where it had to extend its own code. The best passed 14.8% of the checkpoints. Long-run reliability came from splitting the task and checking each step, even with a small model.
Rule
Do not wait for the next model to fix the chain. Put checks in code between steps. Every time you change models, review which parts of the harness (the code around the agent) are still needed.
Why medium confidence. The papers are primary and agree on the mechanism. But they are preprints, without peer review, with benchmarks and synthetic tasks, not real deployments. Opus 4.6's horizon has an interval of 317 to 3,634 minutes. That a better model lets you remove harness parts is a vendor's own case. The production study confirms that reliability is still the main challenge, with a small sample.
Show the 6 exhibits in detailHide the detail1 for, 2 against, 3 qualify
METR data page with time horizon 1.1. It measures the length of tasks that each model solves with 50% and with 80% success, in the time of an expert person. The trend is still exponential, and since 2023 the horizon doubles faster than in the 2025 paper. But the 80% success horizon is 4 to 10 times shorter than the 50% one. METR warns that above 16 hours its measurement is no longer reliable.
Numbers
128.7 days (interval of 104.4 to 158.0)Time for the 50% horizon to double, counting from 2023 (about 4.2 months). Since 2019 it is 187.8 days. It excludes measurements above 16 hours.
293 minutes (162 to 624) and 49 minutes (21 to 105)Horizon of Claude Opus 4.5 with 50% and with 80% success, with the data corrected on March 3, 2026.
719 minutes (317 to 3,634) and 70 minutesHorizon of Claude Opus 4.6 with 50% and with 80% success.
1,045 minutes (509 to 3,304) and 186 minutesHorizon of Claude Mythos Preview (early version, April 2026) with 50% and with 80% success: about 17.4 hours and 3.1 hours. It is above the 16-hour ceiling that METR considers reliable.
352, 342 and 384 minutes50% horizon of GPT-5.2, GPT-5.4 and Gemini 3.1 Pro.
Limitation The figures come from the page data file (benchmark_results_1_1.yaml). On March 3, 2026 METR corrected a regularization mistake, so 1.1 figures published before that date do not match. On September 8, 2026 METR said the page is no longer actively updated. The tasks are software, machine learning and cybersecurity, and 50% success is a low bar for delegating.
“Measurements above 16 hrs are unreliable with our current task suite.”
Orlanski, G., Roy, D., et al. arXiv (UW-Madison and others)
The agent has to extend its own code as the requirements change. No agent solves a full problem end to end. The agent's code gets longer and more tangled with each iteration, while human code stays stable. Asking for quality in the prompt improves the start, but does not stop the decline.
Numbers
14,8 %Checkpoints passed by the best of 15 agents, across 36 problems and 196 checkpoints.
77% and 75.5%Trajectories in which structural erosion rises and in which code verbosity rises.
2.3 and 2.0 timesHow much more verbose and more eroded the agent code is than that of 473 open source Python repositories.
Limitation The figures are from version 2 (May 7, 2026), which expands the benchmark from 20 to 36 problems and changes the results. Version 1 is from March 25, 2026. The abstract was read. It measures code quality, not per-step errors.
Meyerson, E., Paolo, G., Dailey, R., et al. arXiv (Cognizant AI Lab, UT Austin)
The other side of compound reliability: they split the task into minimal steps, each handled by a microagent. At each step they vote among several answers until one leads by 3 votes, and they discard answers that show signs of confusion. That way they complete the Towers of Hanoi with 20 disks without a single error, using a small model without reasoning. The fix is not a bigger model, but decomposing and verifying each step.
Numbers
1,048,575 stepsLength of the task solved without errors, with gpt-4.1-mini.
about 0.22%Per-step error of the model used.
about 3,500 dollarsProjected cost for 95% success; the paper does not give the actual cost of the run.
Limitation Preprint. The steps have to be defined in advance and the strategy is given to the model in each prompt: it is pure execution, not discovery. The analysis assumes independent errors.
“the process inevitably becomes derailed after at most a few hundred steps”
It compares a single agent with a three-agent harness (planner, generator and evaluator) for building full applications. The harness gave clearly better results, but cost over 20 times more. Agents that grade themselves are too generous. The separate evaluator also failed until they tuned it to be skeptical and to test with an external tool. Each harness piece assumes something the model cannot do alone, and with a better model (Opus 4.6) they removed pieces.
Numbers
20 minutes and 9 dollars versus 6 hours and 200 dollarsA retro game with Opus 4.5: single agent (the game did not respond to controls) versus full harness (playable).
3 hours 50 minutes and 124.70 dollarsA digital audio workstation with Opus 4.6 and the simplified harness.
Limitation The vendor's own cases, with no systematic control group.
“agents reliably skew positive when grading their own work”
Kim, Y., Gu, K., Park, C., et al. arXiv (Google and others)
This is a broad controlled comparison between a single agent and four multi-agent architectures. The result depends on the task structure: coordinating several agents helps when the task splits into parts and hurts when it is sequential. Architectures without central verification spread more errors.
Numbers
260 configurationsCombinations of 6 benchmarks, 5 architectures and 3 model families.
from +80.8% to -70.0%Change vs the single agent: from decomposable financial reasoning to sequential planning.
87%Held-out configurations where their model picks the best architecture.
Limitation These are benchmarks, not real deployments. The predictive model explains a modest share of the variance (R² of 0.373). Figures from earlier versions, such as error amplification of up to 17.2 times, were not verified in version 3 and are not used.
A case study and survey of practitioners with agents in production. Most evaluate with people rather than benchmarks (standard tests), and let the agent take few steps before someone steps in. Reliability, meaning that the agent behaves correctly over time, remains the main challenge. Teams tackle it with system design.
Numbers
20 cases and 86 practitionersBasis of the study: 20 in-depth interviews and a survey of practitioners across 26 domains.
74%Rely mainly on human evaluation.
68%Run at most 10 steps before a person steps in.
70%Prompt off-the-shelf models instead of tuning their weights.
Limitation A small sample chosen by the authors; it does not represent all companies. Preprint.
“Reliability (consistent correct behavior over time) remains the top development challenge”