A chain of steps goes right less often than each step alone. If a step goes right 95% of the time, a chain of 10 steps goes right about 60% of the time.
The calculation multiplies the success of each step by itself, once per step. It assumes that the steps fail independently. In practice, one mistake often causes errors in the steps that follow.
Move the values to see your case. The link in the address bar keeps the calculation.
Calculation
Some values in the link were not valid. Those fields show the example value.
Verdict: No: errors multiply. At 90% per step, a 5-step workflow fails about 41% of the time. At 95% per step, a 20-step one succeeds less than 36% of the time. That math assumes independent errors. The flip side: a small model split the task into minimal steps and voted on each one. That way it completed 1,048,575 steps without a single error.
Rule
Compute the reliability of the whole process: per-step success raised to the number of steps. If it falls short of what the task needs, shorten the chain or add a check between steps.
Why medium confidence. The math is arithmetic and assumes independent errors. The two Reddit threads that do it are small. The million-step paper is a preprint (a paper without peer review). Its steps were defined in advance: it is pure execution, not discovery. No source argues that 90% per step is enough.
Show the 3 exhibits in detailHide the detail0 for, 2 against, 1 qualifies
A thread about agents that work in the pilot and degrade in production. One comment does the math: with 90% success per step, a 5-step workflow fails about 41% of the time. This happens because errors multiply. Another says it is a system problem, and more code alone does not fix it.
Numbers
90% per step, 41% failure over 5 stepsA commenter's math (0.9 to the power of 5 is about 0.59).
Limitation Small thread (6 points and 30 comments in the archive). The math is arithmetic, without field data, and assumes independent errors. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“compounding errors is a systematic problem that you cannot code your way out of”
The post argues that price per token is the wrong metric, because you pay the same whether the result is right or wrong. Reliability falls as the chain of steps gets longer, so the cost of checking has to be added. The author concludes that serious work cannot be automated cost-effectively. A comment proposes something more useful: measuring cost per successful outcome.
Numbers
77%, 60% and under 36%Reliability of a 5, 10 and 20 step process with 95% success per step, if errors were independent.
Limitation A very small thread (2 points and 6 comments in the archive) and the post promotes an article by the author. The math is arithmetic, not a measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Cost per successful outcome seems like a far more useful metric than cost per token”
Meyerson, E., Paolo, G., Dailey, R., et al. arXiv (Cognizant AI Lab, UT Austin)
The other side of compound reliability: they split the task into minimal steps, each handled by a microagent. At each step they vote among several answers until one leads by 3 votes, and they discard answers that show signs of confusion. That way they complete the Towers of Hanoi with 20 disks without a single error, using a small model without reasoning. The fix is not a bigger model, but decomposing and verifying each step.
Numbers
1,048,575 stepsLength of the task solved without errors, with gpt-4.1-mini.
about 0.22%Per-step error of the model used.
about 3,500 dollarsProjected cost for 95% success; the paper does not give the actual cost of the run.
Limitation Preprint. The steps have to be defined in advance and the strategy is given to the model in each prompt: it is pure execution, not discovery. The analysis assumes independent errors.
“the process inevitably becomes derailed after at most a few hundred steps”
For an agent that runs alone, look at pass^k. A task that goes right once can fail the next time. There are two ways to count k attempts at the same task:
pass@k measures whether at least one of the k attempts goes right. It fits when a person picks the best of several drafts.
pass^k measures whether all k attempts go right. It is important when the system runs alone, task after task.
With more attempts, pass@k rises and pass^k falls. An example: each attempt goes right 75% of the time. Then all 3 attempts at a task go right about 42% of the time (0.75 × 0.75 × 0.75).
The runs below are synthetic: they are made up to show the calculation. Each run is one attempt at a task, which goes right or wrong.
Sample runs
pass@3
98.4%
At least one of 3 attempts goes right.
pass^3
68.4%
All 3 attempts go right.
With 3 attempts, at least one goes right 98.4% of the time, but all 3 go right only 68.4% of the time. The gap is 30 points.
pass@k: at least one rightpass^k: all right
pass@k and pass^k for each k
k
pass@k
pass^k
1
85.4%
85.4%
2
95.4%
75.4%
3
98.4%
68.4%
4
99.6%
63.3%
5
100%
59.4%
6
100%
56.3%
To measure your agent with real tasks and set an exit criterion with pass^k, use the evaluation plan.
To know whether an agent works, it is enough to see it solve the task once.
Verdict: It is enough only when something checks every attempt, like tests in code. For anything that reaches a customer, what matters is getting it right every time. At 75% per attempt, getting 3 out of 3 drops to about 42%. On OSWorld, an agent running GPT-5 had 10 attempts per task. It solved about 78% at least once, and about 36% on all 10. In customer service for a fintech, the best success rate was 25.52% on one attempt. Across all 4 attempts it fell to 13.40%.
Rule
Measure pass^k, which means getting every repetition of the same task right. Use it for anything without an automated check before it reaches someone. Keep pass@k, which means getting it right at least once, for cases where a test filters out the bad attempts.
Why medium confidence. The step from 75% to about 42% is arithmetic and assumes independent attempts. Two 2026 benchmarks show the same drop when tasks are repeated: one on computer use and one on customer service. But they are preprints, without peer review. One comes from a team at a company that sells support agents. The guide is written by a model vendor. Nobody argues with data that one success is enough. The qualifying sources ask for tests that prove the change. One practitioner says the agent wrote tests that tested nothing.
Show the 5 exhibits in detailHide the detail0 for, 3 against, 2 qualify
This guide to evaluating agents recommends starting early with tasks taken from real failures. It also asks to combine three kinds of graders (code, model and human), grade the outcome rather than the path, and read the transcripts. It separates pass@k, succeeding at least once in k tries, from pass^k, succeeding in all of them. It warns that some failures are grader errors.
Numbers
20 to 50 tasksSimple tasks taken from real failures, as a good starting point.
75% per try, about 42% for 3 out of 3A pass^k example: succeeding all three times is much less likely than succeeding once.
from 42% to 95%Opus 4.5 result on CORE-Bench after fixing grader bugs.
Limitation Written by a model vendor. It is a practice guide, not a study.
“20-50 simple tasks drawn from real failures is a great start”
Gonzalez-Pumariega, G., Agashe, S., Yang, J., Li, A., Wang, X. E. arXiv
The study runs the same OSWorld tasks many times and asks why an agent that succeeds once does not repeat it. It finds three causes: randomness in execution, ambiguous tasks and variation in the agent's behavior. Cosmetic changes to the environment lowered the reliability of GPT-5 and Claude. Writing the evaluation criterion into the instruction improved reliability in all models.
Numbers
about 78% versus about 36%Agent S3 with GPT-5: tasks solved at least once in 10 attempts (Pass@10) versus tasks solved in all 10 attempts (Pass^10). Read from figure 1.
Limitation Preprint. The figures are approximate, read from a figure. Computer use, not customer service.
Shi, Q., Zytek, A., Razavi, P., Narasimhan, K., Barres, V. arXiv (Sierra, Princeton)
The τ-bench team adds a new domain: customer service at a fintech, with about 700 linked knowledge documents. Even the best models, with heavy reasoning, solve about a quarter of the tasks. Also, reliability drops when they repeat the same task. It is part of τ³-bench.
Numbers
25,52 %Best single-attempt success rate (pass^1) with realistic search: GPT-5.2 with high reasoning, on 97 tasks.
13,40 %Best rate of success in all 4 attempts (pass^4).
39.69% and 26.80%Claude 4.5 Opus with the correct documents already given: pass^1 and pass^4.
Limitation Preprint by a team at a company that sells customer service agents. Benchmark with a simulated user.
Whoever submits code has to show it works: test it by hand and leave automated tests that fail if the change is reverted. Coding agents are held to the same standard, and the person remains responsible for what is delivered.
Numbers
859 points and 658 commentsSize of the discussion on Hacker News (item 46313297).
Limitation An opinion essay by a specialist, with no data of its own.
“Your job is to deliver code you have proven to work.”
The author sees a push to review little or no code ("if the tests pass, ship it"). On his existing code, half his time went to fixing subtle errors. The success stories agree on the conditions: code already organized by people, static types, broad tests, narrow tasks and reviewing the plan first. One says the agent wrote tests that tested nothing.
Numbers
461 points and 455 commentsSize of the thread.
Limitation Practitioner opinions. It was read through the Algolia API (post and top-level replies with the most replies).