At one company, a team ran an internal workshop to show off agents. One of the examples was an agent’s “skill” for setting up a work environment. A skill is a package of instructions for the agent.
When they opened it, it was a list of gh and bash commands. Put one after the other, they formed a script.
A participant told the story on a DevOps forum (exhibit 106). The thread drew 517 points and the comment, 454.
The conclusion was that a recurring process works better when it is deterministic. Deterministic means that the same input always gives the same output.
The thesis on trial says the opposite. To automate a process, the modern choice is an agent, and the scripted workflow is out of date.
First, the terms. A technical post by Fly.io (exhibit 16), with 1,070 points on Hacker News, defines an agent like this: a call to the model inside a loop with tools. The model looks at what each tool returns and decides the next step.
In a fixed workflow, by contrast, a person wrote the order of the steps in code. The workflow can call a model at some steps, for example to classify a piece of text. But the model does not choose the path.
There is a whole range between the two ends. The real decision is how much of each process you hand over to the model.
The agent works best where the path changes from case to case, as in a diagnosis. There, a model that reads, cross-checks data and proposes a cause adds something a script cannot.
A platform team running about 20 services tried agents for a year (exhibit 120). It said on a forum that it kept alert correlation.
60% vs 0%
How often alert correlation gets the probable cause right, vs the old system. Exhibit 120
According to the team, it finds the likely cause 60% of the time, against 0% for the previous system. One commenter suspects the post is an ad for a tool it mentions.
In a thread of IT managers (exhibit 117), one of them built a ticket triage. Triage is the first sorting of a case to find out what it is and where it goes.
The workflow has read-only access: it can read, but not change. It reads device management, the directory, two-factor authentication, the network and the ticket history. He says it saves 30 minutes per ticket on complex tickets.
An engineer with 11 years of experience shared another case on a forum (exhibit 79). He says he automated about 80% of his job with a coding agent. The agent picks up the requests and opens the changes.
He spends 2 to 3 hours a day reviewing and testing what the agent delivers. One commenter said that without the review, quality degraded within a week.
The three cases share a trait: each case calls for a different investigation, and a person reviews or makes the change.
The triage thread confirms it from the other side. It asked which repetitive tasks had really been automated with AI. The top answer was “none”.
In another IT managers’ thread (exhibit 144), one said that almost none of his peers deploy agents. Those who did described narrow cases: separate agents for security, diagnostics and performance, with read-only access to logs and APIs. The agent analyzes and recommends, and the person makes the change.
The best place for an agent is the repetitive work people do by hand today.
Verdict: Work that is always done the same way is better as a script. The agent pays off where each case needs a different investigation. A thread asked which IT tasks had really been automated with AI. The top answer was none. The success story was triage of complex tickets, with read-only access to several systems. Its builder says it saves 30 minutes per ticket. Other teams use agents that analyze logs and recommend, and a person makes the change.
Rule
Pick the first agent use case among tasks where each case needs its own investigation. Some examples are triage, which sorts and routes requests, or diagnosis. Do not pick it among the most repetitive tasks.
Why low confidence. Everything comes from Reddit threads, with self-reported savings and no measurement. The supporting case is one person's account after a week of use, in a job the comments describe as simple.
Show the 4 exhibits in detailHide the detail1 for, 3 against, 0 qualify
An engineer with 11 years of experience says he automated about 80% of his job with a coding agent. The agent picks up issues and opens pull requests. He says quality holds because he reviews every change. He spends 2 to 3 hours a day reviewing and testing. One commenter says that when review was removed, quality degraded within a week. Others discuss the ethics of doing it without telling the employer.
Numbers
about 80%Share of the job the author says he automated.
2 to 3 hours a dayTime he spends reviewing and testing.
1,200 points and 276 commentsSize of the thread according to the archive.
Limitation One person's story, with one week of use at the time of the post. Commenters point out it works because the work is simple. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
In an internal workshop, an agent's “skill” for creating an environment turned out to be a list of gh and bash commands. Put together, these commands are a script. The thread concludes that a recurring process works better when it is deterministic, meaning it always does the same thing. A counterpoint says the agent brings the continuous integration loop (automatic tests for each change) to the local machine. In this way, it leaves CI as the last check.
Numbers
517 points and 115 commentsSize of the thread according to the archive. The workshop comment has 454 points.
Limitation Practitioner experience, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Turns out that recurring processes work way better when they are deterministic.”
A thread asks which IT tasks were really automated with AI, and the most upvoted comment answers “none”. On the plus side, one lead built a triage flow (sorting of requests) with read-only access. The flow reads device management, the directory, two-factor authentication, the network and ticket history. The next step is to automate with a person approving from a chat. He says the goal is to grow, not to cut people.
Numbers
30 minutes per ticketSaving on complex tickets, according to the person who built the triage.
67 points and 146 commentsSize of the thread according to the archive. The quoted comment has 47 points and the “none” comment, 134.
Limitation Self-reported saving, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“On more complicated stuff this is saving EASILY 30 minutes per ticket.”
An IT lead for small businesses says almost none of their peers deploy agents, and many reply that they do not. Those who do describe narrow cases: separate agents for security, diagnostics and performance, with read-only access to logs and APIs. These agents analyze and recommend, and the person makes the change. Another uses an agent on their own network telemetry to produce operational summaries.
Numbers
133 points and 174 commentsSize of the thread according to the archive.
Limitation Practitioner stories, without measurement. The quoted comment has 26 points. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“The agents only have read-only access, so they can analyze and recommend actions”
For processes that repeat, fixed code works better than a free-running agent.
A Hacker News thread with 590 points brings another case (exhibit 97). It is a QA agent, that is, a quality-control agent.
The agent had to go through about 200 requirement files, guided only by a prompt. A prompt is the written instruction the model receives. Around file 30, the agent broke.
A deterministic harness fixed it. A harness is the code that wraps the model and controls the path. Another commenter added that, with small pieces, cheaper models perform like frontier ones (the most advanced).
That argument leaves work to the model, but puts the control flow in code. Control flow is the order of the steps, what repeats and when it ends.
Dex Horthy wrote the 12-factor agents guide. A year later, he holds the same view (exhibit 131): a big agent loop cannot handle serious work without structure and verification.
His company let a lightly supervised “software factory” run for three or four months. The code grew to 30,000 to 40,000 lines, and they ended up archiving all of it. It is one person’s anecdote, not a study.
A model vendor reaches the same place. Anthropic compared a single agent with a three-agent harness for building complete applications (exhibit 70).
A retro game with Opus 4.5: single agent (the game did not respond to controls) versus full harness (playable).Exhibit 70
Working time
Single agent20 minutes
Full harness6 hours
0
Cost, in dollars
Single agent9 dollars
Full harness200 dollars
0
For a retro game, the single agent took 20 minutes and spent 9 dollars. The game did not respond to the controls. The harness took 6 hours and spent 200 dollars, and the game was playable.
The vendor says each piece of the harness encodes something the model cannot do on its own. In another post (exhibit 77) it repeats that those assumptions expire with each new model. So you do not write the fixed part once and stop: you revisit it.
No source in the base argues with data that an agent should be the default. The team that tried agents for a year kept the ones that read and summarize: automatic changelogs (records of changes), change review and alert correlation.
It dropped AI-generated Terraform, because someone had to read every line. Terraform is a tool that describes infrastructure in code. Its rule ended up as: agents read everything, but they do not touch the infrastructure.
To automate a process, the modern choice is an agent: the programmed workflow is outdated.
Verdict: The other way around: the fixed workflow is the starting point, and the agent has to earn its place. A commenter says an agent guided only by prompts broke around file 30 of some 200. A deterministic harness fixed it: fixed code that guides each step. A company let a lightly supervised software factory run for months. It ended up archiving all the code. The agent adds value where the path changes from case to case, such as when correlating alerts.
Rule
Write the process as a fixed workflow first. Let the model decide only at the steps where the path changes from case to case, and justify each one with a test.
Why medium confidence. Several teams agree, but there is no controlled experiment. The Hacker News and Reddit cases are unmeasured accounts. The archived factory is one person's anecdote. The posts on harnesses come from a model vendor. Nobody backs the claim with data.
Show the 6 exhibits in detailHide the detail0 for, 4 against, 2 qualify
A thread about an article that asks for control flow written in code, not in prompts. One commenter says a QA agent had to go through about 200 requirement files and broke around file 30. A deterministic harness (code that fixes each step) solved it. Another says that with the work split into small pieces, cheaper models perform like frontier models. Some complain that platforms prefer prompt-only flows.
Numbers
about 200 files, breaks around 30Case of a QA agent driven only by a prompt, according to one commenter.
590 points and 296 commentsSize of the thread.
Limitation User stories, without measured data. It was read on news.ycombinator.com and through the public Algolia API. Quotes from the thread were not checked word by word, so none is used.
In an internal workshop, an agent's “skill” for creating an environment turned out to be a list of gh and bash commands. Put together, these commands are a script. The thread concludes that a recurring process works better when it is deterministic, meaning it always does the same thing. A counterpoint says the agent brings the continuous integration loop (automatic tests for each change) to the local machine. In this way, it leaves CI as the last check.
Numbers
517 points and 115 commentsSize of the thread according to the archive. The workshop comment has 454 points.
Limitation Practitioner experience, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Turns out that recurring processes work way better when they are deterministic.”
An article about Dex Horthy, author of the 12-factor agents guide. A year later he holds the same view: a big agent loop does not hold up for serious work without structure and verification. His company let a lightly supervised "software factory" run for three or four months and then archived all the code. He says the missing piece is better verifiers of code quality.
Numbers
30,000 to 40,000 linesSize the factory code reached before it was archived.
10 of 30 (33.3%)Checkpoints the best models passed on the SlopCodeBench subset he cites; the differences between vendors are within the margin of error.
Limitation It is an article about one person, with his own anecdote; it is not a study.
It compares a single agent with a three-agent harness (planner, generator and evaluator) for building full applications. The harness gave clearly better results, but cost over 20 times more. Agents that grade themselves are too generous. The separate evaluator also failed until they tuned it to be skeptical and to test with an external tool. Each harness piece assumes something the model cannot do alone, and with a better model (Opus 4.6) they removed pieces.
Numbers
20 minutes and 9 dollars versus 6 hours and 200 dollarsA retro game with Opus 4.5: single agent (the game did not respond to controls) versus full harness (playable).
3 hours 50 minutes and 124.70 dollarsA digital audio workstation with Opus 4.6 and the simplified harness.
Limitation The vendor's own cases, with no systematic control group.
“agents reliably skew positive when grading their own work”
A platform team with about 20 services reports what stayed after a year of trying agents. It dropped AI-generated Terraform, because every line had to be read. It also dropped an assistant that answered wrongly with confidence: someone almost ran a command it suggested in production. It kept what reads and summarizes: automatic changelogs, PR review and alert correlation. The rule is that agents read everything, but do not touch infrastructure.
Numbers
60% vs 0%How often alert correlation gets the probable cause right, vs the old system.
289 points and 43 commentsSize of the thread according to the archive.
Limitation One team's story, without independent measurement. A commenter suspects the post is an ad for a review tool it mentions. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“agents can read and summarize anything but they dont get to touch infra”
It separates three pieces with stable interfaces: the session, the harness and the sandbox. The session is an event log outside the model context. The harness is the loop that calls the model, and the sandbox runs the code. That way any piece can fail or be swapped without losing the session, and credentials stay outside the sandbox. It repeats that the harness encodes assumptions that expire with each new model.
Numbers
about 60% and over 90%Drop in time to first token at the median (p50) and at the 95th percentile.
Limitation Vendor product post. It does not compare a fixed workflow against an agent.
The debate has a commercial side. Menlo Ventures surveyed US decision makers (exhibit 32). Only 16% of what large companies deploy qualifies as a real agent. Among startups, 27%.
Menlo is a fund that invests in AI companies.
The thread from chapter 1 (exhibit 73) shows the other side of the label. The agent was pitched as a “virtual employee”, and the replies described it as a model in a loop.
Separate two complaints. A product that is mostly deterministic code, with model steps in the right places, is no scam.
If a big loop without structure cannot handle serious work, that shape is a virtue. The scam is charging you for the word instead of the result.
If what they sell you as an agent is basically a script with a few model steps, you got scammed.
Verdict: It depends on what they charge you for. A product that is mostly deterministic code, with model steps in the right places, is usually a virtue. A big agent loop does not hold up under serious work without structure and checks. The problem is the label. An investment fund surveyed large companies: only 16% of their deployments qualify as real agents.
Rule
Ask the vendor what the model decides and what is written in code. Pay for the result and the controls, not for the word agent.
Why low confidence. The figure on real agents comes from a survey by a fund that invests in AI companies. The point about structure comes from the experience of one person who builds agents and from Reddit threads, without measured data.
Show the 5 exhibits in detailHide the detail1 for, 2 against, 2 qualify
Tully, T., Redfern, J., Das, D., Xiao, D. Menlo Ventures
Of what companies deploy as AI, only a small part qualifies as a real agent. Companies buy more than they build (76%). Menlo says, without a figure, that bought solutions reach production sooner. About half of AI deals reach production, well above traditional software.
Numbers
16% and 27%Deployments at large companies and at startups that qualify as real agents.
76% boughtShare of AI that was bought in 2025 instead of built (53% in 2024).
47% versus 25%AI deals that reach production, versus traditional software.
37 billion dollarsEnterprise spending on generative AI in 2025 (11.5 billion in 2024).
Limitation A survey of 495 US company decision makers (November 7 to 25, 2025) plus market data. Menlo is a fund that invests in AI companies. It does not say buying works better. It says more is bought and that bought solutions reach production sooner.
“companies spent $37 billion on generative AI in 2025”
An article about Dex Horthy, author of the 12-factor agents guide. A year later he holds the same view: a big agent loop does not hold up for serious work without structure and verification. His company let a lightly supervised "software factory" run for three or four months and then archived all the code. He says the missing piece is better verifiers of code quality.
Numbers
30,000 to 40,000 linesSize the factory code reached before it was archived.
10 of 30 (33.3%)Checkpoints the best models passed on the SlopCodeBench subset he cites; the differences between vendors are within the margin of error.
Limitation It is an article about one person, with his own anecdote; it is not a study.
In an internal workshop, an agent's “skill” for creating an environment turned out to be a list of gh and bash commands. Put together, these commands are a script. The thread concludes that a recurring process works better when it is deterministic, meaning it always does the same thing. A counterpoint says the agent brings the continuous integration loop (automatic tests for each change) to the local machine. In this way, it leaves CI as the last check.
Numbers
517 points and 115 commentsSize of the thread according to the archive. The workshop comment has 454 points.
Limitation Practitioner experience, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Turns out that recurring processes work way better when they are deterministic.”
Reporting from a security conference, the author says agents are being sold there as “virtual employees”. The comments bring it down to earth. An agent is a model call in a loop with tools, and its “memory” is files or a database. Another warns that afterwards there is no way to analyze why it decided something. A third describes the sprawl of agents that are orphaned when the person who created them leaves.
Numbers
356 points and 261 commentsSize of the thread according to the archive. The quoted comment has 86 points.
Limitation Opinions from people who run IT, without measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
An agent is a model call inside a loop with tools, and writing one is simple. The hard part is managing context: past a threshold, the system gets worse without warning. Connecting the agent to reality, so it does not believe it solved something it did not, remains an open problem.
Numbers
1,070 points and 395 commentsSize of the discussion on Hacker News (item 45840088).
Limitation The post shows no author or date; the date is that of the Hacker News discussion. Opinion, with no data.
“Past a threshold, the whole system begins getting nondeterministically stupider.”
The fixed workflow is the starting point, and the agent has to earn its place.It earns it where the path changes from case to case, such as correlating alerts or sorting a complex ticket. What is always done the same way stays in code.
Teams with working agents arrive at a similar shape. One study rests on 20 cases and 86 practitioners with agents in production (exhibit 29). It found that 68% let the agent run at most 10 steps before a person steps in.
68%
Run at most 10 steps before a person steps in. Exhibit 29
It also found that 70% use commercial models with prompts, without retraining them. The sample is small and the authors chose it.
Even so, it matches the accounts. On a forum, someone who says they built more than 150 agents for businesses took stock (exhibit 57). By their account, the ones that worked were simple, single-purpose and had a person handle a small part.
The same tension shows up inside Anthropic (exhibit 28). In 2025, its coding agent went from 9.8 to 21.2 consecutive tool calls without a person stepping in.
from 9.8 to 21.2 calls
Consecutive tool calls by the coding agent between February and August 2025; human turns fell from 6.2 to 4.1. Exhibit 28
But most employees say they fully hand off only 0% to 20% of their work. It is a self-report by the vendor’s own employees.
There is a flip side. In an experiment by the same vendor (exhibit 52), sixteen agents worked in parallel. They coordinated only through a shared repository.
In almost 2,000 sessions and for just under 20,000 dollars, they wrote a C compiler of about 100,000 lines. It worked while the job could be split into independent pieces with an external checker.
The checker was a reference compiler and test suites (sets of automatic tests). Few business processes have a checker like that. It continues in chapter 4.
The agents that work in production are small, do one thing and have a person nearby.
Verdict: It is a pattern that repeats among teams with agents that work. In a study of 86 practitioners with agents in production, 68% let the agent take at most 10 steps. After that, a person steps in. Large systems pay off when the work splits into pieces with an external checker. That is how 16 agents built a C compiler for just under 20,000 dollars.
Rule
Give each agent a single job, with clear input and output, and a cap on steps before a person looks. Grow it only when the small version already works and falls short.
Why medium confidence. The production study is primary, but with a small sample chosen by the authors. The rest are Reddit threads with self-reported figures, a survey of a vendor's employees and an experiment by the same vendor.
Show the 6 exhibits in detailHide the detail4 for, 1 against, 1 qualifies
A case study and survey of practitioners with agents in production. Most evaluate with people rather than benchmarks (standard tests), and let the agent take few steps before someone steps in. Reliability, meaning that the agent behaves correctly over time, remains the main challenge. Teams tackle it with system design.
Numbers
20 cases and 86 practitionersBasis of the study: 20 in-depth interviews and a survey of practitioners across 26 domains.
74%Rely mainly on human evaluation.
68%Run at most 10 steps before a person steps in.
70%Prompt off-the-shelf models instead of tuning their weights.
Limitation A small sample chosen by the authors; it does not represent all companies. Preprint.
“Reliability (consistent correct behavior over time) remains the top development challenge”
Someone says they built more than 150 agents for businesses. They report that the ones that worked were simple, single-purpose and had a person for a small part of the task. In the comments, another team says two agents consulted each other for 60 rounds with no result. The cost kept rising until they added a layer that cuts repeated calls.
Numbers
over 150 agentsAgents the author says they built in one year (self-reported).
60 roundsLoop between two agents with no useful output, according to a comment.
Limitation Small thread (9 points and 12 comments in the archive). Self-reported figures. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“simple, single-purpose, and had a human-in-the-loop for 5% of the task”
An IT lead for small businesses says almost none of their peers deploy agents, and many reply that they do not. Those who do describe narrow cases: separate agents for security, diagnostics and performance, with read-only access to logs and APIs. These agents analyze and recommend, and the person makes the change. Another uses an agent on their own network telemetry to produce operational summaries.
Numbers
133 points and 174 commentsSize of the thread according to the archive.
Limitation Practitioner stories, without measurement. The quoted comment has 26 points. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“The agents only have read-only access, so they can analyze and recommend actions”
Huang, S., Seethor, B., Durmus, E., Handa, K., McCain, M., Stern, M., Ganguli, D. Anthropic
Inside the lab itself, people use Claude for more of their work and say they are more productive. But they can fully delegate only a small part. Agents chain more actions with less human involvement. There is concern about losing the very skills needed to supervise AI, and about how junior staff will learn.
Numbers
0% to 20%Share of work most people can fully delegate.
59% and +50%Work in which they use Claude (28% a year earlier) and self-reported productivity (+20% a year earlier), in 132 respondents.
from 9.8 to 21.2 callsConsecutive tool calls by the coding agent between February and August 2025; human turns fell from 6.2 to 4.1.
Limitation Self-report by the vendor's employees, with 2025 data (August 2025 survey, 53 interviews and 200,000 transcripts).
“I am for sure atrophying in my skills as a software engineer”
Sixteen Claude instances worked in parallel, with no orchestrator to hand out the work. They coordinated only through a shared git repository and lock files to claim tasks. It worked while the work could be split into independent pieces with an external verifier (GCC and test suites). With the Linux kernel as one big task, every agent found the same bug, fixed it and overwrote each other's changes.
Numbers
almost 2,000 sessions and just under 20,000 dollarsClaude Code sessions over two weeks and API cost (2 billion input tokens and 140 million output tokens, Opus 4.6).
about 100,000 linesSize of the C compiler in Rust, which builds Linux 6.9 on x86, ARM and RISC-V with a 99% pass rate on most suites.
Limitation A vendor experiment. The generated code is less efficient than unoptimized GCC. The task had a very good external verifier, which few business processes have.
The author advised against multi-agent systems, and ten months later he revisits his position. In his view, several agents writing in parallel still make incompatible decisions, but three patterns do work. They are a reviewer with clean context, a small model that consults a stronger one on hard tasks, and a manager agent that assigns work. In all three, a single thread does the writing.
Numbers
2 bugs per pull requestAverage the reviewer agent finds; about 58% are serious.
about 8 timesGrowth in use of its agent in the largest enterprise segment, over 6 months.
Limitation Published by a company that sells a coding agent; the figures are internal.
“clean context leads to a notable improvement in capabilities”