This page explains how the evidence base was built and the rules it follows. Its figures come from the base on every build (each time the site is generated). When the base changes, the page changes with it.
The base has two kinds of files: exhibits and claims.
An exhibit is a source. It can be a paper, a survey, a regulation or a technical post. It can also be a Reddit or Hacker News thread, a press article, a pricing page, a registry entry or a court decision. Each exhibit is a YAML file in evidence/sources/, with a stable number. YAML is a plain-text data format that you can read without programming. The file states what the source found, in our own words. It also has the figures with their units, the limits and the date of the check.
A claim is a statement that people repeat or argue about regarding agents in companies. Each claim is a file in evidence/claims/. It collects the exhibits for it, the exhibits against it and the ones that qualify it. Then it gives a verdict, a design criterion and a confidence level with its reasons.
This is a real exhibit from the base, exactly as it is in the repository:
evidence/sources/maker-million-steps.yaml
id: maker-million-steps
numero: 19
tipo: paper
titulo: "Solving a Million-Step LLM Task with Zero Errors"
autores: "Meyerson, E., Paolo, G., Dailey, R., et al."
organizacion: "arXiv (Cognizant AI Lab, UT Austin)"
url: https://arxiv.org/abs/2511.09030
idioma: en
fecha: "2025-11-12"
verificado_el: "2026-11-02"
abierta: true
postura: matiza
hallazgo:
es: "La otra cara de la confiabilidad compuesta: parten la tarea en pasos mínimos, cada uno a cargo de un microagente. En cada paso votan entre varias respuestas hasta que una saca 3 votos de ventaja, y descartan las que muestran señales de confusión. Así completan las Torres de Hanoi con 20 discos sin un solo error, con un modelo chico sin razonamiento. La salida no es un modelo más grande, sino descomponer y verificar cada paso."
en: "The other side of compound reliability: they split the task into minimal steps, each handled by a microagent. At each step they vote among several answers until one leads by 3 votes, and they discard answers that show signs of confusion. That way they complete the Towers of Hanoi with 20 disks without a single error, using a small model without reasoning. The fix is not a bigger model, but decomposing and verifying each step."
numeros:
- valor:
es: "1.048.575 pasos"
en: "1,048,575 steps"
que:
es: "Largo de la tarea resuelta sin errores, con gpt-4.1-mini."
en: "Length of the task solved without errors, with gpt-4.1-mini."
- valor:
es: "cerca de 0,22 %"
en: "about 0.22%"
que:
es: "Error por paso del modelo usado."
en: "Per-step error of the model used."
- valor:
es: "unos 3.500 dólares"
en: "about 3,500 dollars"
que:
es: "Costo proyectado para 95 % de éxito; el paper no da el costo real de la corrida."
en: "Projected cost for 95% success; the paper does not give the actual cost of the run."
limitaciones:
es: "Preprint. Los pasos tienen que estar definidos de antemano y la estrategia se le da al modelo en cada prompt: es ejecución pura, no descubrimiento. El análisis supone errores independientes."
en: "Preprint. The steps have to be defined in advance and the strategy is given to the model in each prompt: it is pure execution, not discovery. The analysis assumes independent errors."
cita:
texto: "the process inevitably becomes derailed after at most a few hundred steps"
idioma: en
verificada: true
And this is a claim that uses it. Open the exhibits to see which side each one is on:
If each step is right 90% of the time, the agent is reliable.
Verdict: No: errors multiply. At 90% per step, a 5-step workflow fails about 41% of the time. At 95% per step, a 20-step one succeeds less than 36% of the time. That math assumes independent errors. The flip side: a small model split the task into minimal steps and voted on each one. That way it completed 1,048,575 steps without a single error.
Rule
Compute the reliability of the whole process: per-step success raised to the number of steps. If it falls short of what the task needs, shorten the chain or add a check between steps.
Why medium confidence. The math is arithmetic and assumes independent errors. The two Reddit threads that do it are small. The million-step paper is a preprint (a paper without peer review). Its steps were defined in advance: it is pure execution, not discovery. No source argues that 90% per step is enough.
Show the 3 exhibits in detailHide the detail0 for, 2 against, 1 qualifies
A thread about agents that work in the pilot and degrade in production. One comment does the math: with 90% success per step, a 5-step workflow fails about 41% of the time. This happens because errors multiply. Another says it is a system problem, and more code alone does not fix it.
Numbers
90% per step, 41% failure over 5 stepsA commenter's math (0.9 to the power of 5 is about 0.59).
Limitation Small thread (6 points and 30 comments in the archive). The math is arithmetic, without field data, and assumes independent errors. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“compounding errors is a systematic problem that you cannot code your way out of”
The post argues that price per token is the wrong metric, because you pay the same whether the result is right or wrong. Reliability falls as the chain of steps gets longer, so the cost of checking has to be added. The author concludes that serious work cannot be automated cost-effectively. A comment proposes something more useful: measuring cost per successful outcome.
Numbers
77%, 60% and under 36%Reliability of a 5, 10 and 20 step process with 95% success per step, if errors were independent.
Limitation A very small thread (2 points and 6 comments in the archive) and the post promotes an article by the author. The math is arithmetic, not a measurement. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“Cost per successful outcome seems like a far more useful metric than cost per token”
Meyerson, E., Paolo, G., Dailey, R., et al. arXiv (Cognizant AI Lab, UT Austin)
The other side of compound reliability: they split the task into minimal steps, each handled by a microagent. At each step they vote among several answers until one leads by 3 votes, and they discard answers that show signs of confusion. That way they complete the Towers of Hanoi with 20 disks without a single error, using a small model without reasoning. The fix is not a bigger model, but decomposing and verifying each step.
Numbers
1,048,575 stepsLength of the task solved without errors, with gpt-4.1-mini.
about 0.22%Per-step error of the model used.
about 3,500 dollarsProjected cost for 95% success; the paper does not give the actual cost of the run.
Limitation Preprint. The steps have to be defined in advance and the strategy is given to the model in each prompt: it is pure execution, not discovery. The analysis assumes independent errors.
“the process inevitably becomes derailed after at most a few hundred steps”
The base uses technology sources published on or after October 1, 2025. The search closed on November 2, 2026. Each card shows the date of its check. The search covered five fronts:
Papers from October 2025 on. Mostly from arXiv, a public archive of papers. Also official specifications and engineering posts by the people who build the systems.
Surveys and reports. Studies by consultancies, universities and public bodies on adoption, results, work and costs.
Regulation and standards. The EU AI Act, risk frameworks, security guidance and the state of regulation in Latin America.
Pricing. Vendors’ pricing pages, with the date of the check.
Field experience. Reddit and Hacker News threads from people who run agents, for and against.
Reddit blocks automated readers. reddit.com and its API returned a block (HTTP 403) to the reading tools. So the Reddit threads were read in Arctic Shift, a public archive that keeps the original text of every post and comment. Quotes were copied from that text, not from a summary. Hacker News was read on its own site and through its public API.
The full research is in docs/research/, in eight files. The first five are the original search:
Reddit against.
Reddit for, and criteria.
Papers from 2025 and 2026.
Adoption, costs and regulation.
The check of the Reddit quotes.
The other three are the November 2026 update: capability and evaluation, security, and productivity and cost. That update replaced the sources older than the cutoff. The files list every source that was examined, also the ones that did not go into the base and the ones that left because of their date. The research notes are in Spanish.
Every source and every claim follows these rules. The base validator (npm run evidence:check) checks several of them on every build and every pull request (a change request on GitHub).
Sources from October 2025 on. Every technology source is dated October 1, 2025 or later. Agents changed so much that an older source no longer describes what exists. Laws, regulations and court rulings in force are the exception, because they do not describe technology. A voluntary framework, such as a risk list, is not a regulation and must meet the cutoff. For a paper, a working paper or a journal article, the first publication counts. This includes an online version that comes before print. A revision counts only if it changes the results, and the record says what changed.
Primary source when possible. The link goes to the paper, the regulation, the report or the original thread. An article about a study does not replace the study.
The abierta rule. A source gets abierta: true if it was read in full on the page of its link. A faithful archive of that same page is also valid, such as Arctic Shift for a Reddit thread. If the source is known from another page, the link goes to the page that was read and limitaciones names the original source. If it could not be read in full, it gets abierta: false. Then the site does not present it as fact, and the card marks it “Could not be opened”.
Our own words. The finding summarizes the source in our own words, in Spanish and English. Papers are summarized, not translated.
At most one quote per source. The quote has fifteen words or fewer and is in the original language. The site shows it only if it was checked letter by letter against the original.
Figures with units and nuance. Every figure has its unit and the nuance of the source: “about”, “at least”, “on software tasks”. A verdict cannot contain a figure that is not in one of its exhibits.
Prices and laws with a date. A price, a specification version or the status of a law is valid for the day of the check. That day is verificado_el.
Both sides. Every claim has exhibits for it and exhibits against it or that qualify it. If there truly are none, a note explains why.
Each source has a general stance on agents. It can be for (it supports them), against (it shows limits, failures or risks) or qualifies (it depends on the conditions). That stance describes the source on its own.
Within each claim, the claim sets the side. The same source can support one claim and qualify another. For example, an index shows that models improve fast. That index supports a claim about capability. At the same time, it qualifies a claim about results in companies.
The verdict takes a position in two or three sentences. It says no more than its exhibits. The criterion is the design rule that comes from the verdict. It is written so that you can apply it the next day.
Confidence says how much weight the evidence behind the verdict has. Each claim explains its level under “Why” on its card.
Level
What it takes
Claims
High confidence
Several primary sources that agree.
5
Medium confidence
There are primary sources, but with clear limits (small sample, lab tasks, date, conflict of interest) or without full agreement.
49
Low confidence
The evidence is weak: one side rests only on accounts (Reddit, Hacker News, press), on few sources or on the vendor itself.
11
Reddit and Hacker News add field experience, but a claim cannot rest on them alone. If one side only has threads, the confidence goes down and the card says so.
The Reddit and Hacker News sample is not representative. It shows people who write on technical forums in English. In several subreddits the threads have few votes. Some comments come from vendors or look like ads. The archive search used words in the title, so a large thread can be missing.
Many sources come from those who sell. Model vendors write about their own systems. Consultancies sell services around what they measure. The card says so in limitaciones, and the confidence goes down when this matters.
The cutoff leaves earlier work out. Some widely cited studies on productivity, adoption or security are older than October 2025. That is why they are no longer in the base. If a conclusion depended on them, the chapter now rests it on newer sources or states it more weakly.
The cutoff date ages fast. Models, prices and laws change in months. That is why every source has its date, and the weekly check warns after 180 days.
Language. Most of the evidence is in English. Latin American regulation was read in Spanish and Portuguese.
Today, 34 of the 150 exhibits are Reddit or Hacker News threads, and 141 of the 150 are in English.
The base does not freeze. Every Monday a GitHub Actions workflow runs (evidencia.yml). A workflow is an automatic task of the repository. This one does three things:
It validates the whole base: the schema (the shape each file must have), the ids, and that every claim points to exhibits that exist.
It tests every source link. It separates broken links from links that block robots and links that do not answer in time.
It looks for sources that went 180 days without a check.
If it finds broken links or stale sources, it opens an issue (a notice in the repository). It updates the issue every week and closes it when a run comes out clean. The check warns, but it never breaks the site.
Know a source that is missing? Open the Propose a source form. It asks for the link, the type, the stance, the chapter, the finding in your own words, and whether you read it in full.
Is a claim wrong? Open the Challenge a claim form. It asks for the claim id, what is wrong and the source that supports you. The id is the file name in evidence/claims/ without .yaml. If you cannot find it, the page link is enough.
Do you prefer to send the YAML file already written? The contributing guide explains the format and how to validate it before you send it. Every source in the base is in the evidence.