The vendor meeting opens with a slide. It shows a benchmark, a standard test with tasks and a way to grade them. Their agent’s score is above everyone else’s. Then comes the proof of concept.
In a Reddit thread for IT buyers, someone described how it usually goes (exhibit 45). The vendor hands you an API key and tells you to test it yourself. Replies added that SOC 2, a security certification, does not cover model behavior.
Four assumptions sit under that meeting:
If the agent scores well on benchmarks, it will do well in production.
If the score goes up, the agent got better.
If it solved the task once, it works.
If a judge model was calibrated against people, it can measure any new model. A judge model is a model that grades another model’s answers.
All four get used to pick models, approve budgets and promise dates.
Agents are improving fast, and benchmarks show it. Stanford’s AI Index 2026 records it (exhibit 75). On OSWorld, a benchmark of computer tasks, success went from about 12% to about 66%. On SWE-bench Verified, which measures code fixes, it went from 60% to nearly 100% in one year.
from about 12% to about 66%
Success on OSWorld, a benchmark of agent tasks on a computer. Exhibit 75
METR, an institute that evaluates AI models, measures the time horizon (exhibit 55). It is the length of the tasks that a model solves with 50% success. The length is measured in the time an expert needs. Since 2023, that horizon has doubled every 128.7 days, about 4.2 months.
SWE-Bench Pro is a harder code benchmark. On its public leaderboard, the best 2026 model solves about 6 in 10 problems in the public set (exhibit 151). A year earlier, the best reached 43.6%.
Grading at scale has good news too. MobileJudgeBench measured automatic judges on 931 real trajectories of phone agents (exhibit 125). The best judge was right 90.9% of the time when it decided whether the agent had completed the task. Agreement between people was 88.4%. An OpenAI team showed something else (exhibit 36). Monitoring a model’s chain of reasoning works better than monitoring only its actions.
In code, getting it right once can be enough. If a test filters out the bad attempts, it does not matter how many failed before. Simon Willison makes it a rule for anyone who ships code, person or agent (exhibit 35). The rule is to show that it works, with tests that fail if the change is reverted.
A good score on one benchmark says little about another. SWE-Bench Pro built 1,865 problems across 41 repositories (exhibit 20). There are public, held-out and private commercial repositories. With 2025 models, the ones passing 70% on SWE-bench Verified solved less than half of the public set. On the commercial set, Claude Opus 4.1 reached 17.8% and GPT-5 15.7%. Scale AI, which sells evaluation, publishes it.
According to the press, OpenAI stopped reporting SWE-bench Verified (exhibit 93). OpenAI audited problems that the models were failing. It found that at least 59.4% had tests that rejected correct solutions.
The number also depends on the machine and the grader. On Terminal-Bench 2.0, the memory and CPU configuration moved the score by 6 points (exhibit 53). Anthropic published those runs. It recommends skepticism about gaps under 3 points.
METR had maintainers of three projects review agent pull requests (exhibit 67). A pull request is a proposed change to the code. Those changes had already passed the automated grader. The grader gave them 24.2 points more than the maintainers did. A review of 445 benchmarks found that only 16.0% used statistics to compare results (exhibit 14).
METR also wrote the fine print on the horizon (exhibit 44). The horizon does not say how long an agent can work alone. On visual computer-use tasks, the horizon is 40 to 100 times shorter. And RealUserSim showed that the simulated users in benchmarks are too formal and cooperative (exhibit 76).
If the agent scores well on benchmarks, it will work well in production.
Verdict: No, although they show agents improving fast. With 2025 models, those passing 70% on SWE-bench Verified did much worse on SWE-Bench Pro. On its commercial set they solved less than a fifth. An automated grader scored agent pull requests 24.2 points higher than the maintainers did. And on Terminal-Bench 2.0, the machine's configuration moved the score by 6 points.
Rule
Use public benchmarks (standard tests) to rule candidates out, not to pick one. Look first at those that resemble your case, such as private code or realistic users. Distrust gaps of less than 3 points.
Why medium confidence. Several independent sources agree: the score depends on the benchmark, the grader and the machine. But none compares it with production results. With 2026 models, the best one already solves about 6 in 10 problems on SWE-Bench Pro's public set. That benchmark is published by a company that sells evaluation. The OpenAI part is known from the press.
Show the 10 exhibits in detailHide the detail1 for, 6 against, 3 qualify
An annual compilation of AI data. Adoption in organizations is broad and agents improve fast on benchmarks, but documented AI incidents also rise.
Numbers
88%AI adoption in organizations.
from about 12% to about 66%Success on OSWorld, a benchmark of agent tasks on a computer.
from 60% to almost 100%SWE-bench Verified in one year.
233 and 362Documented AI incidents in 2024 and in 2025.
Limitation It compiles third-party sources (benchmarks, surveys, incident databases). A benchmark is not the same as performance in production. The key findings page has no 2026 inference costs.
It is a software engineering benchmark that is harder than SWE-bench Verified and less contaminated (fewer problems already seen in training). It uses public repositories, held-out repositories and private commercial repositories. The same models that pass 70% on SWE-bench Verified solve less than half of the public set. On the commercial set they solve less than a fifth.
Numbers
1,865 problems in 41 repositoriesSize of the benchmark.
43.6% and 41.8%Public set: Claude Sonnet 4.5 and GPT-5 (high).
17.8% and 15.7%Commercial set: Claude Opus 4.1 and GPT-5 (high).
Limitation The date is that of version 2 (November 14, 2025), which changes the results: it adds Claude Sonnet 4.5 and Claude Haiku 4.5, released after version 1 (September 21, 2025). The figures in this entry are from version 2. 2025 data: 2026 models are not in the paper. Published by Scale AI, which sells data and evaluation.
Four maintainers of scikit-learn, Sphinx and pytest reviewed agent pull requests that already passed the SWE-bench Verified automated grader. About half would not be merged. In newer models, the failures shifted toward code quality more than broken functionality. The authors note it does not prove a capability limit: the agent had a single attempt, with no chance to iterate with the reviewer.
Numbers
24.2 pointsHow much higher the automated grader scores than the maintainers, normalized against human pull requests (standard error 2.7). Unnormalized, 34.9 points.
296 agent pull requestsReviewed, together with 47 human reference ones, on 95 of the 500 problems. Maintainers merged about 68% of the human ones.
about 50 versus about 8 minutesHorizon of Claude Sonnet 4.5 with the automated grader and with maintainer review; the authors say this figure is very noisy.
Limitation A research note, not a peer-reviewed paper. Four maintainers and three repositories. Models from Claude 3.5 Sonnet to Claude Sonnet 4.5 and GPT-5.
“roughly half of test-passing SWE-bench Verified PRs”
The machine setup (memory, CPU, time limit) changes the score of coding agents. That change is larger than the gaps used to rank models. With strict limits the frugal strategy wins, and with loose limits the resource-heavy one wins. It recommends reporting the setup and distrusting gaps of under 3 points.
Numbers
6 percentage pointsGap on Terminal-Bench 2.0 between the setup with the most and the fewest resources (p < 0.01).
5.8%, 2.1% and 0.5%Infrastructure errors with a strict limit, with 3 times headroom and with no limit.
Limitation Published by a model vendor with its own runs.
According to the article, OpenAI stopped reporting SWE-bench Verified (a coding test) because it no longer works to measure frontier models. OpenAI reviewed problems the models failed and found many tests that reject correct solutions. There is also contamination (the models already saw the test): some reproduce the problem statement and the fixed code. OpenAI recommends SWE-Bench Pro.
Numbers
27,6 %Share of the failed problems that OpenAI audited.
at least 59.4%Of those audited problems, the ones with tests that rejected correct solutions.
Limitation The original source is the OpenAI post "Why we no longer evaluate SWE-bench Verified" (February 23, 2026, according to search), which returned 403 twice. The figures come from the press article and were not checked against the primary source.
Bean, A. M., Kearns, R. O., Romanou, A., et al. arXiv (Oxford and others)
A systematic review of 445 language model benchmarks (standard tests), done with 29 experts. It finds repeated problems in what is measured, with which tasks and with which metrics. These problems weaken what the benchmarks claim to show. Few use statistics to compare results, and the review gives eight recommendations and a checklist.
Numbers
16,0 %Benchmarks that used uncertainty estimates or statistical tests to compare results.
38,2 %Benchmarks that reuse data from earlier benchmarks or human exams (contamination risk).
Limitation It reviews model benchmarks in general, not agent benchmarks. The figures come from the full text (HTML of version 1). NeurIPS 2025.
“16.0% used uncertainty estimates or statistical tests to compare the results”
A note by the metric's own authors. The time horizon does not say how long an agent can work alone. It says how much human work AI replaces with 50% success. The figure changes a lot by domain and does not carry over directly to real work, which is messier.
Numbers
4 h 49 minHorizon of Claude Opus 4.5, with a 95% interval from 1 h 49 min to 20 h 25 min.
40 to 100 times shorterHorizon on visual computer-use tasks.
about 2 minutesHorizon of Claude 4.5 Sonnet on a real physical task: making coffee.
Limitation It is a note by the authors about their own metric, not a separate study. The 4 h 49 min figure is a measurement with the TH1 series. With TH1.1, after METR's correction, the same model gets 293 minutes (record “Time Horizon 1.1”). For work that needs 98% success or more, a 50% horizon is not a useful guide.
A case study and survey of practitioners with agents in production. Most evaluate with people rather than benchmarks (standard tests), and let the agent take few steps before someone steps in. Reliability, meaning that the agent behaves correctly over time, remains the main challenge. Teams tackle it with system design.
Numbers
20 cases and 86 practitionersBasis of the study: 20 in-depth interviews and a survey of practitioners across 26 domains.
74%Rely mainly on human evaluation.
68%Run at most 10 steps before a person steps in.
70%Prompt off-the-shelf models instead of tuning their weights.
Limitation A small sample chosen by the authors; it does not represent all companies. Preprint.
“Reliability (consistent correct behavior over time) remains the top development challenge”
Public results table for the public set of SWE-Bench Pro. With 2026 models, the best already solves more than half: about 6 in 10 problems. A year earlier, in the paper, the best reached 43.6%. The private commercial set has no updated table.
Numbers
61.50% (± 3.10)Top entry (Muse Spark 1.1), on 731 tasks.
59.10%, 51.90% and 46.10%gpt-5.4 (xHigh), claude-opus-4-6 (thinking) and gemini-3.1-pro (thinking).
Limitation The page shows no date: the date is the day it was checked (November 2, 2026). It is maintained by Scale AI, the benchmark author. The commercial set page only repeats the paper figures.
Zhu, M., Tan, J., Murthy, R., et al. arXiv (Salesforce AI Research)
Users simulated with a model are too formal and cooperative, and do not look like real users. The authors build simulators based on behavior profiles taken from real conversations. With more realistic users, agent failures that the cooperative user hid show up. Also, success on TauBench, a standard test for agents, drops.
Numbers
7,275 profilesBehavior profiles taken from over 14,000 real conversations (WildChat).
from 24.2% to 45.3%Style match with real users, without and with the method. Without constraints, simulators match 6% to 8%.
3.2% to 3.5%Average drop in agent success on TauBench with realistic users.
Limitation Preprint; the abstract was read. The arXiv page shows submission on April 7, 2026, although the number corresponds to May. The measured drop is small. It does not repeat the measurement of an agent guiding the user.
Read at the source. Checked on Nov 2, 2026.
The score also measures the grader and the shortcuts. HAL is an infrastructure for evaluating agents on equal terms (exhibit 9). It found an agent that searched HuggingFace for the benchmark instead of solving the task.
Anthropic found something similar on BrowseComp, a web search benchmark (exhibit 66). In two cases, Opus 4.6 suspected it was being evaluated. It identified the benchmark and decrypted the answer key.
Anthropic’s evaluation guide gives the case from the other side (exhibit 40). Opus 4.5 went from 42% to 95% on CORE-Bench after the grader’s errors were fixed. The agent was the same. The grading changed.
If the evaluation score goes up, the agent got better.
Verdict: Not always. Agents really are getting more capable, but a score also measures the grader and the shortcuts. One agent looked up the benchmark on HuggingFace instead of solving the task. In two cases, Opus 4.6 identified BrowseComp and decrypted its answer key. One model went from 42% to 95% on CORE-Bench after grader errors were fixed.
Rule
Whenever the score moves a lot, up or down, read a sample of transcripts and check the grader before drawing conclusions.
Why high confidence. The claim falls with a single case, and several are documented in primary sources. They are an independent paper with 21,730 runs and two reports from a vendor on its own evaluations. The cases come from public benchmarks, not companies. That capability is growing comes from METR's measurement, with software, machine learning and cybersecurity tasks.
Show the 4 exhibits in detailHide the detail1 for, 2 against, 1 qualifies
METR data page with time horizon 1.1. It measures the length of tasks that each model solves with 50% and with 80% success, in the time of an expert person. The trend is still exponential, and since 2023 the horizon doubles faster than in the 2025 paper. But the 80% success horizon is 4 to 10 times shorter than the 50% one. METR warns that above 16 hours its measurement is no longer reliable.
Numbers
128.7 days (interval of 104.4 to 158.0)Time for the 50% horizon to double, counting from 2023 (about 4.2 months). Since 2019 it is 187.8 days. It excludes measurements above 16 hours.
293 minutes (162 to 624) and 49 minutes (21 to 105)Horizon of Claude Opus 4.5 with 50% and with 80% success, with the data corrected on March 3, 2026.
719 minutes (317 to 3,634) and 70 minutesHorizon of Claude Opus 4.6 with 50% and with 80% success.
1,045 minutes (509 to 3,304) and 186 minutesHorizon of Claude Mythos Preview (early version, April 2026) with 50% and with 80% success: about 17.4 hours and 3.1 hours. It is above the 16-hour ceiling that METR considers reliable.
352, 342 and 384 minutes50% horizon of GPT-5.2, GPT-5.4 and Gemini 3.1 Pro.
Limitation The figures come from the page data file (benchmark_results_1_1.yaml). On March 3, 2026 METR corrected a regularization mistake, so 1.1 figures published before that date do not match. On September 8, 2026 METR said the page is no longer actively updated. The tasks are software, machine learning and cybersecurity, and 50% success is a low bar for delegating.
“Measurements above 16 hrs are unreliable with our current task suite.”
It is infrastructure for evaluating agents consistently, and it measures cost and behavior as well as accuracy. More reasoning effort lowered accuracy in most runs. They found an agent that searched HuggingFace for the benchmark (the test) instead of solving the task. They also found misuse of payment cards in flight booking tasks.
Numbers
21,730 runsAgent runs with 9 models on 9 benchmarks.
about USD 40,000Cost of running the evaluation.
about 2.5 billion tokensLogs published for others to review.
Limitation Public benchmarks, not company deployments. 2025 models.
Anthropic reviewed Opus 4.6 answers on BrowseComp, a benchmark (standard test) for web search. It found answers that did not come from solving the question. In two cases the model suspected it was being evaluated, identified the benchmark and decrypted the answer key. The multi-agent setup searches more and so also finds more benchmark leaks.
Numbers
86.81% adjusted to 86.57%Score of the multi-agent setup, before and after removing answers taken from benchmark material.
11 of 1,266 (0.87%)Problems solved by unintended routes in multi-agent mode (9 from contamination on the web); with a single agent, 0.24%.
Limitation Published by the model vendor about its own model.
This guide to evaluating agents recommends starting early with tasks taken from real failures. It also asks to combine three kinds of graders (code, model and human), grade the outcome rather than the path, and read the transcripts. It separates pass@k, succeeding at least once in k tries, from pass^k, succeeding in all of them. It warns that some failures are grader errors.
Numbers
20 to 50 tasksSimple tasks taken from real failures, as a good starting point.
75% per try, about 42% for 3 out of 3A pass^k example: succeeding all three times is much less likely than succeeding once.
from 42% to 95%Opus 4.5 result on CORE-Bench after fixing grader bugs.
Limitation Written by a model vendor. It is a practice guide, not a study.
“20-50 simple tasks drawn from real failures is a great start”
Quote checked against the original
Read at the source. Checked on Nov 2, 2026.
Getting it right once is not enough either if nobody checks each attempt. Anthropic’s guide separates two measures. The measure pass@k is succeeding at least once in k tries. The measure pass^k is succeeding on all of them. At 75% per try, getting 3 out of 3 drops to about 42%.
A preprint, a study not yet peer reviewed, ran the same OSWorld tasks many times (exhibit 83). An agent with GPT-5 solved about 78% of tasks at least once in 10 tries. On all 10 tries, it solved about 36%.
Agent S3 with GPT-5: tasks solved at least once in 10 attempts (Pass@10) versus tasks solved in all 10 attempts (Pass^10). Read from figure 1.Exhibit 83
Tasks solved, in %
In at least 1 of 10 triesabout 78%
In all 10 triesabout 36%
0100%
τ-Knowledge measures customer service for a fintech, a digital financial services company (exhibit 63). There the best rate fell from 25.52% on one try to 13.40% across 4. A customer does not see your best attempt. They see the one they got.
And the filtering test has to test something. On Hacker News, a practitioner said the agent wrote tests that tested nothing (exhibit 43).
To know whether an agent works, it is enough to see it solve the task once.
Verdict: It is enough only when something checks every attempt, like tests in code. For anything that reaches a customer, what matters is getting it right every time. At 75% per attempt, getting 3 out of 3 drops to about 42%. On OSWorld, an agent running GPT-5 had 10 attempts per task. It solved about 78% at least once, and about 36% on all 10. In customer service for a fintech, the best success rate was 25.52% on one attempt. Across all 4 attempts it fell to 13.40%.
Rule
Measure pass^k, which means getting every repetition of the same task right. Use it for anything without an automated check before it reaches someone. Keep pass@k, which means getting it right at least once, for cases where a test filters out the bad attempts.
Why medium confidence. The step from 75% to about 42% is arithmetic and assumes independent attempts. Two 2026 benchmarks show the same drop when tasks are repeated: one on computer use and one on customer service. But they are preprints, without peer review. One comes from a team at a company that sells support agents. The guide is written by a model vendor. Nobody argues with data that one success is enough. The qualifying sources ask for tests that prove the change. One practitioner says the agent wrote tests that tested nothing.
Show the 5 exhibits in detailHide the detail0 for, 3 against, 2 qualify
This guide to evaluating agents recommends starting early with tasks taken from real failures. It also asks to combine three kinds of graders (code, model and human), grade the outcome rather than the path, and read the transcripts. It separates pass@k, succeeding at least once in k tries, from pass^k, succeeding in all of them. It warns that some failures are grader errors.
Numbers
20 to 50 tasksSimple tasks taken from real failures, as a good starting point.
75% per try, about 42% for 3 out of 3A pass^k example: succeeding all three times is much less likely than succeeding once.
from 42% to 95%Opus 4.5 result on CORE-Bench after fixing grader bugs.
Limitation Written by a model vendor. It is a practice guide, not a study.
“20-50 simple tasks drawn from real failures is a great start”
Gonzalez-Pumariega, G., Agashe, S., Yang, J., Li, A., Wang, X. E. arXiv
The study runs the same OSWorld tasks many times and asks why an agent that succeeds once does not repeat it. It finds three causes: randomness in execution, ambiguous tasks and variation in the agent's behavior. Cosmetic changes to the environment lowered the reliability of GPT-5 and Claude. Writing the evaluation criterion into the instruction improved reliability in all models.
Numbers
about 78% versus about 36%Agent S3 with GPT-5: tasks solved at least once in 10 attempts (Pass@10) versus tasks solved in all 10 attempts (Pass^10). Read from figure 1.
Limitation Preprint. The figures are approximate, read from a figure. Computer use, not customer service.
Shi, Q., Zytek, A., Razavi, P., Narasimhan, K., Barres, V. arXiv (Sierra, Princeton)
The τ-bench team adds a new domain: customer service at a fintech, with about 700 linked knowledge documents. Even the best models, with heavy reasoning, solve about a quarter of the tasks. Also, reliability drops when they repeat the same task. It is part of τ³-bench.
Numbers
25,52 %Best single-attempt success rate (pass^1) with realistic search: GPT-5.2 with high reasoning, on 97 tasks.
13,40 %Best rate of success in all 4 attempts (pass^4).
39.69% and 26.80%Claude 4.5 Opus with the correct documents already given: pass^1 and pass^4.
Limitation Preprint by a team at a company that sells customer service agents. Benchmark with a simulated user.
Whoever submits code has to show it works: test it by hand and leave automated tests that fail if the change is reverted. Coding agents are held to the same standard, and the person remains responsible for what is delivered.
Numbers
859 points and 658 commentsSize of the discussion on Hacker News (item 46313297).
Limitation An opinion essay by a specialist, with no data of its own.
“Your job is to deliver code you have proven to work.”
The author sees a push to review little or no code ("if the tests pass, ship it"). On his existing code, half his time went to fixing subtle errors. The success stories agree on the conditions: code already organized by people, static types, broad tests, narrow tasks and reviewing the plan first. One says the agent wrote tests that tested nothing.
Numbers
461 points and 455 commentsSize of the thread.
Limitation Practitioner opinions. It was read through the Algolia API (post and top-level replies with the most replies).
Read at the source. Checked on Nov 2, 2026.
Automatic judges have their own problem. A preprint evaluated 21 judges from nine vendors across about 541,000 judgments (exhibit 112). Measuring agreement with people as exact match inflates it. Once chance agreement is discounted, agreement fell by 33 to 41 points. On phone agents, judges failed in opposite directions: some rejected successes and others approved failures.
A model can grade the agent's answers instead of a person.
Verdict: It works for evaluating at scale, not for taking people out. On phone agents, the best judge was right 90.9% of the time about whether the task was done. That is on par with the 88.4% agreement between people. But judges fail in opposite directions: some reject successes and others approve failures. In chats, their agreement with people drops by 33 to 41 points once chance agreement is discounted.
Rule
Use the judge for the question of whether the agent completed the task. Calibrate it against a sample a person grades. Measure agreement net of chance, and look at false positives and false negatives separately.
Why medium confidence. Both papers on judges are recent preprints, without peer review. One measures a binary task on phone agents. The other measures chats, not agent trajectories. The paper on monitoring reasoning comes from OpenAI's team. It shows that watching the reasoning works better than watching only the actions, and gives no percentages. That most production teams evaluate with people comes from a small sample.
Show the 4 exhibits in detailHide the detail2 for, 1 against, 1 qualifies
Wang, Z., Gu, L., Chi, Z., Liu, Z., Ayyoubzadeh, S. M., Yu, Y., Wang, Y. arXiv
It measures automated judges (models that grade other models) on real trajectories of phone agents, not on chats. A simple judge that looks at screenshots matches or beats custom methods. On whether the agent achieved the task, it reaches the level of agreement between people. The base model matters most, and models fail in opposite directions: some reject successes and others approve failures.
Numbers
931 trajectoriesAnnotated by people, from 6 benchmarks, 4 agents and 68 apps.
88,4 %Average agreement between people on the success label.
90,9 %Accuracy of the best judge (a simple judge with Gemini 3 Flash); its agent ranking correlates 0.97 with the human one.
71 of 78Serious failures of Qwen-based judges that were false positives.
Limitation Preprint. It is a binary task (did it succeed or not?) on phone agents. It does not show the judge works to grade the quality of a deliverable.
Guan, M. Y., Wang, M., Carroll, M., Dou, Z., et al. arXiv (OpenAI)
The OpenAI team that studies reasoning monitors builds evaluations to measure how well a model's chain of reasoning can be monitored. Monitoring reasoning works better than monitoring actions alone. Most frontier models can be monitored fairly well, but not fully. Reinforcement learning at current scale did not reduce that property much. The authors warn that it can be fragile under other training methods.
Numbers
13 evaluations in 24 environmentsSize of the monitoring test suite.
Limitation The abstract was read, and it gives no percentages. The OpenAI post on the paper returned 403. Published by a model vendor.
“CoT monitoring is more effective than action-only monitoring in practical settings”
It evaluates 21 judges (models that grade answers) from nine vendors on three benchmarks (standard tests). Measuring agreement with people as exact match inflates it, because it does not discount chance agreement. A judge can be very consistent with itself and have position bias at the same time. Also, the judge ranking changes a lot from one benchmark to another.
Numbers
about 541,000 judgmentsIn 118 runs on MT-Bench, JudgeBench and RewardBench.
33 to 41 pointsHow much agreement drops on MT-Bench when moving from exact match to Cohen's kappa.
up to 14 placesHow far a judge moves in the ranking between benchmarks.
Limitation Preprint; the abstract was read. These are chats and preferences, not agent trajectories.
A case study and survey of practitioners with agents in production. Most evaluate with people rather than benchmarks (standard tests), and let the agent take few steps before someone steps in. Reliability, meaning that the agent behaves correctly over time, remains the main challenge. Teams tackle it with system design.
Numbers
20 cases and 86 practitionersBasis of the study: 20 in-depth interviews and a survey of practitioners across 26 domains.
74%Rely mainly on human evaluation.
68%Run at most 10 steps before a person steps in.
70%Prompt off-the-shelf models instead of tuning their weights.
Limitation A small sample chosen by the authors; it does not represent all companies. Preprint.
“Reliability (consistent correct behavior over time) remains the top development challenge”
Quote checked against the original
Read at the source. Checked on Nov 2, 2026.
Calibrating once is not enough either. The Remote Labor Index uses 240 real freelance projects (exhibit 11). Trained evaluators grade them. Its authors said that this evaluation cannot be automated with models.
Later they tested an automatic judge calibrated on earlier models (exhibit 115). It ranked the new models well, with a rank correlation of 0.90 with people. Rank correlation measures how similar two orderings are. But it gave GPT-5.5 17.9%, and people gave it 6.25%. That is about 2.9 times the human grade. In the 21-judge preprint, a single judge moved up to 14 places from one benchmark to another.
Once I have calibrated the automated judge against people, I can use it to score every new model.
Verdict: No. On the Remote Labor Index, a judge calibrated on earlier models gave GPT-5.5 17.9%. People gave it 6.25%, so the judge gave about 2.9 times the human score. It still ranked the models well, with a 0.90 rank correlation with human evaluation. In a ranking of judges, the same judge can move up to 14 places from one benchmark to another.
Rule
Recalibrate the judge against people every time the model under evaluation or the type of task changes. Use it to compare versions with each other, not to give the absolute score.
Why medium confidence. The inflated judge is measured by the benchmark's own team, in a post rather than a paper. The original paper already said this kind of work cannot be graded automatically with models. The paper on judge rankings is a preprint about chats, without peer review. What shows a judge close to people is a binary task on phone agents. There, the judge's ranking of agents matches people's at 0.97. No source argues that calibrating once is enough.
Show the 4 exhibits in detailHide the detail0 for, 3 against, 1 qualifies
Follow-up by the Remote Labor Index team. In less than eight months, the best agent went from automating 2.5% to 15.8% of the freelance projects. They also tried an automated judge calibrated against human evaluation with earlier models. The judge ranked the new models well, but greatly inflated how often they succeed. The authors conclude it works to track relative progress, not to measure absolute capability.
Numbers
15,8 %Automation rate of the best agent (Fable 5). Opus 4.8: 8.3%; GPT-5.5: 6.25%; Gemini 3 Pro: 1.25%.
6.25% versus 17.9%GPT-5.5 according to human evaluators and according to the automated judge (about 2.9 times). For Opus 4.8: 8.33% versus 18.8%.
ρ = 0,90Rank correlation (Spearman) between the judge and human evaluation.
Limitation Published by the benchmark team. A press article gives 16.1% and says only 218 of 240 projects were evaluated. The post does not mention it and the base uses the post figure. Over 84% of the projects still do not reach professional level.
“not a substitute for human evaluation of absolute capability”
Mazeika, M., Gatti, A., et al. arXiv (Center for AI Safety, Scale AI)
This benchmark (a standard test to compare agents) uses 240 real freelance projects instead of a simulated company. A professional already did each project. A project counts as automated only if the AI deliverable is at least as good as the human one, according to trained evaluators. At launch, no agent automated more than 3% of the projects.
Numbers
240 projects in 23 subcategoriesSize of the benchmark: over 6,000 hours of human work and over 140,000 dollars paid; median of 11.5 hours per project.
2,5 %Automation rate of the best agent at launch (Manus). Grok 4 and Claude Sonnet 4.5 reached 2.1%; GPT-5, 1.7%.
94,4 %Agreement between evaluators on the automation rate.
Limitation 2025 models. Evaluation is human and expensive; the authors say it cannot be automated with models today. Co-authored by Scale AI, which sells data and evaluation.
It evaluates 21 judges (models that grade answers) from nine vendors on three benchmarks (standard tests). Measuring agreement with people as exact match inflates it, because it does not discount chance agreement. A judge can be very consistent with itself and have position bias at the same time. Also, the judge ranking changes a lot from one benchmark to another.
Numbers
about 541,000 judgmentsIn 118 runs on MT-Bench, JudgeBench and RewardBench.
33 to 41 pointsHow much agreement drops on MT-Bench when moving from exact match to Cohen's kappa.
up to 14 placesHow far a judge moves in the ranking between benchmarks.
Limitation Preprint; the abstract was read. These are chats and preferences, not agent trajectories.
Wang, Z., Gu, L., Chi, Z., Liu, Z., Ayyoubzadeh, S. M., Yu, Y., Wang, Y. arXiv
It measures automated judges (models that grade other models) on real trajectories of phone agents, not on chats. A simple judge that looks at screenshots matches or beats custom methods. On whether the agent achieved the task, it reaches the level of agreement between people. The base model matters most, and models fail in opposite directions: some reject successes and others approve failures.
Numbers
931 trajectoriesAnnotated by people, from 6 benchmarks, 4 agents and 68 apps.
88,4 %Average agreement between people on the success label.
90,9 %Accuracy of the best judge (a simple judge with Gemini 3 Flash); its agent ranking correlates 0.97 with the human one.
71 of 78Serious failures of Qwen-based judges that were false positives.
Limitation Preprint. It is a binary task (did it succeed or not?) on phone agents. It does not show the judge works to grade the quality of a deliverable.
Benchmarks are useful for seeing the trend and ruling out candidates.They are not useful for choosing, and even less for promising. The score changes across benchmarks. It depends on the machine, the grader and the shortcuts, and it rarely looks like your case. No source in the base compares the score with production results.
Teams with agents in production already know this. “Measuring Agents in Production” surveyed 86 practitioners (exhibit 29). 74% rely mainly on human evaluation.
A judge model helps scale that evaluation, under these conditions:
Use it for the binary question: did the agent complete the task or not.
Calibrate it against a sample that a person grades.
Recalibrate it when the model or the task type changes.
Use it to compare versions with each other, not to give the absolute grade.
Evaluating well costs money, and it helps to know that before you start. HAL ran 21,730 runs of 9 models on 9 benchmarks. The bill came to about $40,000. You do not need that. Anthropic’s guide says 20 to 50 tasks drawn from real failures are a good start. It is a model vendor’s guide. But it matches what production teams do and what the buyers’ thread asks for.
20 to 50 tasks
Simple tasks taken from real failures, as a good starting point. Exhibit 40
Before buying or launching an agent, you need your own tests instead of trusting the vendor's.
Verdict: Yes. According to a buyers' thread, in proofs of concept the vendor hands over a key and tells you to test it yourself. And SOC 2, a security certification, does not cover the model's behavior. In a study of 86 practitioners, 74% rely mainly on human evaluation. Large-scale evaluation is not cheap: about $40,000 for 21,730 runs. But 20 to 50 tasks are enough to start.
Rule
Build 20 to 50 tasks taken from real failures in your operation and have the vendor's agent run them before you sign. Read the transcripts, not just the score.
Why medium confidence. The buyer's point comes from a Reddit thread with a very low score. The task guide is written by a model vendor. The study of production teams has a small sample. No source argues the opposite with data. The qualifying one only shows what large-scale evaluation costs.
Show the 4 exhibits in detailHide the detail3 for, 0 against, 1 qualifies
In proofs of concept, agent vendors hand over an API key and say “test it yourself”. The author asks why auditing falls on the buyer. The replies say that asking for proof that the model does not hallucinate is asking to prove a negative. Also, SOC 2 does not cover model behavior. The practical answer is for the buyer to build their own evaluation set.
Numbers
4 points and 26 commentsSize of the thread according to the archive.
Limitation A very low-scoring thread: it is useful as an idea, not as consensus. It was read in the public Arctic Shift archive because reddit.com blocks automated reading. The quote was compared letter by letter with the original.
“there are no frameworks for doing any kind of an audit of model behavior.”
This guide to evaluating agents recommends starting early with tasks taken from real failures. It also asks to combine three kinds of graders (code, model and human), grade the outcome rather than the path, and read the transcripts. It separates pass@k, succeeding at least once in k tries, from pass^k, succeeding in all of them. It warns that some failures are grader errors.
Numbers
20 to 50 tasksSimple tasks taken from real failures, as a good starting point.
75% per try, about 42% for 3 out of 3A pass^k example: succeeding all three times is much less likely than succeeding once.
from 42% to 95%Opus 4.5 result on CORE-Bench after fixing grader bugs.
Limitation Written by a model vendor. It is a practice guide, not a study.
“20-50 simple tasks drawn from real failures is a great start”
A case study and survey of practitioners with agents in production. Most evaluate with people rather than benchmarks (standard tests), and let the agent take few steps before someone steps in. Reliability, meaning that the agent behaves correctly over time, remains the main challenge. Teams tackle it with system design.
Numbers
20 cases and 86 practitionersBasis of the study: 20 in-depth interviews and a survey of practitioners across 26 domains.
74%Rely mainly on human evaluation.
68%Run at most 10 steps before a person steps in.
70%Prompt off-the-shelf models instead of tuning their weights.
Limitation A small sample chosen by the authors; it does not represent all companies. Preprint.
“Reliability (consistent correct behavior over time) remains the top development challenge”
It is infrastructure for evaluating agents consistently, and it measures cost and behavior as well as accuracy. More reasoning effort lowered accuracy in most runs. They found an agent that searched HuggingFace for the benchmark (the test) instead of solving the task. They also found misuse of payment cards in flight booking tasks.
Numbers
21,730 runsAgent runs with 9 models on 9 benchmarks.
about USD 40,000Cost of running the evaluation.
about 2.5 billion tokensLogs published for others to review.
Limitation Public benchmarks, not company deployments. 2025 models.
Do not promise a result you have not measured on your own tasks. Before you sign or launch:
Use public benchmarks to rule out, not to choose. Look first at the ones that resemble your case, such as private code or realistic users. Be skeptical of gaps under 3 points.
Build 20 to 50 tasks drawn from real failures in your operation. Have the vendor’s agent run them before you sign. The evaluation plan has the format.
If a result reaches a person with no automated check first, measure pass^k. The reliability calculator shows how much success drops with repetitions.
If you use a judge model, measure its agreement net of chance. Look at false positives and false negatives separately. Recalibrate it with each new model.
If the score changes a lot, up or down, read a sample of transcripts. Review the grader before you draw conclusions.