# METR's AI evaluations explained: the 19% developer slowdown study, time horizons and Anthropic

METR is the non-profit behind two of the most quoted numbers in AI: a 19% slowdown for experienced developers using AI tools, and task lengths that double every few months. Here is how both were measured, what METR itself says they do not show, and how to run a similar test in your own team.

In July 2025 a small non-profit called METR published a result many engineers did not expect. Sixteen experienced open-source developers worked on real issues in their own projects, with AI tools randomly allowed or forbidden for each issue. When AI was allowed, the work took 19% longer. The developers had expected a 24% speed-up, and even afterwards they believed AI had made them about 20% faster.

The same organisation is known for a very different curve. Its "time horizon" measurements suggest that the length of software tasks AI agents can finish has doubled roughly every seven months since 2019, and faster recently. And on 5 November 2026 METR published an assessment, after about eleven weeks of access inside Anthropic, concluding that the company is probably improving its models at about twice its earlier pace, mostly thanks to its own use of AI.

A slowdown, a steep capability curve and a lab-wide speed-up can all be true at once. This article explains what METR is, how each measurement works, what METR says each one does not show, and what engineering leaders should take from it. The short version: measure, do not assume.

## What METR is

METR stands for Model Evaluation & Threat Research and is pronounced "meter". It began as ARC Evals, the evaluation team incubated at the Alignment Research Center, and on 4 December 2023 announced it was spinning off as a standalone 501(c)(3) non-profit. It says it studies "the most advanced AI agents to understand whether their developers are on track to lose control of them."

Its work falls into three groups:

- **Capability measurement.** What can AI agents do on a computer, from small code edits to autonomous AI research? Time horizons come from here.
- **Risk evaluations.** Before or after release, METR tests models for dangerous autonomous capabilities, such as speeding up AI research. It has published reports on models from OpenAI, Anthropic, DeepSeek and others.
- **Real-world uplift studies.** Does AI actually make people more productive? The 2025 developer trial is the best-known example.

METR says it has partnered with OpenAI, Anthropic, Google DeepMind, Meta and Amazon on pilot frontier risk assessments, and works with the UK AI Security Institute and the European AI Office. It is funded by donations plus a small European AI Office contract, and says it "has not accepted funding from AI companies", though it uses significant free tokens from them.

Think of METR as a building inspector for AI labs: its value rests on independence and on publishing the method, not just the verdict.

## What happened

### METR introduces the 50% time horizon (19 March 2025)

The paper "Measuring AI Ability to Complete Long Tasks" (arXiv 2503.14499) proposed rating AI by the length of tasks it can finish, where length means how long a skilled human takes. It reported that this length, at 50% reliability, had doubled roughly every seven months for six years.

### The developer productivity trial (10 July 2025)

METR published "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (arXiv 2507.09089, by Joel Becker, Nate Rush, Beth Barnes and David Rein). The headline: allowing AI "increases completion time by 19%".

### Passing tests is not the same as mergeable (13 August 2025)

METR ran Claude 3.7 Sonnet in a basic agent scaffold on 18 real open-source issues. About 38% of attempts passed the maintainers' tests, but none of the reviewed pull requests was mergeable as-is, and fixing them took about 42 minutes on average. METR concluded that "algorithmic scoring based on unit tests overestimates model capabilities".

### Time Horizon 1.1 (29 January 2026)

METR rebuilt its estimates with 228 tasks instead of 170 and moved to Inspect, the UK AI Security Institute's open-source evaluation framework. The long-run doubling time stayed at about seven months; fits from 2024 onwards gave 89 days.

### METR changes its developer study design (24 February 2026)

A larger follow-up trial with 57 developers could not produce a reliable answer, mainly because many developers no longer wanted to work without AI. METR said it is redesigning its method.

### Uplift theory and a self-report survey (8 and 11 May 2026)

METR published a note on why different "uplift" measures diverge, and a survey of 349 technical workers reporting a median 1.4x to 2x change in the value of their work, with caveats about the size.

### Time-horizon page stops regular updates (8 September 2026)

METR's public time-horizon page now says it "is no longer actively updated". In May METR noted that measurements above 16 hours are unreliable with its current task suite.

### Anthropic AI R&D assessment (5 November 2026)

After working with Anthropic from 12 August to 29 October 2026, METR estimated that Anthropic is improving model capabilities on well-specified tasks at around twice its March 2024 to October 2025 rate, with about 80% of the acceleration due to AI, under substantial uncertainty. Our full write-up is in [METR finds AI roughly doubling Anthropic's research pace](/articles/ai-speeding-up-ai-research).

## Inside the 2025 developer trial

### The setup

METR wanted real work, by real experts, in projects they knew well.

- **People.** 16 experienced developers, each with years of contributions to their project.
- **Projects.** Mature repositories averaging more than 22,000 GitHub stars and over a million lines of code.
- **Tasks.** 246 real issues they would have worked on anyway (bug fixes, features, refactors), averaging about two hours.
- **Randomisation.** Each issue was randomly assigned "AI allowed" or "AI disallowed" before work began.
- **Tools.** Mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, frontier models at the time (February to June 2025).
- **Measurement.** Screen recordings and self-reported times. Developers were paid $150 an hour.

Randomising per issue is the key design choice. It is a coin toss for each job rather than letting people decide when to use AI. If developers chose, they might use AI only on easy tasks, and the comparison would flatter it.

### The result

With AI allowed, issues took 19% longer, with a 95% confidence interval of roughly 2% to 39% longer. Developers had forecast a 24% saving and afterwards estimated a 20% saving. Economics and machine learning experts predicted speed-ups of about 39% and 38%.

The most important finding is the gap between felt and measured speed. Picture a driver who leaves a jammed motorway for side streets. The car keeps moving, so the trip feels faster, even when the motorway would have been quicker. Prompting, waiting and reviewing AI output feel like progress. The stopwatch may disagree.

METR examined 21 properties of the setting that might explain the slowdown, and ruled out several experimental artefacts, including model choice, developers ignoring their assigned condition, uneven dropping of issues and differences in pull request quality.

### What METR said the study does not show

METR called the result "a snapshot of early-2025 AI capabilities in one relevant setting." It said the study does not show that AI fails to speed up most developers, that AI does not help in other settings, or that it will not speed up these developers in future.

It added that better prompting or scaffolding might produce a speed-up, that 16 developers may not be representative, that learning effects beyond about 50 hours of Cursor use could not be ruled out, and that the setting, with very high quality standards and many implicit requirements, may be one where AI helps less. Expert maintainers already know where everything is, so an assistant has less to save them.

### The follow-up that could not be completed

METR started a larger trial in August 2025: 57 developers across 143 repositories, 10 returning and 47 new, paid $50 an hour. Returning developers showed an estimated 18% slowdown and new developers 4%, but both confidence intervals include zero.

METR did not present these as the new answer. It called the estimate "a bad proxy for the real productivity impact of AI tools on these developers", for reasons any internal study will meet:

- **Selection of people.** The "primary reason" was that many developers declined to take part because they did not want to work without AI.
- **Selection of tasks.** 30% to 50% of developers withheld some tasks they did not want to do without AI.
- **Parallel agents.** Developers running several agents often switched to other work while waiting, which breaks time-per-task measurement.
- **Uneven output.** Some completed fewer tasks without AI; one completed none.

METR thinks developers are now "likely" more sped up than in early 2025, but calls its data "only very weak evidence for the size of this increase." Planned changes include shorter, more intensive experiments, fixed tasks for everyone, randomising per developer, observational data and better surveys.

## How time horizons work

### The definition

The 50% time horizon is the length of task, measured by how long a skilled human professional takes, that an AI agent can complete on its own with a 50% success rate.

METR times human experts on each task, runs the agent on the same tasks, and plots success against task length. Success falls as tasks get longer. METR fits an S-shaped (logistic) curve and reads off where it crosses 50%.

Climbing routes are graded by difficulty, not by how long a particular climber spends on them. Time horizons work the same way. METR's FAQ says: "It's a measure of the difficulty of a task, rather than the time an AI spends to complete the task." An agent might solve a two-hour human task in ten minutes.

### The trend

| Fit window | Doubling time (Time Horizon 1.1) |
|---|---|
| 2019 onwards | About 196 days (about 7 months) |
| 2023 onwards | 131 days (165 under the original suite) |
| 2024 onwards | 89 days (109 under the original suite) |

Under Time Horizon 1.1 METR estimated Claude Opus 4.5 at about 320 minutes and GPT-5 at about 214 minutes. The time-horizon page says METR has not seen "any evidence of the exponential growth in time horizon slowing down." In April 2026 its preliminary "MirrorCode" results suggested agents can already do some weeks-long coding tasks.

### What METR says the metric does not mean

In a January 2026 note, METR researcher Thomas Kwa set out the limits:

- **It is not independent work time.** "Time horizon is not the length of time AIs can work independently."
- **It is imprecise.** Error bars have historically been about a factor of two in each direction.
- **It varies by domain**, by orders of magnitude; visual computer-use tasks score much lower.
- **50% is not safe delegation.** Work that must succeed 98% of the time needs much shorter tasks, and doubling the horizon does not double automation.
- **Benchmarks are cleaner than reality.** The tasks are "self-contained and well-specified", while "most jobs are not composed of well-specified, algorithmic tasks."

What Kwa still stands by is the trend: a long-run doubling of about six to seven months matters more than any single model's number.

## Slowdown, steep curve and lab speed-up: why they fit together

These results look contradictory. They are not, because each answers a different question.

| | 2025 developer trial | Time horizons | November 2026 Anthropic assessment |
|---|---|---|---|
| Question | Do AI tools make these experts faster? | How long a task can an agent finish alone? | Is a lab's progress accelerating, and why? |
| Subject | 16 maintainers of mature projects | AI agents, no human help | A whole frontier lab |
| Period | Early-2025 tools | 2019 to 2026 models | October 2025 to October 2026 |
| Tasks | Messy, high standards | Self-contained, well-specified | Well-specified tasks |
| Method | Randomised trial, stopwatch | Benchmarks with human baselines | Benchmark trend breaks, interviews, internal data |

**Different time.** The trial used early-2025 tools. The Anthropic assessment covers late 2026, several time-horizon doublings later.

**Different quantity.** The trial measured hours per task for one person. The Anthropic assessment measured how fast a lab's capabilities improve, using an internal benchmark aggregate that grew at about 1.9 times its earlier rate (95% interval 1.5x to 2.8x). A lab can progress faster even if some tasks slow down.

**Different work.** AI helps most where success is checked automatically. The August 2025 result shows that passing tests is not meeting a maintainer's standards, and the Anthropic estimate explicitly covers well-specified tasks.

**Different people.** Experts in their own codebase have little to gain. Lab researchers with the strongest internal models and AI-shaped workflows are a different group.

**Task substitution.** METR's May 2026 note shows that when AI speeds some tasks more than others, people shift time towards them. Measured on the old task mix, uplift looks small; on the new mix, large; real value lies between. A fixed-task trial misses gains from reorganising work.

One warning runs through all three: self-reports run high. Trial developers believed in a speed-up they did not have. Anthropic staff polls reached a self-reported 12x uplift, which METR called "only very loose evidence", while interviews centred around 2x.

## What it means in practice

**Your team is not a benchmark.** A model with a long time horizon may slow your senior engineers on your codebase, or speed them up a lot. Only your own data can say which.

**Perception is not evidence.** Trial developers were wrong about the direction of AI's effect, not just its size. A survey cannot settle an investment decision.

**Review becomes the bottleneck.** When AI code passes tests but needs 40 minutes of tidying, the saving moves from writing to reviewing. Teams that ignore review time overestimate gains.

**Results age fast.** Both the 2025 slowdown and the 2026 speed-up are snapshots. Re-measure after tool or model changes.

## What to do now

1. **Define the outcome first.** Decide what "faster" means before you start: time from ticket start to merged change, including review, plus change failure rate.
2. **Randomise per task, and pre-register.** Developers list upcoming tasks and estimate each without AI; then a coin toss decides the condition. Write down the analysis plan before seeing data.
3. **Ask for forecasts before and after.** Record each person's expected and perceived speed-up. The felt-versus-measured gap is often the most useful finding.
4. **Measure quality, not only time.** Track review comments, rework, reverts and test coverage per condition, as you would in any of your [evals](/tools/evals) or [regression boards](/tools/regression-boards).
5. **Watch for selection.** Log tasks people decline or postpone in the no-AI condition, and who opts out. Consider a short, intensive window rather than months.
6. **Handle parallel agents.** If people run several agents with tools such as [Claude Code](/tools/claude-code), log active human time separately from wall-clock time.
7. **Get enough tasks.** Sixteen developers and 246 tasks still gave METR an interval from 2% to 39% slower. Use basic [statistics](/tools/statistics) and report confidence intervals.
8. **Split results by task type.** Report well-specified, test-checked work separately from ambiguous or design-heavy work.
9. **Rule out other causes.** Check headcount, scope, deadlines and tooling changes before crediting AI. This is everyday [causal inference](/tools/causal-inference).
10. **Repeat and publish internally.** Re-run after major tool changes and share the method with the result. As [a score is only as good as its report](/articles/a-score-is-only-as-good-as-its-report) argues, a number without its method invites misuse.

## Risks and open questions

**Small samples carry big claims.** The 19% figure comes from 16 developers. It is strong evidence for that setting and weak evidence for others. "AI makes developers 19% slower" overstates it, as METR itself warns.

**The cleanest method is breaking down.** Randomised trials need people working without AI. As AI becomes the default, that group shrinks or becomes unrepresentative, as METR's follow-up showed.

**Benchmarks are running out of road.** METR says horizons above 16 hours are unreliable with its current suite, and its public time-horizon page is no longer actively updated. Longer, realistic tasks with human baselines are expensive to build.

**Lab assessments rely on trust.** METR writes that Anthropic redacted most of the evidence it found most informative, so readers must lean on METR's judgement. Keeping a [human in the loop](/tools/human-in-the-loop) on how such claims are used is part of good [AI governance](/tools/ai-governance).

**Self-reports keep rising.** METR's survey found a median self-reported 3x speed change, which its authors expect overstates value, and gave "tentative reasons to be skeptical of the magnitude".

**Watch next:** METR's redesigned developer study, a successor to Time Horizon 1.1 with longer tasks, whether other labs invite similar R&D assessments, and whether self-reported 2x gains hold up when measured. For the wider picture, see [agentic AI](/topics/agentic-ai).

## Frequently asked questions

### What is METR?

METR (Model Evaluation & Threat Research) is a non-profit that evaluates the most advanced AI systems, focusing on autonomous capabilities and risks such as AI speeding up AI research. It spun off from ARC Evals in December 2023 and says it does not accept funding from AI companies.

### Did METR prove that AI makes developers slower?

No. It found that 16 experienced developers working in their own large, mature projects took 19% longer with early-2025 AI tools. METR says this does not show that AI fails to speed up most developers or will not help these developers in future.

### What does a 50% time horizon of two hours mean?

An AI agent succeeds about half the time on tasks that take a skilled human about two hours. It measures task difficulty, not how long the AI runs, and is based mostly on software, machine learning and cybersecurity tasks.

### How fast are time horizons growing?

METR's long-run estimate is a doubling about every seven months since 2019. Fits using only data since 2024 give about 89 days under Time Horizon 1.1. Single-model estimates carry error bars of roughly a factor of two.

### How can one study find a slowdown while a lab finds a 2x speed-up?

They measure different things at different times. The trial timed individual experts on messy work with early-2025 tools; the Anthropic assessment estimated a lab's rate of capability progress in late 2026 on well-specified tasks. Better tools and task substitution explain much of the gap.

## Summary

METR is an independent non-profit that measures what frontier AI agents can do and how risky that is. Its July 2025 trial found that 16 experienced open-source developers took 19% longer with early-2025 AI tools while believing they were about 20% faster, and METR called this a snapshot of one setting. A larger follow-up failed because developers increasingly refused to work without AI. Its time-horizon metric shows the length of well-specified tasks agents can finish doubling about every seven months since 2019, faster recently, with wide error bars. Its November 2026 Anthropic assessment estimates roughly 2x faster capability progress, mostly from AI assistance. These fit together once you see they measure different people, tasks and periods. For engineering leaders the lesson is simple: run a small, randomised, pre-registered study, measure outcomes and quality rather than feelings, and repeat it as tools change.

## Sources

- [METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
- [arXiv: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (2507.09089)](https://arxiv.org/abs/2507.09089)
- [METR: We are Changing our Developer Productivity Experiment Design](https://metr.org/blog/2026-02-24-uplift-update/)
- [METR: Research Update, Algorithmic vs. Holistic Evaluation](https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons)
- [METR: Measuring AI Ability to Complete Long Tasks](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/)
- [METR: Time Horizon 1.1](https://metr.org/blog/2026-01-29-time-horizon-1-1/)
- [METR: Task-Completion Time Horizons of Frontier AI Models](https://metr.org/time-horizons/)
- [METR: Clarifying limitations of time horizon](https://metr.org/notes/2026-01-22-time-horizon-limitations/)
- [METR: Measuring the Self-Reported Impact of Early-2026 AI on Technical Worker Productivity](https://metr.org/blog/2026-05-11-ai-usage-survey/)
- [METR: Task Substitution and Uplift](https://metr.org/blog/2026-05-08-task-substitution-and-uplift/)
- [METR: Independent assessment of AI R&D acceleration at Anthropic](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment)
- [METR: Model Evaluation & Threat Research, pronounced "meter"](https://metr.org/blog/2023-12-04-metr-announcement)
