We are Changing our Developer Productivity Experiment Design
METR (AI lab research) · 24 Feb 2026
Read original sourceOpen accessNot peer reviewed
Where this sits in the debate
- Will AI actually boost productivity and growth? · its finding: it depends (why studies differ)
Direction of the finding as read from the abstract by an AI model (AI-written).
Source excerpt
METR previously published a paper which found the use of AI tools caused a 20% slowdown in completing tasks among experienced open-source developers, using data from February to June 2025. To understand how AI is impacting developer productivity over time, we started a new experiment in August 2025 with a larger pool of developers using the latest AI tools. Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI, which likely biases downwards our estimate of AI-assisted speedup. We additionally believe there have been selection effects due to a lower pay rate (we reduced the pay from $150/hr to $50/hr), and that our measurements of time-spent on each task are unreliable for the fraction of developers who use multiple AI agents concurrently. Based on conversations with study participants, we believe it is likely that developers are more sped up from AI tools now — in early 2026 — compared to our estimates from early 2025. However, because of the selection effects in our experiment, our data is only very weak evidence for the size of this increase. Our raw results show some evidence for speedup. Our early 2025 study found the use of AI causes tasks to take 19% longer, with a confidence interval between +2% and +39%. For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%. Among newly-recruited developers the estimated speedup is -4%, with a confidence interval between -15% and +9%. However the true speedup could be much higher among the developers and tasks which are selected out of the experiment. Some developers self-report very high speedups, though as we documented in our ea
About this publication
Lab or institute research, not peer reviewed. A report or blog post published by an AI lab or research institute on its own site. It has not been through independent peer review, and the publisher may have a stake in the findings.
- Published
- 2026 (8 months ago)
- Citations
- Not recorded for this source
Citation counts come from OpenAlex; none are recorded for this source — absence of a count is not absence of citations.
Related by theme, not necessarily by method or conclusion.
- Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality
SSRN · 2023 - Generative AI at Work
The Quarterly Journal of Economics · 2025 - The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
arXiv · 2023 - The Productivity J-Curve: How Intangibles Complement General Purpose Technologies
American Economic Journal Macroeconomics · 2020
Overview assembled automatically from collected metadata. How this site is built.