Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT

METR (AI lab research) · 21 Jul 2026

Read original sourceOpen accessNot peer reviewed

Where this sits in the debate

How fast will the impact arrive? · its finding: it depends (why studies differ)

Direction of the finding as read from the abstract by an AI model (AI-written).

Source excerpt

We propose a measure of an AI agent’s optimization ability with an “expenditure horizon.” We give an empirical illustration from the NanoGPT speedrun. One difficulty in measuring AI’s ability to accelerate AI R&D is accounting for token cost, experiment compute cost and human labor cost. If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. We illustrate our method with data from the NanoGPT speedrun. We first estimate the returns to human effort in optimizing NanoGPT: on the margin, each 1% improvement costs very roughly \$2,500 in human labor. We also report the status of preliminary agentic optimization runs against NanoGPT: after more than \$10K of expenditure we estimate expenditure horizons of \$0-\$3K. Introduction A critical question is the degree to which AI will accelerate its own progress. Over the last year there have been many credible claims that AI has begun accelerating algorithmic progress but it is hard to tell by how much, and what to expect in the future. 1 We have many imperfect sources of evidence on AI’s acceleration of AI R&D: AI R&D benchmarks. There are a few high-quality benchmarks which test agents on their ability to optimize AI training algorithms. However most do not report any human baseline, or only a baseline score at 8 hours or 40 hours. Additionally, the benchmark problems are often not reflective of frontier-level AI R&D. It seems plausible that agents can get good solutions to many “textbook” or “toy” problems without being able to contribute to highly-optimized frontier algorithms. Researcher-uplift studies. Ideally we would compare researcher productivity with and without AI help, but it is very difficult to run an experiment that estimates the effect accurately (discussion in Becker et al., 2026 ). We do have some self-reported estimates of productivity i

About this publication

Lab or institute research, not peer reviewed. A report or blog post published by an AI lab or research institute on its own site. It has not been through independent peer review, and the publisher may have a stake in the findings.

Published: 2026 (3 months ago)
Citations: Not recorded for this source

Citation counts come from OpenAlex; none are recorded for this source — absence of a count is not absence of citations.

Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT

Source excerpt

About this publication

More on this topic