Open Weight AI Models Require Proportional Evaluation Approaches

Patricia Paskov, Christopher Rodriguez, Sunishchal Dev, Stephen Casper · arXiv · 18 Jun 2026

Where this sits in the debate

Should governments act on AI now? · its finding supports the claim (why studies differ)

Direction of the finding as read from the abstract by an AI model (AI-written).

Abstract

Open-weight AI models (OWMs), or models released with publicly-available weights, are distributing rapidly and approaching the performance levels of leading closed-weight AI models (CWMs). While OWMs offer substantial scientific and economic benefits, their release introduces distinct risk factors for which existing evaluation practices, largely designed for CWM deployment, fail to account. In this paper, we argue that these risk factors demand distinct proportional evaluation (PE) approaches: evaluating without system-level safeguards (PE1), assessing robustness to modifications that undo model-level safeguards (PE2), testing selective capability amplification (PE3), and proxying worst-case misuse (PE4). We systematically review current evaluation practices of OWMs released in 2025 through April 2026, finding that only one of the 37 families of models reviewed fulfills PE1-4 and most do not fulfill any. This paper targets policymakers, funders, and researchers involved in AI evaluation. As OWMs grow increasingly capable, their evaluation warrants close attention from developers, funders, and governance bodies alike.

About this publication

Preprint. A preprint — posted publicly before or without journal peer review; findings may still be revised.

Published: 2026 (4 months ago)
Citations: Not recorded for this source

Citation counts come from OpenAlex; none are recorded for this source — absence of a count is not absence of citations.

Open Weight AI Models Require Proportional Evaluation Approaches

Abstract

About this publication

More on this topic