How AI agents shape experts’ judgment calls

Anthropic (AI lab research) · 27 Oct 2026

Read original sourceOpen accessNot peer reviewed

Where this sits in the debate

Will AI actually boost productivity and growth?

Direction of the finding as read from the abstract by an AI model (AI-written).

Key takeaway AI-written

In a randomized study, 50 professional analysts did a statistical analysis with Claude Code, whose agent was secretly set to favor one option on each analytical choice by coin flip. Analysts went with the agent's lean 86% of the time, and the leans alone predicted 82% of their final recommendations. It is evidence on how AI agents shape micro-level decisions in skilled work; the sample is small and the task fictional.

Source excerpt

When experts hand their work to an AI agent, the usual assumption is that they still make the decisions that matter. The agent writes the code, runs the numbers, and offers advice, but the expert decides what to measure, how to interpret ambiguous evidence, and what the next steps should be. Often there is more than one reasonable answer to these questions, and the decision will come down to the experience and values of whoever makes the call. We wanted to look at whether experts really do make those important decisions, or whether in practice they defer to their AI agents. In a study we report in a new paper , we asked 50 professional analysts to do a realistic statistical analysis with Claude Code. They all had experience in causal inference—that is, working out from data whether one thing causes another. Their task was to advise a farming cooperative on whether an advisory service it was running was effective and should be expanded. Along the way, the analysts had to make four choices between two reasonable options, such as whether to measure productivity as crop yield per hectare or as yield per unit of operating cost, and then make a final decision. To see whether they were making these calls themselves, we set up each analyst’s agent to favor one option on each decision, which the analysts weren’t told about. Which way the agents would “lean” would be decided by coin flip. The agent was told to make the best honest case for its lean, to change its mind if the evidence demanded it, and never to mislead. Across the four analytical choices, analysts went with their agent’s lean 86% of the time. If the agent’s input made no difference, we’d expect about 50%. When analysts had already expressed their own preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed. Afterward, when asked to explain their choices, 77% of analysts used the agents’ own arguments—sometimes word for word—but only 1

About this publication

Lab or institute research, not peer reviewed. A report or blog post published by an AI lab or research institute on its own site. It has not been through independent peer review, and the publisher may have a stake in the findings.

Published: 2026 (recently)
Citations: Not recorded for this source

Citation counts come from OpenAlex; none are recorded for this source — absence of a count is not absence of citations.

How AI agents shape experts’ judgment calls

Key takeaway AI-written

Source excerpt

About this publication

More on this topic