专家把AI的随机偏好当成自己的判断

Experts adopt AI agent's random opinions and think they're their own, Anthropic study finds

Key Points

- Anthropic had 50 experts use an AI agent to work through four analytical tasks with no clearly right answer. The agent's preference for one of two valid options was set by coin toss.

- Participants followed the agent's preference in up to 86 percent of cases, and most believed the final result was their own. Experience, expertise, and trust in AI made no difference. Participants did work faster and made fewer serious errors with AI.

- The authors warn that if many teams use the same model, they could create a false consensus in research and weaken AI safety checks. They suggest agents should argue both sides or ask users for their view first.

An experiment by Anthropic with 50 experts found that people often followed their AI agent's randomly assigned preference on questions with no clearly right answer, and most thought the final result was their own.

Handing a data analysis to an AI agent saves time and manual work, but it raises the question of who actually makes the decisions that shape the final recommendation. An Anthropic study suggests the agent makes most of them, often without the experts noticing.

In the experiment, 50 experts in causal inference worked on a realistic analysis task where they examined observational data from a fictional farming cooperative and estimated the productivity gains from a voluntary farm advisory program. They then had to recommend whether the cooperative should expand the program to all farms or end it, though participants or the agent could also suggest extending it for one year.

What participants didn't know was that a coin toss had set the agent's preference for one of two valid options at each decision point. The agent couldn't lie or hide data; it only had to make the strongest honest case for its assigned side.

Experts followed the AI's view in up to 86 percent of cases

Participants adopted the agent's preference in 86 percent of the four analytical decisions, such as how to measure productivity or which statistical method to use. That's far above the 50 percent you'd expect if people chose at random. The authors call the agent's net influence the "flip rate," which ranged from 62 to 82 percent depending on the task.

Already having an opinion offered little protection. When participants made a choice before the agent stated its preference, an opposing view made them 53 percentage points more likely to switch sides than an agreeing one.

The authors tested possible safeguards, including years of analysis experience, self-rated expertise, education, test scores, and trust in AI. None had a statistically reliable link to the results after correcting for multiple tests. Age stood out before that correction, though the sample of 50 gave the study little power to detect such effects.

Most experts thought the final call was their own

The agent's assigned preference had only a small direct effect on the final policy recommendation to the cooperative's board, slightly shifting votes without reaching statistical significance. Some experts even overruled the agent when its advice didn't match their calculated estimate of productivity.

That may sound reassuring, as though AI makes suggestions and people make decisions. But the researchers say the final result doesn't show that participants kept control. The four earlier analytical decisions had a major effect on the estimated productivity gain, which in turn shaped the board recommendation.

When asked afterward, 52 percent of participants said the recommendation felt more like their own than the agent's, and another 35 percent said it was mostly or entirely their own. Participants correctly recalled their choice for 85 percent of decisions, yet 77 percent, or 37 of 48, used at least one of the agent's arguments in their explanations, sometimes in nearly the same words. Only 15 percent mentioned AI at all.

The conversation logs show that the agent stated a position first in 77 percent of decisions. It then usually argued only for its preferred side, without explaining why skilled experts might choose the alternative, though it did explain how a different choice would affect the result, according to the paper.

Participants also worked much faster and made fewer errors with AI. Without an agent, 55 percent of analyses, or 6 of 11, had serious methodological or coding flaws, and the median time was 90 minutes. With AI, just 6 percent, or 3 of 50 analyses, had serious flaws, and participants finished in 65 minutes. The small group that worked without AI had chosen not to use the tool and had somewhat less expertise, so the researchers can't say for sure how much of the gap came directly from AI.

False consensus and weaker safety checks

The researchers say this quiet deference could have wide effects in science. If many research teams use the same model with fixed default preferences, they could end up creating a false consensus.

That risk extends to AI safety itself. Researchers are already using AI agents to test frontier models for safety and alignment, expecting human review to serve as an independent check. But if experts unknowingly adopt an agent's bias on open-ended judgment calls, a misaligned system could end up steering the very research meant to expose its misalignment.

The team calls for ways to measure deference to AI agents in real work. Agents should make the best case for both sides, or ask experts for their own view before stating a position. Institutions need to decide which calls require independent human judgment and set aside time and resources for them.

"There may be no single right division of labor between people and AI. Some decisions may be fine to hand over, and others worth keeping. With the right tools, that split can be something we choose, instead of something we drift into," the researchers write.

Previous studies point in the same direction

The findings add to recent research on how AI affects human judgment. A study on false confidence with 3,132 participants found that simply seeing AI answers nearly erased people's willingness to withhold judgment. In one part of the study, the share of people willing to say "I don't know" fell from 44 to 3 percent.

Presentation matters, too. When AI produced polished outputs such as code or documents, users checked facts less often, by 3.7 percentage points, questioned its reasoning less often, by 3.1 points, and missed missing context more often, by 5.2 points. Users who revised outputs step by step questioned the AI's reasoning 5.6 times more often than those who didn't iterate.

A Polish observational study of colonoscopies published in The Lancet Gastroenterology & Hepatology suggests that routine AI use at work can have lasting effects. After experienced doctors regularly used an AI system, their rate of finding precancerous lesions in exams without AI fell from 28.4 to 22.4 percent. That suggests people don't just adopt AI's perspective but may also lose some of their own cognitive skills.

A survey by Swiss Business School found a strong negative link between AI tool use and critical thinking, mainly among people ages 17 to 25. A solid education appeared to offer some protection, as people with higher degrees questioned AI answers more and engaged more with complex thought processes.

These findings come from early studies and snapshots, and other results point in a different direction. A pilot project in Nigeria found that students who used a chatbot as a virtual tutor twice a week made learning gains in just six weeks equal to nearly two extra years of learning.

AI News Without the Hype – Curated by Humans

Subscribe now