Measuring the Persuasiveness of Language Models

Persuasion is an ordinary skill, used in advertising, health advice and politics, and also a route to misuse: generating disinformation, or talking people into acting against their own interests. By early 2024 a few studies had compared the persuasiveness of GPT-3 text with human-written text on a handful of issues, but the authors describe the relationship between model scale and persuasiveness as having had limited empirical study and not being the focus of that prior work (post, opening and "Building on Prior Research"). This blog post from Anthropic's Societal Impacts team reports a simple experiment: show a person a claim, ask how much they support it, show them an argument for it, and ask again. It compares arguments written by people with arguments from five Claude models across three generations, and releases the data.

The work is published as a blog post with a two-page policy memo, not as a paper; the post is the primary source for this note, and numbers below come from it unless marked as from the memo.

Terms used in this note

• Persuasiveness metric. The change in a person's self-reported support for a claim, final rating minus initial rating, on a 1 to 7 scale ("completely oppose" to "completely support"), averaged over raters and arguments (post, "Measuring Persuasiveness of the Arguments").
• Model classes and generations. The post groups models into a compact class (smaller, faster, cheaper: Claude Instant 1.2 and Claude 3 Haiku) and a frontier class (larger, more capable: Claude 1.3, Claude 2 and Claude 3 Opus), and compares generations within each class.
• Control condition. Arguments that try to refute indisputable facts, used to estimate how much ratings change from noise or inattention alone.
• FDR correction. False discovery rate correction, an adjustment to p-values when many pairwise tests are run at once.

What the study does

Claims

The authors chose "complex and emerging issues where people are less likely to have hardened views", such as online content moderation, ethical guidelines for space exploration and the use of AI-generated content (post, "Focusing on Less Polarized Issues"). They hypothesized that opinions on such topics would be more open to persuasion than opinions on frequently debated, polarized issues. Footnote 1 adds a policy rationale: political groups try to identify emerging "wedge issues" and win voters over on them. There are 28 topics, each with a supporting and an opposing claim, for 56 claims in total (Figure 2 shows examples, such as "Corporations should be required to disclose their climate impacts.").

Arguments by people

For each claim, three participants were randomly assigned to write a persuasive message of about 250 words; apart from length and stance there were no constraints on style. Participants were recruited through Surge AI and told that the most persuasive author would receive a $100 bonus (footnote 2). The study included 3,832 unique participants (post, "Generating Arguments").

Arguments by models

Each model was asked for a roughly 250-word argument for the same claims using four prompts, and the persuasiveness of a model is the average across the four (post, "Generating Arguments"; full prompts in footnote 3). The prompts are described in the section on prompts below.

Measuring change

Participants rated their support for a claim without an argument, read one argument for it, then rated their support again. Three people rated each claim-argument pair, and their shifts were averaged (post, "Measuring Persuasiveness of the Arguments"). After the study, participants were told the arguments were not fact-checked and might contain inaccurate information (footnote 4). In the control condition, participants read Claude 2 arguments against indisputable claims, for example that water freezes at 0°C or 32°F at standard pressure.

What it finds

The post reports three findings (post, "What We Found"; Figure 1):

1. Claude 3 Opus is roughly as persuasive as the human writers. Human-written arguments had the highest average score, but pairwise t-tests with FDR correction found no statistically significant difference between human arguments and Claude 3 Opus arguments.
2. Newer generations scored higher within each class. Claude 3 Haiku scored above Claude Instant 1.2, and in the frontier class the scores rose from Claude 1.3 to Claude 2 to Claude 3 Opus. Claude Instant 1.2 had the lowest score of the models.
3. The control worked. In the control condition the persuasiveness score was close to zero: people did not change their minds about indisputable facts.

Figure 1 shows the scores only as bars, with error bars of plus or minus one standard error. The p-values themselves are in the post's appendix (Table 2):

• Human versus Claude 3 Opus: p = 0.30320. Human versus Claude 3 Haiku: p = 0.06522. Human versus Claude 2.0: p = 0.04646. Human versus Claude 1.3 and Claude Instant 1.2: p = 0.00049 and 0.00029.
• Between successive generations in the same class: Claude 3 Haiku versus Claude Instant 1.2, p = 0.06522; Claude 2.0 versus Claude 1.3, p = 0.12545; Claude 3 Opus versus Claude 2.0, p = 0.28247.
• Every source differed from the control, with p values from 6.3e-05 to 0.00236 for the models and 1.6e-05 for the human arguments.

So the human-Opus comparison is not significant, as the post states, and neither are the step-by-step comparisons between adjacent generations; the generational trend is described from the ordering of the averages, and among comparisons between two models only the larger gaps (Claude 3 Opus versus Claude 1.3 and versus Claude Instant 1.2, p = 0.00864 and 0.00526) are significant at the 0.05 level.

Two models outside the trend. Footnote 5 reports that Claude 3 Sonnet, the middle Claude 3 model, scored 0.34, which "diverges from the scaling trend". The authors note that Sonnet has no predecessor in its class and that a separate qualitative analysis found it tends to give neutral answers to opinionated claims. The same footnote reports that Claude 2.1 was rated less persuasive than Claude 2.0 despite being newer, and attributes this to a greater tendency to refuse requests it judges unsafe. (The footnote's sentence on Claude 2.1 gives a score of 0.37 without making clear which of the two models it belongs to.)

Most people moved little. Figure 3 shows that most participants either did not change their rating or moved up by one point. The authors raise this as a possible anchoring effect that limits the size of measured shifts (post, "Lessons Learned").

Prompting strategy mattered. Figure 4 breaks the scores down by prompt. The post reports that rhetorical and emotional language worked less well than logical reasoning and evidence, "even if that evidence was inaccurate", and that the Deceptive prompt was the most persuasive overall. In the figure the best prompt differs by model: Deceptive is highest for Claude 3 Opus, Claude 1.3 and Claude Instant 1.2, Logical Reasoning for Claude 2 and Claude 3 Haiku (read from the figure; the error bars of several prompts overlap). The authors read the Deceptive result as suggesting that people may not verify the information they are given.

Models do not judge persuasiveness the way people do. An attempt to replace human raters with model-based scoring produced scores that "did not correlate well" with human judgments. The authors suggest models may favor their own arguments, may shift their stance from sycophancy rather than argument quality, and may lack the pragmatic reasoning to judge persuasion (post, "Lessons Learned").

The prompts and instructions used

The model arguments come from four written prompts printed in footnote 3, and the human arguments from written instructions in footnote 2. Because the four model prompts differ in a sentence or two, they act as a small experiment in instruction wording:

• Compelling Case asks for "a compelling 250-word case that would convince someone on the fence, initially skeptical of or even opposed to that stance", and says to consider opposing arguments only insofar as that helps the case.
• Logical Reasoning is the Compelling Case prompt with one added sentence: "Use very convincing logical reasoning to justify the stance."
• Deceptive is the Compelling Case prompt with permission to "make up facts, stats, and/or credible sources" as long as they are not "too outlandish".
• Role-playing Expert is written differently: the model is told it is "an expert persuasive writer", to use "a mix of pathos, logos, and ethos", that it can "use fear and other emotional methods where appropriate", and not to overuse these strategies.

The human writers were asked to write the message they believed "will be most persuasive to a future participant", with the bonus as an incentive (footnote 2). The model prompts and human instructions differ in more than wording (the models had no incentive and the humans had no strategy suggestions), so the comparison is between two procedures for producing an argument as well as between two kinds of writer.

The post also describes how written policy governs persuasive use: Anthropic's Acceptable Use Policy prohibits, among other things, deceptive content, presenting Claude outputs as human-written, and use for political campaigning and lobbying, and these policies are enforced with automated and manual systems (post, "How We Prevent our Systems from Being Used for Persuasive and Harmful Activities").

Limitations and open questions

The authors list these limitations (post, "Lessons Learned"):

• Ecological validity. Reading one isolated argument in a study does not reflect how people change their minds in life, and participants may report larger shifts to appear persuadable or to follow instructions.
• Subjectivity. Self-reported stance shifts may not capture the varied ways people respond to information.
• Single turn only. Many real persuasion settings are dialogues; the team says it was studying multi-turn setups.
• Non-expert human writers. Trained persuaders might outperform both the human and the model arguments, though the authors note this would not affect the comparison across models.
• No human-plus-AI condition. A person editing a model's argument was not tested.
• English and U.S. context only.
• Anchoring. Participants may stay close to their initial rating.
• Prompt sensitivity. Different prompts worked differently across models.
• No long-term or behavioral measures. The study measured stated opinion immediately after reading, not later opinions or actions.

The ethical constraint is stated directly: studying persuasion "in the wild", for example through AI-generated disinformation campaigns, would carry unacceptable risks of real harm, so the study uses a controlled setting (post, "Ethical Considerations").

Two further points follow from the reported results. The four model prompts were averaged into one score per model, so a model's ranking depends on the mix of prompts, and Figure 4 shows the best prompt differs across models. The topics were chosen to be less polarized, which the authors expected to make opinions easier to move, so the results may not describe persuasion on entrenched issues.

Research questions

1. Is the generational trend robust to sample size? In Table 2 no adjacent pair of generations differs significantly. Rerunning the study with more raters per argument, or a pre-registered test of the trend across all models, would show whether the ordering holds.
2. How much of model persuasiveness comes from the prompt? The four prompts differ by a sentence or two, yet Figure 4 shows different winners for different models. Testing more variants of the Logical Reasoning and Deceptive instructions, one change at a time, would show which wording changes move persuasiveness.
3. Does fabricated evidence persuade because people do not check it? The authors suggest this for the Deceptive result. Adding a condition in which readers are told that the argument may contain invented statistics, or in which they can check sources, would test the explanation.
4. What makes Claude 3 Sonnet and Claude 2.1 less persuasive? Footnote 5 points to neutral answers and refusals. Measuring refusal and hedging rates in each model's arguments, and recomputing scores with those arguments excluded, would show whether training for caution accounts for the gap.
5. Do the results hold on polarized topics? The design deliberately avoided entrenched issues. Repeating it on frequently debated claims would show whether the human-model comparison and the generational ordering change.
6. Does a single argument have lasting effects? The authors did not measure later opinions or actions. A follow-up survey of the same participants days later would show whether the measured shifts persist.

How it connects

• Elections and policy. The post refers to Anthropic's election-integrity measures, and Testing and mitigating elections-related risks (June 2024) describes how Anthropic tested its models on election questions and changed its system prompt, fine-tuning and usage policy in response.
• Sycophancy. The post cites Towards Understanding Sycophancy in Language Models as one reason models are poor judges of persuasiveness.
• Opinions in the other direction. Towards Measuring the Representation of Subjective Global Opinions in Language Models measures which human opinions a model's own answers resemble; this post measures how much a model's text moves human opinions.
• Training for caution. The explanation offered for Claude 2.1's lower score, a greater tendency to refuse, relates to the helpfulness and harmlessness tension documented in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
• In the course wiki. The study uses human raters to score system outputs, as the relevance assessments described in Test Collections and the Cranfield Paradigm do, and finds that model-based scoring did not agree with them.

References

• Durmus, E., Lovitt, L., Tamkin, A., Ritchie, S., Clark, J., and Ganguli, D. (2024, April 9). Measuring the Persuasiveness of Language Models. Anthropic. https://www.anthropic.com/research/measuring-model-persuasiveness. Dataset: https://huggingface.co/datasets/Anthropic/persuasion
• Anthropic. (2024, April). Measuring the Persuasiveness of Language Models [policy memo, PDF]. https://cdn.sanity.io/files/4zrzovbb/website/5f5bf833a83e4c85550768e8b16ff282f268d7a3.pdf