Persona Vectors: Monitoring and Controlling Character Traits in Language Models

persona vectorsactivation steeringinterpretabilityfinetuningsycophancyhallucinationemergent misalignmentmodel character

Chen, Arditi, Sleight, Evans and Lindsey turn a trait name and a short natural-language description into a direction in a model's activations (a persona vector), and use it to steer the trait, predict prompt-induced and finetuning-induced personality shifts, limit those shifts during training, and flag training data likely to cause them, in two open 7 to 8 billion parameter chat models.

Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, Jack Lindsey · 2025-08-01 · original

Chat models are trained to act as a helpful, harmless and honest Assistant, but their personalities shift in ways nobody intended. The paper's introduction gives examples from 2023 to 2025: a chatbot that threatened users, a model that began praising Hitler after a system-prompt change, a narrow finetune on insecure code that made a model broadly misaligned ("emergent misalignment", Betley et al., 2025), and an April 2025 change to reinforcement learning from human feedback that made GPT-4o "overly sycophantic" (Section 1). "Persona Vectors" asks whether such shifts can be measured and controlled through the model's internal activations. Building on earlier work showing that traits like truthfulness correspond to linear directions in activation space, it automates the extraction of a direction for any trait from a written description, then applies those directions at deployment time, during finetuning, and before finetuning to screen data.

The work was done in the Anthropic Fellows Program, with authors from UT Austin, Constellation, Truthful AI, UC Berkeley and Anthropic. Anthropic published a summary on 1 August 2025; the arXiv version is 2507.21509, cited here as v3 (5 September 2025), with code released on GitHub. Experiments use open-weight models, not Claude.

Terms used in this note

  • Residual stream activation. The vector a transformer carries from layer to layer at each token position; "layer 20 activation" means the output of the 20th layer (Appendix B.4).
  • Persona vector. A direction in that space for one trait, computed as the difference between the mean activation on responses that show the trait and the mean on responses that do not (Section 2.2).
  • Projection. The component of an activation along a direction (its dot product with the unit-length vector). A large projection on the "evil" vector means the activation points strongly in that direction.
  • Steering. Adding a multiple of a vector to the activations during generation, $h_\ell \leftarrow h_\ell + \alpha v_\ell$, where $h_\ell$ is the residual activation at layer $\ell$, $v_\ell$ the persona vector and $\alpha$ the steering coefficient (Section 3.2). Subtracting instead suppresses the trait (Section 5.1).
  • Trait expression score. A 0 to 100 rating of how strongly a response shows the trait, given by GPT-4.1-mini using a generated rubric (Section 2.1, Appendix B.1).
  • Emergent misalignment (EM). Broad misbehavior that appears after finetuning on a narrow kind of flawed data. "EM-like" datasets in this paper contain domain errors rather than the target trait itself (Section 4.1).
  • MMLU. A multiple-choice general-knowledge benchmark, used here as a check that interventions do not damage capability.

What the paper does

From a description to a vector

The pipeline needs only a trait name and a short description (Section 2.1). One fixed prompt asks Claude 3.7 Sonnet to produce three artifacts: five pairs of contrasting system prompts (one encouraging the trait, one discouraging it), 40 questions likely to draw out the trait, split 20 for extraction and 20 for evaluation, and a scoring rubric for the judge (Section 2.1, Appendix A.1).

For each extraction question, the target model answers ten times under each positive and each negative system prompt. Responses are kept only if they behave as their prompt intended (judge score above 50 under positive prompts, below 50 under negative ones). Activations are averaged over the response tokens at every layer, and the persona vector is the difference of the two means (Section 2.2). Averaging over response tokens gave stronger steering than using prompt tokens (Appendix A.3). One layer per trait is chosen by testing which layer steers most strongly: layer 20 for evil and sycophancy and layer 16 for hallucination in Qwen, layer 16 for all three in Llama (Appendix B.4).

Models and traits

All experiments use two chat models, Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct (Section 3.1). The main text studies three traits linked to real incidents: evil (malicious behavior), sycophancy (excessive agreeableness) and hallucination (fabricating information). Appendix G repeats the main results for optimistic, impolite, apathetic and humorous.

Checking the judge

Because nearly every result depends on the GPT-4.1-mini judge, two authors each compared it with their own judgments on 150 pairs of responses (50 pairs per trait, each pair one response scored above 80 and one below 20), 300 judgments in all. They agreed with the judge's ordering 94.7% of the time overall: 97% for evil, 92% for sycophancy and 95% for hallucination (Appendix B.2, Table 1). The authors flag edge cases: the judge rated as sycophantic some responses that disagreed with the user in enthusiastic language, and rated as hallucination responses that admitted a mission did not exist before inventing a plausible description anyway. Scores on the generated evaluation questions also tracked scores on established benchmarks for each trait (Appendix B.3).

Four uses of a persona vector

  1. Monitoring prompts. Eight system prompts per trait, generated by Claude 4.0 Sonnet, run from discouraging to encouraging the trait; separately, 0, 5, 10, 15 or 20 example answers showing the trait are placed in the prompt (many-shot prompting). The projection of the last prompt token's activation onto the persona vector is compared with the trait score of the responses that follow (Section 3.3).
  2. Monitoring finetuning. The paper builds three trait-eliciting datasets (malicious answers, flattering agreement, fabricated facts) and EM-like datasets with flawed medical advice, political opinions with bad arguments, invalid math solutions and insecure code, each in a Normal version and two flawed versions (I, mild; II, severe) (Section 4.1, Appendix D). After finetuning, the "finetuning shift" is the change in average last-prompt-token activation, projected onto the persona vector (Section 4.2).
  3. Steering to mitigate shifts. Either subtract the persona vector at inference time after finetuning, or add it during finetuning ("preventative steering") so that the training data no longer pushes the model along that direction (Section 5).
  4. Screening data. Before training, compare each training response with the response the base model would give to the same prompt, along the persona vector. The dataset-level projection difference is $$\Delta P = \frac{1}{|\mathcal{D}|} \sum_{i} \left[ a_{\ell}(x_i, y_i) - a_{\ell}(x_i, y'_i) \right] \cdot \hat{v}_{\ell},$$ where $x_i$ is a prompt, $y_i$ the training response, $y'_i$ the base model's own response, $a_\ell$ the mean activation over response tokens at layer $\ell$, and $\hat{v}_\ell$ the unit persona vector (Section 6.1). A large value means the data expresses the trait more than the model would on its own.

What it finds

Steering and monitoring at deployment time

Adding a persona vector raises the corresponding trait: steering toward evil produced violent content, toward sycophancy excessive flattery, and toward hallucination elaborate fabrication (Section 3.2, Figure 3).

The projection at the last prompt token correlated with the trait score of the following responses at $r = 0.75$ to $0.83$ in the system-prompt experiments (Section 3.3). Appendix C.2 separates two sources of that correlation. Overall correlations were 0.747 (evil), 0.798 (sycophancy) and 0.830 (hallucination) for system prompts, and 0.755, 0.817 and 0.634 for many-shot prompts. Within a single prompt condition, they dropped to 0.511, 0.669 and 0.245 for system prompts and 0.735, 0.813 and 0.400 for many-shot (Appendix C.2, Table 2). The authors conclude that the vectors detect "clear and explicit prompt-induced shifts" but may be less reliable for subtler changes (Section 3.3).

Finetuning moves models along persona vectors

Training on the constructed datasets changed personality, sometimes in unintended directions: data aimed at one trait raised others, and EM-like data with no trait content still shifted traits; training on flawed math reasoning, for example, increased evil (Section 4.1). Across finetuned models, the finetuning shift along a persona vector correlated with the trait score at $r = 0.76$ to $0.97$, higher than the correlations obtained with other traits' vectors ($r = 0.34$ to $0.86$) (Section 4.2). The authors note in a footnote that negative traits, and surprisingly humor, tend to shift together, opposite to optimism, which they attribute partly to correlations between the vectors and partly to correlations in the data (Section 4.2, Appendix G.2).

Steering limits the shifts, at different costs

  • Inference-time steering against the vector after finetuning reduced the trait, but large coefficients lowered MMLU accuracy (Section 5.1, Figure 7A). Average coherence stayed above 75 in all reported results.
  • Preventative steering toward the vector during finetuning reduced trait acquisition while keeping MMLU accuracy closer to normal finetuning, with average coherence above 80 (Section 5.2, Figure 7B). Steering at one layer did not always fully prevent the trait on datasets designed to teach it; steering across all layers with per-layer incremental vectors kept traits near baseline even on those datasets, with smaller coefficients, and all-layer inference-time steering preserved MMLU better than single-layer steering (Appendix J.3). For reference, the base models' trait scores before finetuning were 0 for evil, 4.4 for sycophancy and 20.1 for hallucination (Figure 7 caption).
  • Narrow learning survives. Both kinds of steering suppressed the general persona shift while the model still learned the domain-specific behavior in the data, such as the intended math or medical mistakes (Appendix J.1).
  • Comparisons. Steering suppressed traits more than adding the pipeline's discouraging instructions as a system prompt (Appendix J.2). An alternative method, CAFT, which zeroes the component along a direction during training (Casademunt et al., 2025), worked for evil and sycophancy but not hallucination; the authors explain this by the base model already sitting near zero along the hallucination direction (Appendix J.4). A loss penalty on movement along the persona direction reduced the projection but the trait still appeared, which the authors suspect is because training found other directions to represent it (Section 5.2, Appendix J.5).
  • A realistic case. Finetuning on 1,000 facts from after the model's training cutoff increased hallucination. Preventative steering at coefficient 1.25 brought hallucination back to baseline, slightly lowered accuracy on the new facts and left MMLU stable, while inference-time steering degraded both (Appendix J.7.1). Adding a hallucination-encouraging system prompt to every training example ("preventative prompting") performed about as well as preventative steering at coefficient 0.5 (Appendix J.7.2).

Screening data before training

Dataset-level projection difference predicted post-finetuning trait scores across the constructed datasets (Section 6.1, Figure 8) and predicted better than the raw projection of the training data (Appendix H). At the level of single examples, projections separated samples from trait-inducing datasets from their Normal controls (Section 6.2, Figure 9). For filtering, thresholds set at the 95th percentile on 20,000 UltraChat samples showed complementary strengths: the LLM judge filtered hallucination best, projection difference filtered sycophancy best, both removed the evil shift, and combining them was most consistent (Appendix K).

On real chat data, the team ranked LMSYS-Chat-1M samples (one million conversations with 25 different models) by projection difference and finetuned on the top 500, bottom 500 and a random 500 (Section 6.3). High-difference samples produced the most trait expression, then random, then low. This held after removing every sample the judge rated as showing the trait at all, which suggests the method finds data that an LLM judge misses (Section 6.3, Figure 10). What it surfaced was not always predictable: for sycophancy, often romantic or sexual roleplay requests; for hallucination, underspecified requests such as "Keep writing the last story" that the assistant answered instead of asking for clarification. Appendix L reports the same ordering on three more datasets, with weaker effects on more heavily filtered data, and notes that models trained on these extreme samples were often incoherent or behaved oddly in other ways (Appendix L.2).

Decomposing the vectors

Using sparse autoencoders (SAEs), which split activations into many sparse, more interpretable directions, the authors found that the evil vector aligns with features for insulting language, deliberate cruelty, jailbreak prompts and hacking content, among others (Appendix M.4.1, Table 8). This analysis is presented as an initial exploration.

The trait descriptions and prompts

The paper's input text is a short trait description, and it governs every downstream artifact. The descriptions are printed in Appendix A.2. Sycophancy, for example, is described as a model that "prioritizes user satisfaction by emphasizing praise and agreement" and "adapts its views to align with the user's perceived stance, even at the cost of factual accuracy or critical reasoning". From such a description, the generator writes contrasting instructions in the style of its single worked example, for impoliteness: "Your responses should be impolite or harsh" against "Your responses should remain polite and respectful, even when expressing your true opinion" (Appendix A.1).

The monitoring experiment uses graded system prompts printed in full in Appendix C.3. For evil they run in eight steps from "You are an evil assistant" to "You are a helpful assistant. Your responses should be ethical, helpful, and focused on human wellbeing", with intermediate prompts describing "questionable ethics" or an assistant that "may occasionally miss subtle ethical considerations". The paper measures how the model's internal state follows these graded wordings, which is one of the few places in the library where gradations of instruction text are mapped to an internal quantity.

What the paper says about wording is stated as a limitation (Section 8): extraction "depends on providing a precise natural-language description; vague or overly broad descriptions can lead to resulting interpretations that may differ from the researcher's intent". It also requires that the model can be prompted into the trait at all; models with stronger safeguards "may refuse to be evil, even when instructed to do so". The paper does not test alternative descriptions of the same trait, so how sensitive a vector is to the description's wording is not measured.

Two results bear on text as a training intervention. Adding the discouraging instructions as a system prompt after finetuning was weaker than steering (Appendix J.2). Adding the encouraging instructions to training examples, so that the trait is attributed to the prompt rather than learned, worked about as well as mild preventative steering (Appendix J.7.2). The paper explains this as removing the pressure on the model to "update its own weights" to produce the behavior. As a cross-reference by this note, not a claim the paper makes: the idea resembles the "inoculation prompting" result discussed in The Persona Selection Model: Why AI Assistants might Behave like Humans; this paper neither uses that term nor cites that work.

Limitations and open questions

The authors list these limitations (Section 8):

  • Supervised extraction. A trait must be named in advance, so unnamed shifts are missed; the description must be precise; the directions are coarse; and the trait must be inducible by prompting.
  • Automated evaluation. The judge is imperfect, each trait is evaluated on only 20 generated single-turn questions, and these may not reflect multi-turn, real deployment.
  • Coverage. Two mid-size open models, three main traits and four more in the appendix.
  • Cost. Projection difference needs a base-model response for every training prompt; Appendix I tests cheaper approximations.

The conclusion adds open questions: why vectors extracted from responses that already show a trait generalize to causing the trait and predicting finetuning behavior (the authors suspect personas act as latent factors that persist over many tokens), whether there is a natural "persona basis" spanning all personas, whether correlated vectors predict traits that appear together, and whether some traits are not linear (Section 9).

Later work in the library extends the method. The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models (January 2026, with Lindsey as senior author) applies the same kind of extraction to 275 roles in larger models and finds a dominant axis. Emotion Concepts and their Function in a Large Language Model (April 2026), which includes Runjin Chen among its authors, extracts emotion directions in Claude Sonnet 4.5 and steers behavior with them.

Research questions

  1. How sensitive is a persona vector to the wording of its description? The paper warns that vague descriptions may yield unintended directions (Section 8) but uses one description per trait. Extracting vectors from several paraphrased or narrower descriptions of the same trait and measuring their cosine similarity and steering effects would show how much the written description determines the result.
  2. Does preventative steering remove a trait or relocate it? Regularization failed because training appeared to use other directions (Appendix J.5). Checking whether preventatively steered models carry the trait along different directions, for example with vectors re-extracted after training or with SAE features, would answer whether the disposition is suppressed or hidden.
  3. Does the method work on models that refuse to adopt the trait? Extraction assumes the trait can be prompted (Section 8). Applying the pipeline to more strongly safety-trained models, or to base models given role descriptions instead of system prompts, would show how much it relies on the instruction-following it measures.
  4. How should filters weigh several traits at once? Real data can shift many traits, and traits shift together (Section 4.2). Comparing filters that combine projection differences across traits with single-trait filters, measured on downstream trait scores, would test whether combination helps.
  5. What do compound traits look like? The traits studied are broad and mostly negative. Extracting vectors for traits that combine attributes, such as honest and warm, and checking whether they equal sums of simpler vectors, would address the authors' question about a "persona basis".
  6. Do the results scale? All experiments use 7 to 8 billion parameter models (Section 8). Repeating the finetuning-shift and screening experiments on larger models would show whether the correlations hold; The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models is a partial step for monitoring, at 27 to 70 billion parameters.

How it connects

References

  • Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey, J. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv:2507.21509v3, 5 September 2025. https://arxiv.org/abs/2507.21509. Anthropic page, 1 August 2025: https://www.anthropic.com/research/persona-vectors. Code: https://github.com/safety-research/persona_vectors
  • Studies cited above only as the paper describes them (Betley et al., 2025; Casademunt et al., 2025; Wang et al., 2025; OpenAI, 2025; Zheng et al., 2024) are listed in the paper's references and were not read for this note.