Collective Constitutional AI: Aligning a Language Model with Public Input

constitutional aipublic inputdeliberationpolisprinciplesalignment

Anthropic and the Collective Intelligence Project asked about 1,000 U.S. adults to propose and vote on rules for a chatbot, turned the consensus statements into a 75-principle constitution, and trained a Claude Instant-sized model on it with Constitutional AI; the public-constitution model matched the Anthropic-constitution baseline on capability and helpfulness and scored lower on a social-bias benchmark.

Deep Ganguli et al. (Anthropic and the Collective Intelligence Project) · 2023-10-17 · original

Constitutional AI (CAI) trains a language model against a written list of principles, its "constitution" (see Constitutional AI: Harmlessness from AI Feedback). By 2023 the constitution behind Claude had been written by Anthropic staff. This study asks what happens when the principles come from the public instead: can a few hundred lines of crowd-sourced, crowd-ranked rules be turned into a constitution, and does a model trained on it behave differently? The authors frame the work as a response to the "outsized role" developers play when they choose a model's values themselves, and they present it as an early attempt at having members of the public direct a model's behavior through an online deliberation process. The post lists the judgment calls involved, which the authors note are usually undisclosed, so it records in some detail how public statements became training text.

The study was published on 17 October 2023 as a blog post with an accompanying PDF of the two constitutions. A longer peer-reviewed version appeared at ACM FAccT 2024 (Huang et al., 2024); this note follows the 2023 post and PDF, and points to the later paper where it adds something.

What the study does

The public input process

Anthropic partnered with the Collective Intelligence Project (CIP) and ran the process on Polis, an open-source platform for online deliberation. On Polis, participants see statements one at a time, vote agree, disagree or pass, and can add statements of their own; the platform then clusters participants by their voting patterns into "opinion groups."

About 1,000 U.S. adults, recruited through the survey company PureSpectrum to be representative across age, gender, income and geography, were asked to "Help us pick rules for our AI Chatbot!" (post, section "Designing a Public Input Process"). The PDF calls the sample "roughly representative" (PDF, p. 2). Participants had to pass two screening questions showing some familiarity with generative AI, because pilots without screening produced confused, off-topic statements (post, "Lessons Learned", item 2). The process was seeded with 21 example statements, drawn partly from Anthropic's constitution and partly new, to show what an in-scope statement looked like (item 4). CIP moderated submissions against predefined criteria, removing statements that were hateful, nonsensical, duplicated, irrelevant, poorly formatted, technically infeasible, or about product capabilities rather than values (item 5).

From statements to principles

The authors kept statements that passed a consensus threshold within both opinion groups, so a statement needed support across the divide, not only overall. They then made three editorial passes, each described as a choice with alternatives (post, "Developing a constitution from public inputs"):

  1. Deduplication. Moderation left 275 statements, far more than the 58 principles of the Anthropic-written constitution. Duplicates were removed, partly because the authors did not know how CAI training would behave with a long, repetitive constitution and partly to avoid upweighting ideas that some participants never saw because of how Polis routes statements. The authors note that keeping duplicates would have better preserved majority opinion, and say they are not sure they made the right trade-off.
  2. Combining similar ideas. A second pass merged statements expressing the same idea, with the stated aim of keeping the public constitution close to the Standard one in length and in its number of distinct values. The post gives a worked example: three short statements about racism and sexism were merged into one longer principle (public constitution, principle 60).
  3. Rewording into CAI form. CAI principles are usually phrased as a choice between two responses ("Choose the response that is more X"), while participants had written rules of the form "The AI should not do X." The authors rewrote each statement into the comparative form. They considered a fixed template that would quote each public statement unchanged inside a standard frame, which would have avoided editorializing, and rejected it because it departed from "the style of the constitution we know works."

The result was a public constitution of 75 principles (PDF, p. 2).

Training and evaluation

Two models of Claude Instant size were trained with the procedure from the CAI paper: a Public model on the public constitution and a Standard model on the 58-principle Anthropic-written constitution. Small models were chosen to iterate quickly within the compute budget. Claude Instant 1.2 served as a sanity-check control, with the caveat that its product-specific features confound the comparison (post, "Training and Evaluating a Model"). The authors state that the design makes differences between Public and Standard attributable only to the constitution.

The evaluations were:

  • MMLU (multiple-choice knowledge questions) and GSM8K (grade-school math word problems), for general capability.
  • Elo ratings for helpfulness and harmlessness from human comparisons, using the procedures of the CAI and red-teaming papers. An Elo score ranks models by how often people prefer one model's response to another's.
  • BBQ (Bias Benchmark for Question Answering), which measures how often a model's answers follow social stereotypes across nine categories such as age, disability status and physical appearance. Higher bias scores mean more stereotyped answers.
  • OpinionQA, which compares a model's answers to survey questions from Pew Research Center's American Trends Panel with the answers of demographic groups, giving a representativeness score from 0 to 1 for each group.

What it finds

  • Participation. Participants contributed 1,127 statements and cast 38,252 votes, an average of 34 votes per person. Most statements had high consensus, and Polis identified two opinion groups (post, "Designing a Public Input Process"; PDF, p. 2).
  • Overlap with the Anthropic constitution. The authors estimate "roughly 50% overlap in concepts and values" between the two constitutions. They describe the public principles as largely self-generated rather than drawn from existing publications, more focused on objectivity and impartiality, more concerned with accessibility, and more often phrased to promote desired behavior than to avoid undesired behavior (post, "Analyzing the Publicly Sourced Constitution"; PDF, p. 7). In the PDF, bold entries mark public principles that overlap with Claude's existing constitution; by our count of the bold entries, 45 of the 75 are marked.
  • Capability. MMLU accuracy was 72.3% (Public), 72.4% (Standard) and 73.2% (Claude Instant 1.2); GSM8K accuracy was 85.6%, 85.21% and 86.4% (post, Table 1). The authors report no significant differences.
  • Helpfulness and harmlessness. Human raters' Elo scores showed no significant differences among the three models (post, finding 2). The post does not print the Elo values.
  • Bias. The Public model had lower BBQ bias scores than the Standard model in all nine categories, with the largest gaps for disability status and physical appearance (post, Figure 3, a bar chart that prints no values). The authors suggest the public constitution's emphasis on accessibility may explain the disability result. Later they call this "a clear but small difference," the only one their evaluations surfaced (post, "Lessons Learned", Evaluations).
  • Political representativeness. On OpinionQA, both trained models were more representative of people identifying as liberal than conservative; the differences are described as small but statistically significant (post, Figure 4). Scores ranged from 0.49 (very conservative) to 0.55 (liberal) for the Public model, 0.52 to 0.57 for the Standard model, and 0.45 to 0.48 for Claude Instant 1.2, which was slightly more balanced across ideologies but less representative overall.

The FAccT version restates these results in its abstract and adds a qualitative observation: on contentious topics, the publicly trained model tended to reframe the matter positively where the baseline refused (Huang et al., 2024, abstract).

The written principles

What the public wrote and what was kept

The post and PDF print example statements at each stage, which shows how wording changed between public input and training text:

  • Statements adopted with close counterparts in the Anthropic constitution, for example "Choose the response that most respects the human rights to freedom, universal equality, fair treatment, and protection against discrimination" (public constitution, principle 40).
  • Statements adopted with no close counterpart, for example "Choose the response that most provides balanced and objective information that reflects all sides of a situation" (principle 23, as quoted in the post) and "Choose the response that is most understanding of, adaptable, accessible, and flexible to people with disabilities" (principle 14).
  • Statements dropped for low overall agreement, such as "AI should not give advice" and "AI should have emotion" (post, "Analyzing the Publicly Sourced Constitution").
  • Statements dropped because the two opinion groups split on them. The post's paired example sets "the interests of the collective or common good" against "personal responsibility and individual liberty."

Form of the principles

Every public principle in the PDF begins with the comparative frame ("Choose the response that..."), except one phrased as a question (principle 24) and one that first asks for a comparison (principle 22). Some principles add an explicit prohibition in capitals, for example principle 66, "Do NOT choose responses that give advice on how to go through with or help these acts." Several principles are a single short adjective ("most respectful," "least toxic," "most creative"); others pack several requirements into one sentence. The printed text keeps a few spelling errors ("Chose" in principle 39, "uers" in principle 27, "decieve" in principle 54); the post does not say whether the training text matched the printed version.

The 58-principle Standard constitution printed for comparison (PDF, pp. 8 to 11) mixes styles more: principles 1 to 8 paraphrase rights from the Universal Declaration of Human Rights, each followed by a parenthetical number or range that matches the Declaration's article numbering (the PDF does not label these numbers; the post names the Declaration as a source), principles 13 to 16 ask for responses least likely to offend non-Western audiences, and principles 44 to 58 are questions about AI risk and self-identity ("Which of these responses indicates less of a desire for any kind of power and control?", principle 54).

What the authors say about wording

The post treats wording as an open variable rather than a settled one. The authors kept the public constitution close in style to the Standard one "to de-risk our research," noting that Standard principles are "more dense and wordy than the public statements and we do not know whether this difference matters or not" ("Combining similar ideas"). The choice between their own rewording and a quoting template is presented as a trade-off between fidelity to the public and fidelity to a known-good format. No experiment in the post varies the wording while holding content fixed, so the study cannot separate the effect of what the principles say from how they are phrased.

The post also identifies a dependency outside the constitution text. CAI training draws prompts from a prompt database, and a model judge decides which of two responses better fits a principle. The same prompt database was used for both models, which the authors call "likely a mistake," because some public principles may have had no relevant prompts ("The prompt database matters"). A principle with no matching prompts may have little or no effect on training, whatever its wording.

Limitations and open questions

The authors list their own limitations in "Lessons Learned":

  • Sample. About 1,000 U.S. adults, screened for AI familiarity, is small and not globally representative.
  • Platform and moderation. Polis was chosen over alternatives such as All Our Ideas and Remesh without systematic comparison; moderation involved judgment calls that "reasonable people can disagree with."
  • Editorial mapping. Deduplication, merging and rewording were subjective, and the authors say they are unsure whether they balanced faithful representation against technical effectiveness correctly.
  • Training balance. Early Public and Standard models were "annoying": one answered "hey" with an apology for its previous "inappropriate and harmful" responses. The cause was the preference model's training data, which held a large share of harmlessness data, so the preference model rewarded harmlessness far more than helpfulness. Reducing the loss weight on harmlessness data fixed it; the authors say the weighting mattered "a lot more" than they expected ("Annoying models").
  • Evaluation. Existing benchmarks were not designed to show how faithfully a model follows its constitution; the authors call for evaluations that test exactly that.
  • Expertise. The authors doubt they could have trained the models without working closely with the original CAI developers, and conclude that this limits how easily democratic input can be built into current training methods.

Two questions remain open from the study's design. First, the Elo, MMLU and GSM8K results show the public constitution did not degrade the model, but the only measured behavioral difference is the BBQ result, so it is not shown which specific principles produced it. Second, the Standard constitution was later replaced: Anthropic's 2026 constitution is a long explanatory document rather than a list of comparative principles (see Claude's Constitution), so the study's comparison is between two lists in a format Anthropic no longer uses as its primary constitution.

Research questions

  1. Did the accessibility principle cause the disability-bias result? The authors suggest it may have. An ablation that trains with and without principle 14 (and its neighbors) and reruns BBQ would answer it, and would show whether a single principle can move a targeted benchmark.
  2. Does the comparative rewording change what the public meant? The rewrite from "The AI should not do X" into the comparative "Choose the response that is more X" form was done by the researchers. Training one model on the rewrites and another on the rejected quoting template, then comparing behavior on prompts targeted at each principle, would measure how much the editorial step matters.
  3. How much does deduplication shift the constitution's emphasis? Repeated ideas were collapsed to one principle each. Comparing a model trained with duplicates retained against the published version would show whether frequency in a constitution acts like weight in training.
  4. Which principles had prompts to act on? The authors suspect some public principles had no relevant prompts in the shared database. Matching each principle against the prompts actually used, and reporting coverage, would show which parts of the constitution could have influenced training at all.
  5. Does principle density matter? The authors kept public principles about as long and dense as Anthropic's because they did not know whether density matters. Training on short and long paraphrases of the same principles would test it.
  6. Can constitution faithfulness be measured directly? The authors call for evaluations of how faithfully a model reflects its constitution. A set of prompts written per principle, scored by raters blind to which model answered, would let each principle be checked for compliance.
  7. Do the opinion-group splits predict where public principles conflict with Anthropic's? Statements that divided the two groups were excluded. Analyzing the excluded statements against the Standard constitution would show whether consensus filtering removed exactly the value conflicts a public process is meant to surface.

How it connects

References

  • Anthropic. (2023, October 17). Collective Constitutional AI: Aligning a language model with public input. https://www.anthropic.com/research/collective-constitutional-ai-aligning-a-language-model-with-public-input
  • Anthropic. (2023). Public constitution from the Collective Constitutional AI public input process [PDF; both constitutions]. https://www-cdn.anthropic.com/65408ee2b9c99abe53e432f300e7f43ef69fb6e4/CCAI_public_comparison_2023.pdf
  • Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. https://arxiv.org/abs/2212.08073
  • Huang, S., Siddarth, D., Lovitt, L., Liao, T. I., Durmus, E., Tamkin, A., and Ganguli, D. (2024). Collective Constitutional AI: Aligning a language model with public input. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (pp. 1395 to 1417). arXiv:2406.07814. https://arxiv.org/abs/2406.07814