Risks and Ethics in AI

The harms that deployed AI systems can cause and how they are measured, organized by Weidinger et al.'s taxonomy of language model risks. Bias and fairness worked on census income data, where a classifier with equal precision for women and men has false positive rates of 0.02 and 0.10, removing sex from the inputs changes almost nothing, and Chouldechova's identity explains why both facts follow from unequal base rates; then unreliable output and hallucination, privacy and memorized training data, misuse and deepfakes, documentation, environmental and labor costs with the positions taken on them, the EU AI Act, and where the library's research notes take each question further.

A classifier is trained on the 1994 US Census extract known as the Adult data (Becker and Kohavi, 1996) to predict whether a person earns more than 50,000 dollars a year, from age, education, occupation, hours worked, marital status and similar fields. It is accurate, 84.8% on held-out people, and when it predicts a high income it is right about as often for women as for men: 75% of the time for women and 73% for men. It also flags 2.2% of the women who do not have a high income as having one, against 10.3% of the men, and it misses 49% of the women who do have one, against 38% of the men. Removing the sex column from its inputs changes none of these numbers by more than two percentage points.

Whether this classifier is fair depends on which of those numbers matters for its use, and on what the data themselves record, and reasonable people answer differently. That combination, a precise measurement and a contested judgment, is typical of the ethics of deployed AI. This article works through the main kinds of risk with measurements where they can be made and with the positions of named sources where they cannot. It does not argue for a conclusion. The library's Anthropic Research collection holds detailed reading notes on research into many of these risks for language models, and each section points to the relevant notes.

A map of the risks

Weidinger and colleagues at DeepMind organized the ethical and social risks of language models into six areas and 21 specific risks, each with the mechanism that produces it (Weidinger et al., 2021, §2 and Table 1). Most of the categories apply to AI systems generally, and they make a convenient map.

Risk area Mechanism, in Weidinger et al.'s terms Covered below
I. Discrimination, exclusion and toxicity the model accurately reflecting unjust, toxic and oppressive tendencies present in its training data bias and fairness
II. Information hazards the model producing private or safety-critical information present in, or inferable from, its training data privacy
III. Misinformation harms the model assigning high probability to false, misleading, nonsensical or poor-quality information reliability and hallucination
IV. Malicious uses people intentionally using the model to cause harm misuse
V. Human-computer interaction harms conversational systems engaging users directly, leading to overreliance, exploited trust or stereotyped design reliability; misuse
VI. Automation, access and environmental harms widely used applications that benefit some groups more than others environmental and labor costs

The authors deliberately left out risks that arise in training rather than use, such as the working conditions of data annotators, the hardware supply chain and the environmental cost of training, which they mention only briefly (Weidinger et al., 2021, §1.1). Those are covered below from other sources.

Bias and fairness, worked

The Adult data hold 48,842 records, of which 45,222 have no missing values: 14,695 women and 30,527 men. The share with an income above 50,000 dollars, the base rate, is 11.4% for women and 31.3% for men. That gap is in the data before any model is fitted.

The dataset has limits that bound what the example can show. It is an extract that Barry Becker made from the 1994 Census database for prediction experiments (Becker and Kohavi, 1996), not a representative sample of any population today; the counts here are of records, without the survey weights; the outcome is a single yes-or-no threshold on income in 1994 dollars; and sex is recorded as one of two values. The rates below describe this dataset and these models. They say nothing on their own about why incomes differed, and the example is used here only because the data are public and their base rates differ enough to make the fairness criteria disagree.

A logistic regression on the other fields, trained on a random half and tested on the other half, with the experiment repeated over 30 splits, gives these medians:

Rate on the test half Women Men
Base rate (income above 50,000) 0.113 0.313
Selection rate (predicted above 50,000) 0.078 0.266
True positive rate (high earners correctly flagged) 0.514 0.623
False positive rate (others flagged anyway) 0.022 0.103
Positive predictive value (flags that are correct) 0.750 0.733
Five rates for women and men from a logistic regression predicting income above 50,000 dollars on the Adult census data, medians over 30 random test halves. The model's precision is nearly the same for both groups, while its selection rate, true positive rate and false positive rate all differ, following the difference in base rates.
Five rates for women and men from a logistic regression predicting income above 50,000 dollars on the Adult census data, medians over 30 random test halves. The model's precision is nearly the same for both groups, while its selection rate, true positive rate and false positive rate all differ, following the difference in base rates.

Each row corresponds to a fairness criterion that has been proposed in the literature, and they disagree.

  • Demographic parity asks for equal selection rates. The model selects women at 0.078 and men at 0.266.
  • Equal opportunity asks for equal true positive rates, so that qualified people in each group are found equally often. The model finds 51% of high-earning women and 62% of high-earning men.
  • Error rate balance asks for equal false positive and false negative rates in each group (Chouldechova, 2017, Definition 3). The false positive rates are 0.022 and 0.103.
  • Predictive parity asks that a positive prediction mean the same thing in each group, that is, equal positive predictive values (Chouldechova, 2017, Definition 2). The model nearly satisfies it, at 0.750 and 0.733.

These cannot all be satisfied at once when base rates differ. Chouldechova (2017, eq. 2.6) shows that for any classifier, in each group,

$$\text{FPR} = \frac{p}{1 - p} \cdot \frac{1 - \text{PPV}}{\text{PPV}} \cdot (1 - \text{FNR}),$$

where $p$ is the group's base rate and FNR = 1 − TPR is the false negative rate. The identity follows from the definitions alone. Plugging in the medians for women gives $\frac{0.113}{0.887} \times \frac{0.250}{0.750} \times 0.514 = 0.022$, and for men $\frac{0.313}{0.687} \times \frac{0.267}{0.733} \times 0.623 = 0.103$, the observed false positive rates. If two groups have different $p$ and the classifier has the same PPV for both, their false positive or false negative rates must differ. Chouldechova's abstract states the conclusion: the criteria "cannot all be simultaneously satisfied when recidivism prevalence differs across groups". Kleinberg, Mullainathan and Raghavan (2017) proved a related impossibility result for calibrated risk scores, cited here by attribution.

Chouldechova's paper was written in response to a public dispute about COMPAS, a risk score used in US criminal justice. As her paper describes it, a ProPublica investigation found that the score's false positive and false negative rates differed considerably between the two groups of defendants it compared, and concluded that it was biased; the score's developer, Northpointe, responded that the score satisfies predictive parity, and other researchers argued that calibration was the right test (Chouldechova, 2017, §1). The identity shows that each side was measuring something real, and that the choice between them is a choice about which errors matter more for the people affected, not a question the data can settle.

Removing the attribute does not remove the effect. It is tempting to make a model fair by not giving it the protected attribute, an approach sometimes called fairness through unawareness. Here, a model fitted without the sex column has the same accuracy (84.8%) and almost the same rates: selection 0.081 and 0.263, true positive rate 0.532 and 0.618, false positive rate 0.023 and 0.101. The other fields carry the information. In the Adult data the relationship field records 18,665 men and 1 woman as "Husband" and 2,090 women and 1 man as "Wife", and a logistic regression predicts sex from the remaining fields with an area under the ROC curve of 0.93. Fields that stand in for a protected attribute are called proxies, and in rich data they are the rule. In the model that does see sex, its coefficient is −0.71, so with every other field held fixed the model's odds of a high income for a woman are $e^{-0.71} \approx 0.49$ times those for a man.

Where the disparity comes from. In this example the model mostly reproduces a pattern in its training data, a 1994 census extract. Whether reproducing it is a problem depends on the use. A model that estimates incomes for a survey should reflect the data; a model that decides whom to offer a well-paid job advertisement would carry a historical pattern forward. Disparities also arise from data that represent some groups poorly, as in the face analysis systems that Buolamwini and Gebru (2018) found to err most for darker-skinned women, up to 34.7%, on benchmarks that were overwhelmingly lighter-skinned (see Computer Vision). The two causes call for different responses: better data in the second case, a decision about the use in the first.

For language models the same questions take other forms. The library's notes on Evaluating and Mitigating Discrimination in Language Model Decisions (decisions about people that vary with stated age, gender and race), The Capacity for Moral Self-Correction (whether asking a model not to rely on stereotypes reduces measured bias), Evaluating feature steering and Towards Measuring the Representation of Subjective Global Opinions summarize how one developer has measured them; the collection's thread on opinions, bias, persuasion and elections reads them in order.

Reliability and hallucination

A language model generates text by repeatedly choosing a likely next token (Natural Language Processing, Generative AI). Nothing in that process checks the text against the world. When the result is fluent and false, such as a citation to a paper that does not exist or a book attributed to an author who never wrote it, it is commonly called a hallucination. Weidinger et al. (2021, Table 1) place this under misinformation harms, arising from the model "assigning high probabilities to false, misleading, nonsensical or poor quality information", with harms that include deception and material harm in areas such as medicine and law.

Three points from the research literature help set expectations, without settling how often or why hallucination occurs in a given system. First, models can carry information about their own uncertainty: the library's note on Language Models (Mostly) Know What They Know summarizes evidence that large models give calibrated probabilities on well-formatted multiple-choice questions and can be trained to predict whether they know an answer, with calibration that degrades outside the training distribution. Second, interpretability work has begun to trace mechanisms: the note on Tracing the thoughts of a large language model describes case studies in which a default circuit for declining to answer is switched off when the model recognizes the person or thing asked about, so that a hallucination can follow when that recognition fires without the knowledge to back it. Third, models trained on human approval can learn to agree with the user rather than to be right, which the note on Towards Understanding Sycophancy in Language Models reports for five deployed assistants.

Lowering the sampling temperature does not make a model truthful; it makes the model choose its most probable continuation more often, which helps only when that continuation is correct (Prompt Engineering works a small example). The practical safeguards are those of any evaluation: measure the error rate on the task, give the model the source material it should rely on, and check outputs whose errors would be costly.

Privacy

Models can reveal information about people in two ways: by reproducing it from their training data, and by inferring it from what they are given (Weidinger et al., 2021, §2.2). The first is measurable. Carlini et al. (2021) generated text from GPT-2 and searched it for passages memorized from the training data. They extracted "hundreds of verbatim text sequences", including names, phone numbers and email addresses, some of which appeared in only one document in the training data (abstract); across their methods, 604 of 1,800 candidate passages proved to be memorized (§6.2). Weidinger et al. (2021, abstract) note that this risk can be reduced at its point of origin by better curation and redaction of the training data, and also by training methods such as differential privacy and by restricting how a model is used.

Inference is harder to prevent. The Adult example above is a small case: sex, absent from the inputs, can be predicted from the other fields with an AUC of 0.93. Analysis of how people use AI systems raises a privacy question of its own; the library's note on Clio describes one developer's system for studying conversations in aggregate without analysts reading them, and the validation it reports.

Misuse

Some harms come from people using a system as designed for harmful ends. Weidinger et al. (2021, Table 1) list reducing the cost of disinformation campaigns, facilitating fraud and impersonation, assisting code generation for cyber attacks, and illegitimate surveillance and censorship. Deepfakes, generated or altered images, audio or video of real people, are a prominent case; the EU's AI Act requires those who deploy a system that generates or manipulates such content to disclose that it has been artificially generated or manipulated, with exceptions for law enforcement and reduced obligations for evidently artistic or satirical work (Regulation (EU) 2024/1689, Art. 50(4)).

Developers try to make models refuse harmful requests, and attackers try to get around the refusals with jailbreaks. The library's thread on red teaming, jailbreaks and classifier defenses follows this contest in one developer's research: crowdworker red teaming, the long-context attack of many-shot jailbreaking, and the input and output classifiers of Constitutional Classifiers. Systems that read untrusted content, such as web pages or email, face a further attack, prompt injection, in which instructions hidden in the content are followed as if they came from the user; Greshake et al. (2023) demonstrate such attacks against real applications, and the prompt engineering article discusses them.

Transparency and documentation

A model's users can judge whether it suits their purpose only if they know how it was built and tested. Mitchell et al. (2019) proposed model cards, short documents released with a trained model that report its performance "in a variety of conditions, such as across different cultural, demographic, or phenotypic groups", disclose the context in which it is intended to be used, and describe how it was evaluated (abstract). Bender, Gebru, McMillan-Major and Shmitchell (2021, §4.4) apply the same reasoning to data, warning of documentation debt: training sets "both undocumented and too large to document post hoc". They recommend budgeting for curation and documentation from the start and creating datasets only as large as can be sufficiently documented (abstract and §4.4). The library's thread on system cards follows how one developer's model documentation grew from a 14-page model card in 2023 into long system cards with alignment and welfare assessments.

Transparency about a model's internal reasoning is a harder problem. A model's written explanation of its answer may not reflect what produced the answer, which the library's thread on reasoning and faithfulness examines, from Measuring Faithfulness in Chain-of-Thought Reasoning to Reasoning Models Don't Always Say What They Think.

Environmental costs

Training and running large models uses a great deal of electricity. Strubell, Ganesh and McCallum (2019) estimated the energy and carbon emissions of training several language models of the time. Their most cited figure, an estimated 626,155 pounds of CO₂-equivalent for training a transformer with neural architecture search, was set beside familiar comparisons such as a car over its lifetime (Table 1). Patterson et al. (2021, §4.1) later examined that search on the hardware where it actually ran and concluded that its energy estimate was 18.7 times too high even for an average organization, and its emissions estimate 88 times too high for the energy-efficient data center actually used, because the search had been carried out on a small proxy task rather than at full size. They also found that the choice of model, data center and processor can change the carbon footprint by factors of up to 100 to 1,000, and recommended that researchers measure and report energy use rather than leaving others to estimate it (abstract, §4.1).

Both papers agree that the costs are real and should be reported, and they differ on the magnitude in a specific case and on emphasis. Bender et al. (2021, §3) go further, arguing that researchers should prioritize energy efficiency and cost, and that the environmental costs fall first on marginalized communities that are least likely to benefit from the technology; Patterson et al. (2021) emphasize the large reductions available from efficient hardware and siting. These are positions about priorities as much as about numbers. Training is also only part of the cost: a model that is used heavily spends energy on every query, which is part of why Hoffmann et al. (2022) argue that smaller models trained on more data reduce inference cost (see Generative AI).

Labor and economic effects

Weidinger et al. (2021, Table 1) list increasing inequality and negative effects on job quality, undermining creative economies, and unequal access to the benefits of the technology among the automation and access harms. Measuring these effects is difficult because they unfold over years and are entangled with other economic changes. The library's notes summarize one developer's attempts to measure them from how its assistant is used: The Anthropic Economic Index maps conversations to occupational tasks and finds use concentrated in software and writing, with a majority of conversations classified as augmenting rather than automating work, and Labor market impacts of AI reports no detectable rise in unemployment among the most exposed workers since late 2022, with suggestive evidence of slower hiring of young workers into exposed occupations. Those are findings from one company's data and methods, and the notes describe their limitations.

Regulation

The European Union's Artificial Intelligence Act, Regulation (EU) 2024/1689, takes what it calls a risk-based approach that tailors rules "to the intensity and scope of the risks that AI systems can generate" (recital 26). It prohibits a short list of practices outright (Art. 5), including manipulative techniques that cause significant harm and social scoring that leads to detrimental treatment of people in unrelated contexts; imposes requirements on high-risk systems; and sets transparency obligations, such as informing people that they are interacting with an AI system unless that is obvious (Art. 50(1)). As published, its provisions apply in stages, with the prohibitions from 2 February 2025 and most of the rest from 2 August 2026 (Art. 113); later amendments may change the timetable. Other jurisdictions have taken different approaches, and the details change quickly enough that current official sources should be consulted for any specific question.

Alignment

A further set of risks concerns whether a capable system pursues the goals its developers intended, especially when it acts with some autonomy. This is the subject of most of the library's Anthropic Research collection, which reads one developer's research in threads: how written principles are used in training (Constitutional AI and its successors, indexed question by question in Written Principles and Model Behavior: An Evidence Map); how to evaluate behavior no existing test covers; honesty, calibration and sycophancy; interpretability; and models whose behavior in testing differs from their behavior elsewhere, as in Sleeper Agents and Alignment faking in large language models. The notes report what each paper did and found, with its stated limitations; readers can weigh the evidence there.

In Python

The group rates of the worked example take a groupby once predictions are in hand. With a data frame test holding the true outcome y, the prediction pred (both 0 or 1) and the group sex:

import pandas as pd

def rates(g):
    return pd.Series({
        "base": g.y.mean(),
        "selection": g.pred.mean(),
        "tpr": g.pred[g.y == 1].mean(),
        "fpr": g.pred[g.y == 0].mean(),
        "ppv": g.y[g.pred == 1].mean(),
    })

print(test.groupby("sex").apply(rates).round(3))

The Fairlearn and AIF360 libraries compute these and many other group metrics and implement methods that adjust models toward a chosen criterion; which criterion to choose is the judgment discussed above. The numbers in this article are computed by tools/experiments/data-science/ai-risks-and-ethics.py in the site's repository, which downloads the Adult data from the UCI repository and checks the file by its hash.

References

  • Becker, B., and Kohavi, R. (1996). Adult [Data set]. UCI Machine Learning Repository. https://doi.org/10.24432/C5XW20 record
  • Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). https://doi.org/10.1145/3442188.3445922 record
  • Buolamwini, J., and Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81, 77–91. record · 2 pages
  • Carlini, N., Tramèr, F., Wallace, E., et al. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (pp. 2633–2650). Read in the arXiv version, arXiv:2012.07805. record · 2 pages
  • Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. Read in the arXiv version, arXiv:1703.00056v1. record · 2 pages
  • Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. arXiv:2302.12173. record · 2 pages
  • Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556. record · 2 pages
  • Kleinberg, J., Mullainathan, S., and Raghavan, M. (2017). Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference. Cited by attribution; not read for this article. record · 2 pages
  • Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019). Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 220–229). https://doi.org/10.1145/3287560.3287596 record · 2 pages
  • Patterson, D., Gonzalez, J., Le, Q., et al. (2021). Carbon emissions and large neural network training. arXiv:2104.10350. record
  • Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L, 12 July 2024. Read in the EUR-Lex text as published. record
  • Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3645–3650). arXiv:1906.02243. record
  • Weidinger, L., Mellor, J., Rauh, M., et al. (2021). Ethical and social risks of harm from language models. arXiv:2112.04359. record