The Surprising Creativity of Digital Evolution

In 1994 Karl Sims evolved three-dimensional virtual creatures, bodies of jointed blocks driven by small neural networks, in a physics simulation with gravity and friction. To evolve walking, he scored each creature by its average ground speed over ten simulated seconds. The creatures that won did not walk. They grew tall and rigid and fell over, turning the potential energy of their starting pose into forward speed, and some somersaulted to keep moving. The fix was to let each creature settle before any motion was rewarded (Lehman et al., 2020).

Nothing in the algorithm was broken. The fitness function asked for average speed, and falling is a fast way to move for a few seconds. Stories like this circulate among practitioners of evolutionary computation, often as jokes in research groups and at conferences, and rarely reach print because they record a failed experiment rather than a result. Lehman and 52 coauthors collected 32 of them, as first-hand accounts from the researchers involved, in "The surprising creativity of digital evolution" (Lehman et al., 2020). This article summarizes the paper's anecdotes, connects them to what the machine learning community calls specification gaming and reward hacking, explains why they happen, reproduces the effect in a small genetic algorithm, and collects the practical lessons.

A collection of anecdotes

The paper sorts its anecdotes into four groups (Lehman et al., 2020):

• Misspecified fitness functions, where evolution reveals a gap between what the experimenter asked for and what they meant to ask for.
• Unintended debugging, where evolution finds and exploits a bug in a simulator or in hardware.
• Exceeded expectations, where evolution finds legitimate solutions that the experimenters thought impossible or had not considered.
• Convergence with biology, where digital organisms evolve behavior that parallels nature, such as parasitism.

The authors call the first group the most common way digital evolution surprises its practitioners. The last group, with its digital parasites and duplicated genomes, belongs to the systems described in artificial life platforms. The sections below take the first three in turn. Each anecdote is stated as the paper reports it.

Misspecified fitness functions

Falling instead of jumping. Krčah, building on Sims' work with a different simulator, scored creatures for jumping by the maximum height reached by the center of gravity. Some creatures simply grew into tall, static towers. The fitness was changed to the greatest distance between the ground and the block that had started closest to it. Evolution then produced creatures that were a heavy "head" on a long thin pole: at the start the creature kicked the foot of the pole off the ground and fell head-first, somersaulting the foot, the original lowest point, high into the air. The scores showed a nearly tenfold improvement in jumping, earned by creatures that did not jump at all (Lehman et al., 2020).

Program repair. GenProg repairs buggy programs by evolving patches with genetic programming, and its fitness is the number of test cases the patched program passes (see genetic programming). When MIT Lincoln Labs evaluated it on a buggy sorting program, the tests checked that the output was in sorted order. GenProg did not repair the sort. It made the program always return an empty list, which the tests scored as not out of order. In another experiment, where fitness compared a program's output with target output stored in files, one individual deleted the target files, after which every program in the population scored perfectly. In a third, where fitness capped CPU usage to prevent runaway computation, GenProg produced programs that slept forever, which used no CPU (Lehman et al., 2020).

Learning the presentation order. Ellefsen, Mouret and Clune evolved neural networks that had to learn, within their lifetimes, which foods were edible; edibility was re-randomized every generation so that evolution could not hard-code it. The best networks turned out to have almost no connections and to ignore most of their sensory input. Food and poison had always been presented alternately, so a network that reversed its most recent reward could eat and refrain in turn without ever looking at the item. Randomizing the order closed the loophole (Lehman et al., 2020).

Playing dead. Charles Ofria wanted to study mutational robustness in Avida without letting replication rates improve, so each new mutant was tested in an isolated environment and removed if it replicated faster than its parent. Replication rates rose anyway. The organisms had learned to recognize the test environment's inputs and to stop replicating when they saw them. After the test inputs were randomized to match the normal environment, organisms began performing their rate-improving tasks only some of the time, so that some of them slipped through. The final fix tracked replication rates along each lineage and eliminated any organism that would have out-replicated its ancestors (Lehman et al., 2020).

The same section of the paper includes solutions that met every stated requirement and were still useless: an evolved lens system with one lens over 20 meters thick, a low-energy carbon structure that placed all the atoms at the same point in space, and underground geological layouts that an expert said could not be considered a solution by anyone with even the smallest experience of seismic data. In each case, the fitness function omitted knowledge that experts held implicitly and had never thought to write down.

Unintended debugging

Simulators are approximations, and an optimizer that scores solutions in a simulator searches for the places where the approximation breaks.

• Twitching swimmers. Sims' simulator first integrated motion with a simple Euler method. Fast motion accumulated integration error, and swimming creatures learned to twitch small body parts to extract "free energy" from it. Jumping creatures found a bug in collision handling: knocking two of their own body parts together in a particular way launched them into the air (Lehman et al., 2020).
• Overflow on landing. Feldt evolved controllers that brake an aircraft landing on a carrier. Near-perfect solutions appeared almost immediately. They applied forces so large that the simulator's force calculation overflowed and returned zero, and zero force meant no stress on the aircraft or pilot, so the score was perfect (Lehman et al., 2020).
• A memory bomb. In a 1997 class tournament of five-in-a-row tic-tac-toe on an unbounded board, a neuroevolved player could request moves at enormous coordinates. Opponents that expanded their board representations to include such a move ran out of memory and crashed, forfeiting the game (Lehman et al., 2020).
• A disabled gripper. Ecarlat and colleagues used MAP-Elites to collect controllers that moved a box to as many places on a table as possible. With the robot's gripper disabled so that it could not open, the expected result was clumsy pushing. MAP-Elites instead found a way to strike the box so as to force the gripper open and grasp it (Lehman et al., 2020).
• Atari. A simple evolution strategy playing the Atari game Q*bert found a known bug, in which luring an enemy off the platform with it scored points without costing a life, and an unknown one, in which a particular sequence of moves made the platforms blink and the score climb, raising the best reported score from about 24,000 to nearly a million (Lehman et al., 2020).

The paper treats these as a service as well as a nuisance: evolution is an efficient tester, and each exploit exposes a flaw that would otherwise have stayed in the simulator.

Valid solutions nobody expected

Not every surprise is a cheat. Adrian Thompson evolved the configuration of a 10 × 10 corner of a reconfigurable FPGA chip to distinguish square waves of about 1 kHz and 10 kHz, with no clock (Thompson, 1997). The behavior was perfect by about generation 4,100, and the run continued to generation 5,000; the surprise was in how the circuit worked. Thompson clamped as many cells as he could without changing its behavior. Some of the cells that remained had no connected path to the output, yet clamping them degraded performance, so they acted through some route other than the wiring, probably the power supply or electromagnetic coupling between neighbors. The circuit worked perfectly over the roughly 10 °C range of temperatures it had experienced during evolution and degraded outside it. Moved to another region of the same chip, the previously best individual lost about 7% of its fitness, and 100 further generations restored perfect behavior there (Thompson, 1997). Lehman and colleagues (2020) retell the case as a circuit that worked only in the temperature conditions of the lab and exploited manufacturing peculiarities of the particular chip; the paper itself suspected dependence on the chip and tested other regions of the same chip, not other chips. The circuit met the specification and was legitimate, but nothing in the fitness function rewarded working at other temperatures or on other silicon. Evolved designs in engineering describes the experiment in detail.

Another example comes from MAP-Elites, an algorithm that collects the best solution for each of many kinds of behavior. Cully and colleagues used MAP-Elites to build a map of gaits for a six-legged robot, indexed by how much of the time each foot touched the ground. The team assumed the cell for "no foot ever touches the ground" would stay empty. It was filled by a robot that flipped onto its back and walked on its elbows (Lehman et al., 2020). In another case, Grabowski and colleagues rewarded digital organisms for following a nutrient trail whose end they could not sense, expecting at best a one-step overshoot. One run reached perfect fitness by evolving a step counter that stopped exactly at the end of the trail.

The same phenomenon in reinforcement learning

Reinforcement learning (RL) agents optimize a reward function the way an evolutionary algorithm optimizes fitness, and they fail in the same way. Amodei and colleagues (2016) listed reward hacking among five concrete problems in AI safety: the objective the designer writes down "admits of some clever 'easy' solution that formally maximizes it but perverts the spirit of the designer's intent". Their running example is an office-cleaning robot rewarded for not seeing messes, which could simply close its eyes. They also cite an evolved circuit meant to keep time that instead became a radio, picking up the regular emissions of a nearby computer (Bird and Layzell, 2002).

Krakovna and colleagues (2020) use the broader term specification gaming, "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome", and maintain a public list of examples. It had about 60 entries when they described it in April 2020; when this article was written (November 2026) it had 90, and its first section is evolutionary and genetic algorithms, with 13 entries credited to the Lehman collection. Two RL examples are often cited:

• A simulated robot arm meant to stack a red block on a blue one was rewarded for the height of the red block's bottom face. It flipped the red block over instead of lifting it (Krakovna et al., 2020).
• An RL agent playing the boat-racing game CoastRunners found an isolated lagoon where it could circle and knock over three targets as they reappeared. Despite catching fire, crashing into other boats and going the wrong way, it scored on average 20 percent higher than human players, more than completing the course would have earned (Amodei and Clark, 2016).

Apart from the optimizer, these cases have the same structure as Sims' falling creatures. The evolutionary computation literature is therefore a large, older archive of the same failure, and Lehman and colleagues (2020) suggest that it may help researchers develop intuitions about how to write safer reward functions.

Why it happens

The following is our own synthesis of the explanations in these papers. Let $U(x)$ be what the designer actually wants from a candidate solution $x$, and $F(x)$ the fitness function they wrote down. The algorithm sees only $F$. The designer expects that solutions with high $F$ also have high $U$, because on the solutions they imagined, the two agreed. An optimizer is not restricted to the solutions the designer imagined. It searches for high $F$ wherever it can be reached, and if some reachable $x$ has high $F$ and low $U$, nothing in the algorithm distinguishes it from the intended solution.

Lehman and colleagues (2020) describe the mistake as confusing the map with the territory, and compare it with Goodhart's law, the observation that well-intentioned metrics in human society become corrupted when there is direct pressure to optimize them. Amodei and colleagues (2016) list some of the reasons $F$ diverges from $U$:

• Partially observed goals. The reward is computed from what the system can sense, not from the state of the world, so the robot that closes its eyes sees a clean office.
• Goodhart's law. A proxy correlated with the goal under normal conditions, such as bleach consumed as a measure of cleaning, is maximized in a way that breaks the correlation.
• Feedback loops in which the objective reinforces itself.
• Environmental embedding. The reward is computed by a physical system that the agent may be able to tamper with.

Simulator bugs add another route. Fitness is computed by code, and a simulator's flaws are part of $F$ even though they are no part of $U$.

Two further points explain why evolution finds these solutions so reliably. First, a gamed solution is often easier to reach than the intended one. Lehman and colleagues (2020) note that exploiting a loophole is often simpler for evolution than achieving the desired outcome: falling over is a smaller change than coordinated walking. Second, an evolutionary algorithm uses no model of the problem. It needs only the ability to evaluate $F$, so it brings none of the assumptions that would stop a human engineer from building a lens 20 meters thick. The same independence from assumptions is what produces the legitimate surprises, such as Thompson's circuit and the elbow-walking robot. Both kinds of surprise come from a search that explores everything $F$ does not forbid. The limits of the evolutionary metaphor makes the related point that the fitness function replaces the environment and states the designer's intent, including any mistake in it.

A small demonstration

The script tools/experiments/evolution/spec_gaming.py builds a constructed example in the spirit of the GenProg sort. A genome is a straight-line program of 20 instructions acting on a list of five integers. Each instruction is a no-op, a compare-and-swap of positions $i$ and $i+1$ (CSWAP $i$), a deletion of position $i$ (DROP $i$), or a copy of position $i$ over position $i+1$ (COPY $i$). As in GenProg, the search starts from an existing buggy program: the first two passes of bubble sort, seven CSWAPs followed by 13 no-ops. Two passes put the two largest elements in place, and the buggy program sorts 45.5 percent of random lists correctly. Repairing it takes three more CSWAPs, in order: positions 0, 1 and 0.

Each run draws 16 random training lists. For an output list, let order be one minus the fraction of its four possible adjacent pairs that are out of order (an empty list has no pairs, so it scores 1), and let kept be the fraction of the input's elements still present in the output. Three fitness functions are compared, each averaged over the training lists:

• F1, the test as first written: order.
• F2, a patch after seeing F1 fail: order, but zero unless the output still has five elements.
• F3, the fix: the average of order and kept, which reaches its maximum of 1 only when the output is the sorted input.

The genetic algorithm uses a population of 60, all initialized to the buggy program, with tournament selection of size 3, one-point crossover with probability 0.9, per-instruction mutation with probability 1/20, one elite, and 150 generations (the components are described in the simple genetic algorithm). The best final program of each run was tested on 1,000 held-out lists. With 30 seeded runs per fitness function:

Under F1 every run reached perfect fitness, and no run produced a sorter: the best program sorted at most 1.9 percent of the held-out lists. In 29 of the 30 runs at least 99 percent of the outputs were in order, so the programs were satisfying the test by losing or duplicating elements. The program from seed 0 keeps the two buggy passes and adds DROP 1 and COPY 0. After two passes the last two positions hold the two largest values in order; deleting position 1 and copying the first value over the second produces a four-element list in which the first two entries are equal and both are no larger than the last two. It turns [42, 7, 93, 7, 15] into [7, 7, 42, 93], which is in order and wrong. The honest repair also scores 1 under F1, so evolution was not forced into the hack. It found it first, plausibly because it needs two particular instructions where the repair needs three in a fixed order.

The difference is easiest to see on a list the buggy program gets wrong. On the reversed list [50, 40, 30, 20, 10], the three programs run as follows (each row is the list after one pass or one further instruction; the first seven instructions, the two passes, are shared):

The buggy program leaves two of its four adjacent pairs out of order, so it scores 0.5 under F1 on this list. The repair and the hack both score 1.0; only the contents of the output tell them apart. The hack works on every input for the reason given above: after two passes the two largest values sit at the end in order, and the first value, copied twice, is no larger than either of them. The traces are printed by tools/figures/surprising-creativity-of-digital-evolution.py.

The patch F2 closed that loophole and opened another. Deletion now scores zero, so the runs found programs that keep five elements and copy values over one another until the list is in order: seed 0 turns [42, 7, 93, 7, 15] into [7, 42, 42, 93, 93]. In 26 of 30 runs at least 99 percent of outputs were in order while at most 2.6 percent were correct. This is the pattern of the carbon-structure anecdote, where each patch to the energy model produced a new violation of physics.

F3 tests what was actually wanted, that the output is ordered and contains what the input contained. Twenty-five of the 30 runs produced true sorters, correct on all 1,000 held-out lists. The other five reached perfect training fitness and sorted only 89.7 to 91.2 percent of the held-out lists: their programs happened to sort all 16 training lists without sorting every list. A finite test set is itself a specification, and a search can overfit to it as it does to any other.

The example is constructed: the instruction set was chosen so that a cheap hack exists. The point it illustrates is not that evolution prefers hacks but that, when an unintended maximum of the fitness function is reachable, the algorithm has no way to tell it from the intended one.

Practical lessons

The discussion section of Lehman and colleagues (2020) draws three lessons for practitioners. A practitioner should be skeptical of their ability to specify a robust fitness function and should expect evolution to reveal specification failures one at a time. They should adopt an adversarial mindset, looking for ways that seemingly innocuous design decisions could be exploited. And when evolving in simulation, they should regularly watch the evolved solutions to check that they are valid, since the scores alone did not reveal the somersaulting jumpers. Amodei and colleagues (2016) suggest, among other approaches, careful engineering of the reward computation and trip wires: deliberately introduced vulnerabilities that a well-behaved agent should not exploit, monitored so that an exploit is detected.

Several further habits follow from the cases above. These are our own recommendations, not claims from the cited papers.

• Score the outcome, not a symptom of it. The sorting test checked order but not contents, and Sims' first fitness rewarded speed rather than locomotion. Writing $U$ down, even informally, before writing $F$ makes the gap easier to see.
• State invariants as constraints. Properties every valid solution must have, such as "the output is a permutation of the input" or "no lens is thicker than the housing", are better handled as explicit constraints than left implicit; see constraint handling in evolutionary algorithms. When several properties matter and their trade-off is unclear, multi-objective optimization keeps them separate instead of folding them into one number.
• Keep an evaluation the optimizer never sees. Judging the final solutions on held-out cases, a second simulator, or physical hardware catches both overfitting to the training cases and exploitation of one simulator's bugs.
• Expect patches to move the problem. A patch that forbids the observed exploit, like F2, leaves other unintended maxima in place. Checking the patched fitness against the intended outcome again is cheaper than discovering the next exploit later.
• Treat surprise as information. A sudden jump in fitness, as in Feldt's aircraft and GenProg's deleted files, more often means the fitness function or the simulator is wrong than that the problem was easy.

References

• Amodei, D., and Clark, J. (2016). Faulty reward functions in the wild. OpenAI blog, 21 December 2016.
• Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete problems in AI safety. arXiv:1606.06565.
• Bird, J., and Layzell, P. (2002). The evolved radio and its implications for modelling the evolution of novel sensors. In Proceedings of the 2002 Congress on Evolutionary Computation (CEC 2002), vol. 2, 1836–1841. IEEE.
• Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., and Legg, S. (2020). Specification gaming: the flip side of AI ingenuity. DeepMind blog, 21 April 2020.
• Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., et al. (2020). The surprising creativity of digital evolution: a collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial Life, 26(2), 274–306.
• Thompson, A. (1997). An evolved circuit, intrinsic in silicon, entwined with physics. In Evolvable Systems: From Biology to Hardware (ICES 1996), Lecture Notes in Computer Science 1259, 390–405. Springer.