# Collapse of cooperation in evolving games

See allHide authors and affiliations

Edited by Christian Hilbe, Harvard University, Cambridge, MA, and accepted by the Editorial Board October 22, 2014 (received for review May 9, 2014)

## Significance

This study offers a new perspective on an age-old question: When does cooperation emerge in populations? Two-player games used to study this question produce an array of counterintuitive results. And yet a consensus has emerged that, in an evolving population, cooperation tends to triumph over cheating––through reciprocity and generosity. But, what happens when players can influence not only their tendencies to cooperate, but also the rewards they reap for cooperation? We analyze coevolution of strategies and payoffs and find that, as individuals maximize the benefits of cooperation, they often pave the way for its collapse. Our analysis provides a framework for studying the coevolution of games and strategies, and suggests that maintaining cooperation may be more difficult than previously thought.

## Abstract

Game theory provides a quantitative framework for analyzing the behavior of rational agents. The Iterated Prisoner’s Dilemma in particular has become a standard model for studying cooperation and cheating, with cooperation often emerging as a robust outcome in evolving populations. Here we extend evolutionary game theory by allowing players’ payoffs as well as their strategies to evolve in response to selection on heritable mutations. In nature, many organisms engage in mutually beneficial interactions and individuals may seek to change the ratio of risk to reward for cooperation by altering the resources they commit to cooperative interactions. To study this, we construct a general framework for the coevolution of strategies and payoffs in arbitrary iterated games. We show that, when there is a tradeoff between the benefits and costs of cooperation, coevolution often leads to a dramatic loss of cooperation in the Iterated Prisoner’s Dilemma. The collapse of cooperation is so extreme that the average payoff in a population can decline even as the potential reward for mutual cooperation increases. Depending upon the form of tradeoffs, evolution may even move away from the Iterated Prisoner’s Dilemma game altogether. Our work offers a new perspective on the Prisoner’s Dilemma and its predictions for cooperation in natural populations; and it provides a general framework to understand the coevolution of strategies and payoffs in iterated interactions.

Iterated games provide a framework for studying social interactions (1⇓⇓⇓⇓–6) that allows researchers to address pervasive biological problems such as the evolution of cooperation and cheating (2, 7⇓⇓⇓⇓–12). Simple examples such as the Iterated Prisoner’s Dilemma, Snowdrift, and Stag Hunt games (13⇓⇓⇓⇓–18) showcase a startling array of counterintuitive social behaviors, especially when studied in a population replicating under natural selection (16, 19⇓⇓⇓⇓⇓–25). Despite the subject’s long history, a systematic treatment of all evolutionary robust cooperative outcomes for even the simple Iterated Prisoner’s Dilemma has only recently emerged (21, 26⇓⇓–29).

Understanding the evolution of strategies in a population under fixed payoffs already poses a steep challenge. To complicate matters further, in many biological settings the payoffs themselves may also depend on the genotypes of the players. Changes to the payoff matrix have been studied in a number of contexts, including one-shot two-player games (13), payoff evolution without strategy evolution (30, 31), under environmental “shocks” to the payoff matrix (32⇓–34), and using continuous games (22, 23, 35). Here we adopt a different approach, and we explicitly study the coevolutionary dynamics between strategies and payoffs in iterated two-player games. We decouple strategy mutations from payoff mutations, and we leverage results on the evolutionary robustness of memory-1 strategies with arbitrary payoff matrices to explore the relationship between payoff evolution and the prevalence of cooperation in a population. We identify a feedback between the costs and benefits of cooperation and the evolutionary robustness of cooperative strategies. Depending on the functional form (35) of the relationship between costs and benefits, this feedback may either reinforce the evolutionary success of cooperation or else precipitate its collapse. In particular, we show that cooperation will always collapse when there are diminishing returns for mutual cooperation.

## Methods and Results

### Iterated Two-Player Games.

In an iterated two-player game, players *X* and *Y* face each other in an infinite number of successive “rounds.” In each round the players simultaneously choose their plays and receive associated payoffs. We study games with a *c*) and “defect” (*d*), using the traditional language for the Prisoner’s Dilemma. The four corresponding payoffs for player *X* facing player *Y* are *X*’s play is denoted first. In general, *X* may choose her play in each round depending on the outcomes of all previous rounds.

We will focus on memory-1 players (21, 26⇓⇓–29, 36⇓⇓–39), whose choice each round depends only on the previous round. Such a strategy is described by the probabilities of cooperation given the four possible outcomes of the previous round: *X* facing opponent *Y*, denoted *SI Appendix*).

### Evolution of Strategies.

We consider a well-mixed population of *N* individuals evolving under weak mutation who are each characterized by a memory-1 strategy. An individual’s reproductive success depends on her total payoff when pitted in pairwise iterated games against all other individuals in the population, so that the composition of strategies in the population evolves over time. The strategies that tend to succeed in evolving populations can be understood in terms of evolutionary robustness (21, 26). A strategy is evolutionary robust if, when resident in a population, no new mutant strategy is favored to spread by natural selection. Evolutionary robustness is a weaker condition than that of an evolutionary stable strategy (ESS) (16, 40) (*SI Appendix*). Robustness is a useful notion because there is rarely if ever an ESS, as many strategies

It may seem restrictive to focus on memory-1 strategies (36, 37). However, Press and Dyson (27) have shown that a memory-1 player can treat all opponents as although they also have memory-1. As a result of this lemma, a memory-1 player that resists invasion by all memory-1 opponents also resists invasion by all longer-memory opponents. So, even though we restrict our analysis to memory-1 strategies, we nonetheless identify the memory-1 strategies that are robust against all opponents, regardless of the opponent’s memory capacity within a given iterated game (*SI Appendix*). However, our analysis does not allow a player to retain memory of prior interactions with different opponents, in which case such a superlong-memory player could discern the composition of strategies in the population and gain an evolutionary advantage (42⇓–44).

### Coevolution of Strategies and Payoffs.

Here we expand the traditional purview of evolutionary game theory by allowing heritable mutations that affect a player’s payoffs, as well as mutations that affect her strategy, so that the composition of payoffs and strategies in a population coevolve over time (Fig. 1). We study evolution in the iterated two-player public-goods game, with payoffs initially chosen to produce a Prisoner’s Dilemma.

In the public-goods game, a player cooperates by contributing an amount *C* to a public pool, producing a group benefit whose size depends in some way on the total amount contributed by both players. The benefit is then shared between the two players. We consider a generalized form of the public-goods game in which a player *X* contributes a cost

We allow heritable mutations that affect the payoff matrix, by allowing small changes to a player’s cost *C* or synergy factor *α*. It is natural to assume that the benefit of mutual cooperation increases with the cost of contribution, and so we enforce the relationship *X* is comprised of her strategy vector **p**, her contributed cost *X* faces player *Y* has entries *X* and *Y* in the iterated game, however, depend on the players’ strategies as well as this payoff matrix.

We consider populations evolving under weak mutation, so that any new mutant genotype either fixes or is lost before another genotype is introduced (Fig. 1). As a result, a mutation that changes a player’s strategy is introduced into a population of individuals who all share the same payoff matrix. Strategy mutations are drawn uniformly from the space of all memory-1 strategies, whereas mutations that alter an individual’s cost *C* or synergy factor *α* are drawn locally.

### Collapse of Cooperation in the Iterated Prisoner’s Dilemma.

How does cooperation fare when both strategies and payoffs evolve in a population? To study this we first analyze coevolution of strategies and payoffs in games restricted to the Iterated Prisoner’s Dilemma. We assume a linear relationship between *B* and *C*, by stipulating *B* and *C.*) We choose *γ* and *k* such that

Starting from a game with fixed payoffs *A*), in agreement with previous results (14, 21). However, when both strategies and payoffs coevolve there is a striking reversal of fortunes. Evolution favors increasing the benefits of mutual cooperation as well as the benefits of unilateral defection (Fig. 2*B*). The evolution of the payoff matrix is accompanied by a dramatic collapse of cooperation, so that the population is eventually dominated by defection (Fig. 2*A*). Paradoxically, defection comes to dominate even as the payoffs available for mutual cooperation continually increase (Fig. 2*B*). Moreover, this collapse of cooperation is often accompanied by an erosion of mean population fitness (Fig. 2*C*).

There is a simple intuition for this disheartening evolutionary outcome: Initially, the population is typically composed of self-cooperating strategies and so mutations to *C* that increase the reward for mutual cooperation,

### The Volume of Robust Strategies.

We can understand the collapse of cooperation, and the coevolution of strategies and payoffs more generally, by determining which strategies are evolutionary robust and how robustness varies as payoffs evolve. To do so, we have analytically characterized all evolutionary robust memory-1 strategies for arbitrary *SI Appendix*). In particular, we have proven the following necessary condition: a robust memory-1 strategy must be one of three types––self-cooperate, self-defect, or self-alternate. Self-cooperative strategies *SI Appendix*) and they determine whether a population tends to adopt self-cooperation, self-defection, or self-alternation (Fig. 2 *A* and *B*).

For the generalized public-goods game illustrated in Fig. 1, for example, the evolutionary robust strategies of each type satisfy

As these equations show, in the case of the Prisoner’s Dilemma with

The robust self-cooperators, self-defectors, and self-alternators contain many of the “classic” strategies known to be successful in the Iterated Prisoner’s Dilemma (1, 20, 21, 28). For example, “win–stay–lose–shift” (21) belongs to the robust self-cooperators [provided *SI Appendix*).

### Functional Relationships Between Costs and Benefits.

So far we have assumed that benefits of cooperation increase linearly with costs. However, our analysis in terms of volumes of robust strategies allows us to study any functional relationship between benefits and costs. We can therefore extend our results to explore more generally under what circumstances the collapse of cooperation will occur.

We first consider cases in which *C*––that is, when the benefit of mutual cooperation always increases with the cost of cooperation [see *SI Appendix* for analogous results with *C* then any social dilemma will quickly disappear, as benefits for cooperation quickly become much greater than costs, and cooperative strategies reach high frequency (*SI Appendix*, Figs. S4 and S6). If *C* but with intercept *SI Appendix*, Fig. S5). In the more realistic and interesting cases, for which the benefits of mutual cooperation either saturate or increase sublinearly with *C*, then the collapse of cooperation will always occur as *C* increases, just as in the linear case with *C* gets large. Examples of these functional forms, which feature diminishing returns for increasing costs of cooperation, are shown in *SI Appendix*, Fig. S6*A* for *SI Appendix*, Fig. S6*B* for

Alternatively, *C* such that there is an optimal value of *C*. In this scenario the long-term prevalence of cooperation in a population will depend on the ratio *SI Appendix*, Fig. S8).

### Evolution Away from the Prisoner’s Dilemma.

So far we have explored public-goods games whose payoffs form a Prisoner’s Dilemma. However, our analysis in terms of the volume of robust strategies applies to arbitrary *α* and *C*. The parameter *α* has a simple biological interpretation in terms of synergism (*C* may be augmented or depreciated when both players cooperate, as opposed to when only one cooperates. As before, we assume *C*. If *α* and *C* evolve independently and can adopt any values then this mutation scheme produces all possible qualitative *SI Appendix*, Fig. S3). In particular, when

Fig. 4 illustrates the emergence of qualitatively new games in a population initialized at the Iterated Prisoner’s Dilemma. In this figure we assume that mutations that increase the cost *C* will either increase antagonism (Fig. 4, *Top*) or increase synergy (Fig. 4, *Bottom*). As payoffs and strategies coevolve in this more general framework, the benefits and costs of cooperation initially increase, resulting again in the collapse of cooperation (Fig. 4*A*). However, the subsequent increase (or decrease) in *α* leads to qualitatively different outcomes for the payoff matrix. Under the mutation scheme in which *α* increases with *C*, the Prisoner’s Dilemma eventually gives way to a Snowdrift game and populations are dominated by self-alternating strategies (Fig. 4, *Top*). Under the mutation scheme in which *α* decreases with *C*, the Prisoner’s Dilemma gives way to a Stag Hunt game and self-cooperating strategies recover high prevalence in populations (Fig. 4, *Bottom*).

When *α* and *C* are allowed to evolve independently, the resulting coevolutionary dynamics are similar to the case of increasing antagonism: both *α* and *C* increase and the population is eventually dominated by self-alternating strategies (*SI Appendix*, Fig. S7). If *α* is constrained to the range *SI Appendix*, Fig. S7 *A* and *B*), whereas if *α* is unconstrained it evolves to values exceeding unity and the game *SI Appendix*, Fig. S7 *C* and *D*). The instability of the Iterated Prisoner’s Dilemma in favor of the Iterated Snowdrift or Iterated Stag Hunt games is striking, although the potential for increasing antagonism or synergy in evolving games may be subject to physical constraints in natural populations.

## Discussion

We have studied coevolution of strategies and payoffs in populations of individuals reproducing according to their payoffs in pairwise interactions. We have focused primarily on payoff mutations that enforce a tradeoff, by simultaneously increasing the benefits and costs of cooperation. However, our framework for analyzing payoff-strategy coevolution, based on computing the evolutionary robustness of strategy sets, can be applied to any mutation scheme and can produce a potentially vast array of evolutionary outcomes. For example, the way in which benefits are shared between players who mutually cooperate but contribute differentially to the public good may alter how payoffs and hence strategies evolve (23). In general, for two-player public-goods games, if there are diminishing returns for increasing costs of cooperation, then the ratio

What types of payoff mutations can arise in a natural population will depend upon the biological context. Examples of the tradeoff between costs and benefits that we have studied (Figs. 2 and 4) are found in nature at many scales (8, 9, 46), from human societies, where individuals modulate both how frequently and how much they punish free-riders (8, 47), to microorganisms such as the marine bacteria *Vibrionaceae* (9). Our framework decouples strategy and payoff evolution, allowing us to explore the relationship between the “behavior” prevalent in populations (that is, the frequency of cooperation) and the amount contributed to the public good when cooperation occurs (that is, the cost *C*). In *Vibrionaceae* populations, for example, individuals cooperate in a public-goods game by sharing siderophores required for iron acquisition. Mutations that alter whether the siderophore biosynthetic pathway is activated or not alter an individual’s strategy, whereas mutations that improve or degrade the siderophore transport pathway alter an individual’s payoffs, by imposing a greater or lesser metabolic cost along with an increased benefit from the public good.

Decoupling strategy evolution from payoff evolution may not be appropriate in all biological contexts. Alternative modeling frameworks such as continuous games, which allow players to modulate their levels of investment in a social interaction (22, 23, 35), can provide a contrasting or complementary perspective on the evolution of cooperation. What is clear from *Vibrionaceae* populations, as well as many other biological systems with opportunities for cooperative interactions (9, 46, 48⇓⇓–51), is that both cooperators and defectors are often found at appreciable frequencies in nature. As we have shown, the predicted prevalence of these behaviors depends critically on the payoffs resulting from social interactions. Understanding the feedback between strategy evolution and payoff evolution is therefore critical for understanding social interactions in natural populations.

## Acknowledgments

We thank A. Traulsen for productive discussion and for careful reading of our manuscript and J. Coyne for careful reading of our acknowledgments. J.B.P. acknowledges funding from the Burroughs Wellcome Fund, the David and Lucile Packard Foundation, US Department of the Interior Grant D12AP00025, US Army Research Office Grant W911NF-12-1-0552, and Foundational Questions in Evolutionary Biology Fund Grant RFP-12-16.

## Footnotes

- ↵
^{1}To whom correspondence should be addressed. Email: jplotkin{at}sas.upenn.edu.

Author contributions: A.J.S. and J.B.P. designed research, performed research, analyzed data, and wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission. C.H. is a guest editor invited by the Editorial Board.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1408618111/-/DCSupplemental.

## References

- ↵.
- Axelrod R,
- Hamilton WD

- ↵.
- Axelrod R

- ↵.
- Nowak MA

- ↵
- ↵.
- Rapoport A,
- Chammah AM

- ↵.
- Sigmund K

- ↵.
- Axelrod R,
- Axelrod DE,
- Pienta KJ

- ↵.
- Boyd R,
- Gintis H,
- Bowles S

- ↵.
- Cordero OX,
- Ventouras L-A,
- DeLong EF,
- Polz MF

- ↵.
- Nowak MA

- ↵
- ↵.
- Waite AJ,
- Shou W

- ↵.
- Akçay E,
- Roughgarden J

- ↵
- ↵.
- Kümmerli R, et al.

- ↵
- ↵.
- Starnini M,
- Sanchez A,
- Poncela J,
- Moreno Y

- ↵.
- Pacheco JM,
- Santos FC,
- Souza MO,
- Skyrms B

- ↵
- ↵
- ↵.
- Stewart AJ,
- Plotkin JB

- ↵
- ↵
- ↵.
- Fudenberg D,
- Maskin E

- ↵
- ↵.
- Akin E

- ↵.
- Press WH,
- Dyson FJ

- ↵.
- Hilbe C,
- Nowak MA,
- Sigmund K

- ↵.
- Stewart AJ,
- Plotkin JB

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Doebeli M,
- Hauert C,
- Killingback T

- ↵.
- Lindgren K

- ↵
- ↵
- ↵.
- Hauert C,
- Schuster HG

- ↵.
- Maynard Smith J

- ↵
- ↵
- ↵
- ↵.
- Lee C,
- Harper M,
- Fryer D

- ↵
- ↵.
- Greig D,
- Travisano M

- ↵.
- Hauert C,
- Traulsen A,
- Brandt H,
- Nowak MA,
- Sigmund K

- ↵
- ↵.
- Vulic M,
- Kolter R

- ↵
- ↵
- ↵

## Citation Manager Formats

## Article Classifications

- Biological Sciences
- Evolution

- Social Sciences
- Social Sciences