## New Research In

### Physical Sciences

### Social Sciences

#### Featured Portals

#### Articles by Topic

### Biological Sciences

#### Featured Portals

#### Articles by Topic

- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology

# Unified reduction principle for the evolution of mutation, migration, and recombination

Contributed by Marcus W. Feldman, January 31, 2017 (sent for review November 29, 2016; reviewed by Reinhard Burger, Yoh Iwasa, and Hamish G. Spencer)

## Significance

Evolution by Darwinian natural selection can not only shape how organisms survive and reproduce, but also affect transmission of genetic and other information between generations. Modifier-gene models for the evolution of information transmission have revealed a universal tendency for more faithful transmission to evolve in populations at equilibrium where natural selection is balanced by errors in information transmission. This is shown to be a very general property of models that include mutation and migration under selection and recombination under selection on diploids. The breadth of this reduction principle focuses attention on the departures from its mathematical assumptions, which may explain those biological phenomena of information transmission between generations for which the reduction principle fails.

## Abstract

Modifier-gene models for the evolution of genetic information transmission between generations of organisms exhibit the reduction principle: Selection favors reduction in the rate of variation production in populations near equilibrium under a balance of constant viability selection and variation production. Whereas this outcome has been proven for a variety of genetic models, it has not been proven in general for multiallelic genetic models of mutation, migration, and recombination modification with arbitrary linkage between the modifier and major genes under viability selection. We show that the reduction principle holds for all of these cases by developing a unifying mathematical framework that characterizes all of these evolutionary models.

The theory of the reduction principle and its departures has its beginnings, like so many other threads of mathematical evolutionary theory, in Fisher’s *The Genetical Theory of Natural Selection* (1). Fisher describes how the asymmetric flow from fitter genotypes to less-fit genotypes due to homologous recombination between pairs of polymorphic loci creates an “agency” to reduce recombination between genes. Fisher notes, however, that due to the ubiquity of chromosomal crossing-over “it is an inevitable inference that some other cause must induce an equally powerful selection in favor of crossing over.” Fisher continues, “the mathematical difficulties of an exact investigation are worthy of a far more extended treatment.” (ref. 1, p. 119). Decades after Fisher’s observations, the mathematical difficulties have begun to be resolved.

Recombination affects the distribution of genotypes among an organism’s offspring. As such, it can be regarded as a process that affects information transmission between generations. Mutation is another such process. With migration, the geographic location, in addition to genetic information, is transmitted between generations. The evolutionary dynamics of the control of information transmission between generations are the subject of modifier-gene theory.

In evolutionary genetics, modifier theory is the study of the dynamics of a gene (or genes) whose alleles modify the rate of a process that changes genetic information between organismal generations, information that is represented in the state of “major genes” that are under selection. Modifier genes can evolve even when they have no direct effects on fitness, and the primary focus of modifier-gene theory is the dynamics of such “neutral” modifier genes. Neutral modifier alleles change in frequency due to their associations with genes that are under direct selection.

The first mathematical model for the evolution of a modifier gene was that of Nei (2), who assumed that in a large diploid population, a gene with alleles

In subsequent studies of the evolution of modifiers of recombination rates, mutation rates, and migration rates, the same result kept appearing: In populations at an equilibrium balance between natural selection and either mutation, recombination, or migration, a new modifier allele can invade if it reduces the rate of the modified process and cannot invade if it increases that rate (5⇓⇓⇓–9). In ecological models, where the term “migration” is usually replaced by “dispersal,” and in which genetics play no role, reduced dispersal has also been shown to evolve (10). A comparison between such ecological, game-theoretic, or adaptive-dynamics arguments and the formal population genetic arguments that characterize modifier theory was made in Feldman et al. (11). The generality of the result prompted Feldman et al. (12) to call this phenomenon a “reduction principle.” Fisher’s agency was in fact observed to be a general outcome of the evolutionary dynamics of genetic information transmission.

The possibility that a mathematical unity underlies the reduction principle motivated the study by Altenberg (13). What emerged is that the behavior of modifier-gene models possesses a universal part and a particular part. The universal part—the mathematical unity that underlies the reduction principle—is reflected in an important theorem of Karlin (14), which can be summarized as “mixing reduces growth.” Karlin developed the theorem to study the effect of population subdivision on the preservation of genetic diversity, a problem seemingly unrelated to modifier-gene theory. The particular part of modifier-gene models concerns the details of the genetics, for example, how the modifier gene is linked to the major loci, whether the organism is diploid or haploid, the number of information-altering processes, the kind of selection on the major genes, and other such details. These details greatly complicate the mathematical analysis.

The reduction principle was proved to hold for modifiers of mutation, recombination, and migration (dispersal) (13, 15) for arbitrary numbers of genotypes and selection regimes, but only in the case of tightly linked modifiers or modifiers with extreme reduction, by applying Karlin’s theorem. The universal aspect of the reduction principle was shown, but under restricted conditions.

In the converse direction, the particular part of modifier-gene models was solved in a series of papers that include arbitrary numbers of modifier-gene alleles, arbitrary linkage to the major loci, arbitrary selection regimes, and arbitrary control by the modifier locus, in the case of modifiers of recombination (16), mutation (17), and migration (18). The main analytic technique there—the analysis of spectral radii through the characteristic polynomial—limited the universal part to two alleles at the major loci in the case of recombination and mutation modification and two patches in the case of migration modification (19).

The mathematical complications in the particular parts of modifier models were resolved for diploid modifiers (20) through application of an extension of Karlin’s theorem to essentially nonnegative matrices. We apply this method here to multiallele genetic models that include mutation modifiers with haploid selection and migration modifiers with haploid selection. Further, we give the complete results for modifiers of recombination under diploid selection, a case that, although discussed in Altenberg (20), has not been explicitly shown.

We first introduce separately the modification models for mutation, migration, and recombination and exhibit the local stability matrices that determine whether a new modifier allele will invade. Next, we show that there is a unified structure that characterizes these matrices and review the mathematical tools needed to bound the spectral radii of these structurally unified matrices. Finally, we formulate what we call the “unified reduction principle” and outline its simple proof.

## The Models

In what follows we describe models for the modification of mutation, migration, and recombination. In all three models, we consider a large population and a character under selection that is determined by a “major locus” (“major loci” in the recombination case). The genotypes determining this character are multiallelic. Linked to it is a “modifier locus” that has no direct effect on fitness and whose function is to determine the rate of mutation, migration, or recombination. We assume that the modifier locus has two alleles,

### The Mutation Modification Model.

Here we consider a large population of haploids and a character determined by a major locus with

Let

Let **1** and **2**, where only **I** is the **S** is given by

The local stability of **1** and **2** near

The scenario in Eq. **9** is well known from modifier theory (e.g., ref. 19), and the matrix **2** near **2**, as

Using Eqs. **12** and **13**, Eq. **11** can be rewritten as**14** reduces to**5** and **6**, we can write in matrix notation as**D** and **S** are given as in Eqs. **7** and **8**, respectively, with

At **14** reduces for

As **8** and **20**, **S** and **S** is a nonnegative irreducible matrix. Also it is easily seen that**6** and **23**, it is shown that when

### The Migration Modification Model.

Here we assume a large population of haploids that occupies two demes and a modifier locus with two alleles

On the boundary where only modifier allele **24** and **26** are**I** is the **S** is a reducible column-stochastic matrix, which can be rearranged as a diagonal block matrix, with **D** given by

This equilibrium **25** and **27**, **34** and **35**, **29** implies that **34** and **35** reduce to**I**, **S**, and **D** are as in Eqs. **31** and **32**.

For **34** and **35**, **D** is as in Eq. **31** but with

Combining Eqs. **39** and **41**, and because **S**, **Q**, and

### The Recombination Modification Model.

Consider a diploid population and a character determined by two major loci with alleles

With three loci there are two recombination intervals

At each generation the gametes

For the external stability of equilibria on the boundary, where only **T** from

To these ends we need two segregation tables: Table 1, for **T**, gives the probabilities that the gamete

From Table 1 we derive the transformation **T** from **50** **52** as

Let **53** can be written (for all

Because

In the same way, Table 2 determines the external local stability given by the transformation

### Observation 1.

*When* *for all*

In fact, substituting **56** we have**53**, as **57** reduces to

In what follows we assume that crossover events in the two intervals

### Observation 2.

*With no recombination interference, the external stability matrix* *can be represented as* *where* **D** *is a diagonal matrix*

Here, **I** is the identity matrix, **S** and **Q** is a positive column-stochastic matrix. In addition,*Proof of Observation 2*.

Note that the diagonal matrices **55** can be written as**64** we would have **64** implies that **63** are nonscalar.

## Proof of Observation 2

We start with **57**, **S1** as**I** is the **S** is the

When **56** becomes**Q** is a **60**. By *Observation 1*, **62** follows from Eq. **61**. Also, from Eq. **S5** and the definition of *Q*, we have

## A Unified Mathematical Structure for Invasion

Although the three models for modification of mutation, migration, and recombination are different in structure, they share similar mathematical representations for their equilibrium equations and external stability matrices.

In fact, if **I** is the identity matrix, **S** is a nonnegative column-stochastic matrix, and **D** is a positive nonscalar diagonal matrix. **S** is an irreducible matrix in the mutation modification models. In the recombination and migration modification cases, **S** is a reducible matrix of irreducible blocks, and each part of **D** associated with each block of **S** is a nonscalar matrix.

In all three modification models the external stability matrix **66**, **Q** and

Our goal is to determine when the equilibrium

The mathematical tools for analyzing the spectral radius of matrices like

## Mathematical Tools

Karlin (14) proved the following theorem.

### Theorem 1.

*Karlin*’*s theorem*: *Let* **S** *be an arbitrary nonnegative irreducible column-stochastic matrix and consider the family of matrices**Then*, *for any diagonal nonscalar matrix* **D** *with positive terms on the diagonal*, *the spectral radius of* *is strictly decreasing as* *increases.*

Observe that in our modification models, when **S** and **D** satisfy the conditions of the theorem. Indeed, with mutation modification, **S** and **D** satisfy these conditions, and because the spectral radius of **S** can be a reducible matrix and Karlin’s theorem does not yield strict monotonicity of the spectral radius with respect to **D**. Moreover, Karlin’s theorem cannot be directly applied to the case when **69**

An essentially nonnegative matrix [also referred to as a Metzler–Leontief (ML) matrix] is a square real matrix all of whose off-diagonal elements are nonnegative. The spectral bound of a matrix **A**, denoted by **A** and

Let *i*) **B**. (*ii*) If **z** is a positive vector, then *iii*) If **B** is a nonnegative matrix, then

Altenberg (20) proved the following generalization of Karlin’s theorem.

### Theorem 2.

*Let* **U** *be an ML matrix that is irreducible and column stochastic. Let* *be the family of matrices**Then, for any nonscalar positive diagonal matrix* **D**, *the spectral bound* *is strictly decreasing as* *increases.*

The case where **U** is reducible was also analyzed by Altenberg (20). Specifically, if **U** is a reducible column-stochastic ML matrix of the form**D** associated with each of the blocks

## The Unified Reduction Principle

### Theorem 3.

*Consider a population with a multiallelic major locus and a biallelic* (*with alleles* *and* *modifier locus and a stable equilibrium* *where only the* *modifier allele is present*, *producing the mutation*, *migration*, *or recombination rate* *. Then* *is stable to the introduction of the* *allele at the modifier locus*, *with associated rate* *if* *and it is unstable if*

*Proof:* Because **66**, and

When **S** is nonnegative, column stochastic, and irreducible in the case of mutation modification. **D** is a positive diagonal nonscalar matrix. In the recombination and migration modification model, **S** is a reducible block matrix, and the associated submatrices of **D** are positive diagonal nonscalar matrices.

Thus, following Karlin and Altenberg, the spectral radius of **72** is a decreasing function of

If **D** is a positive diagonal nonscalar matrix. Let

Observe that **67** and **68** we have

Hence **73** and **80** we have proven the reduction principle for the three models for all

## Discussion

Through a simplification of the technique developed by Altenberg (20) we have shown that the reduction principle is a general property of mutation, migration, and recombination modifiers in multiallele models with arbitrary recombination between the modifier and major genes. These cases thus combine the “universal part” of the reduction principle with several additional “particular parts.”

A version of the reduction principle appears in a number of models without explicit genetics, principally ecological models for the evolution of dispersal. In ecological models, where the term migration is usually replaced by dispersal and in which genetics play no role, reduced dispersal has also been shown to evolve (10). Karlin’s theorem was independently discovered by Kirkland et al. (24) and applied to discrete-patch models of dispersal evolution. The reduction principle was found to hold in reaction–diffusion models of dispersal, where space is continuous, and dispersal is represented not by a finite matrix but by a differential or integral operator (10, 25⇓⇓–28). These results were incorporated into the reduction principle by Altenberg (29) through a theorem that generalizes Karlin’s theorem to all resolvent-positive operators, including second-order differential operators, Schrödinger operators, and nonlocal dispersal kernels.

Another kind of information that may be transmitted between organisms is cultural information. The reduction principle was demonstrated (ref. 13, pp. 203–206) to hold for a model of a vertically transmitted cultural trait whose faithfulness of transmission is determined by another vertically transmitted cultural trait. The applicability of the reduction principle to cultural inheritance is a largely unexplored area.

The mathematical proof of the reduction principle is valid under the minimal assumptions of infinite population size, constant-viability selection, a population at equilibrium under a balance between selection and a process that alters transmitted information, and equal scaling of transition probabilities during replication [referred to as “linear variation” (20)]. Violations of the reduction principle entail specific departures from these assumptions.

The principle sources of departure are as follows: (*i*) Allele frequencies at the major loci are not at equilibrium, due to being in a transient phase, or at a periodic or chaotic attractor, or due to fluctuating selection, or due to genetic drift. (*ii*) The modifier gene does not scale the transition probabilities between genotypes equally (or between other types of transmitted information, such as patch location), due to the presence of other transforming processes [the principle of partial control (13, 30)] or biases in the direction of mutation or dispersal. (*iii*) Natural selection acts directly on the modifier gene rather than indirectly as a consequence of transmission modification.

A modifier allele that increases recombination may invade if introduced while the major loci are proceeding toward fixation (31⇓–33). Increased recombination may also evolve when the major loci are under cyclically fluctuating selection, either exogenously caused (34) or induced by host–parasite dynamics (35⇓–37). Similarly, mutation-increasing alleles may invade under some patterns of fluctuating selection on the major gene (38⇓⇓⇓⇓⇓–44). However, the direction of change in mutation rate is sensitive to the form of selection on the major gene(s) (44⇓–46). Finally, migration-increasing alleles can succeed under some forms of temporal variation in the geographic pattern of selection on the major genes (30, 46, 47). In general, the reduction principle does not hold if selection occurs at the level of fertility differences between mating pairs or if there is mixed inbreeding and outcrossing with viability selection (48⇓–50).

A further class of models for which the reduction principle fails includes two (or perhaps more) kinds of information transmission among the major loci. An important example has the major loci subject to both mutation and viability selection and the modifier affecting recombination among the major loci. In this case, an allele reducing recombination can succeed if there is diminishing-returns epistasis among the major loci, but increased recombination may succeed if there is synergistic epistasis (12, 51, 52).

Frequency-dependent selection brings no new behavior to the dynamics if the new modifier allele is introduced into a population at a stable equilibrium with viability differences among the major loci (13, 15). However, it is possible for frequency-dependent selection to create periodic or chaotic attractors (53⇓–55) where the genotype frequencies are constantly changing. In such cases modifiers that increase mutation rates may invade (56, 57).

Departures from the reduction principle represent a wide diversity of biological phenomena and processes. We may ask whether in mathematical models that show these departures from the reduction principle there is a universal part as in the reduction principle itself. Very few relevant results have been obtained. We point to one general theorem (47) showing that when growth rates among multiple patches alternate every generation between

Here we have expanded the class of models for which the reduction principle holds. A complete picture of the conditions under which the reduction principle or departures from it hold remains a largely open theoretical question.

## Acknowledgments

This work was supported in part by the Stanford Center for Computational, Evolutionary, and Human Genomics, Stanford University; the Konrad Lorenz Institute for Evolution and Cognition Research; the University of Hawaii at Manoa; and the Mathematical Biosciences Institute through National Science Foundation Award DMS 0931642.

## Footnotes

↵

^{1}L.A., U.L., and M.W.F. contributed equally to this work.- ↵
^{2}To whom correspondence should be addressed. Email: mfeldman{at}stanford.edu.

Author contributions: L.A., U.L., and M.W.F. designed research, performed research, analyzed data, and wrote the paper.

Reviewers: R.B., University of Vienna; Y.I., Kyushu University; and H.G.S., University of Otago.

The authors declare no conflict of interest.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1619655114/-/DCSupplemental.

Freely available online through the PNAS open access option.

## References

- ↵.
- Fisher RA

- ↵.
- Nei M

- ↵
- ↵
- ↵.
- Feldman MW,
- Balkau B

- ↵
- ↵.
- Feldman MW,
- Krakauer J

- ↵
- ↵
- ↵
- ↵.
- Feldman MW,
- Otto SP,
- Christiansen FB

- ↵.
- Feldman MW,
- Christiansen FB,
- Brooks LD

- ↵.
- Altenberg L

*A Generalization of Theory on the Evolution of Modifier Genes*(ProQuest, Ann Arbor, MI), ProQuest document ID: 303425436. Available at search.proquest.com/docview/303425436/abstract. Accessed February 21, 2017. - ↵.
- Karlin S

- ↵.
- Altenberg L,
- Feldman MW

- ↵
- ↵
- ↵.
- Liberman U,
- Feldman MW

- ↵.
- Feldman MW,
- Liberman U

- ↵
- ↵.
- Bürger R

- ↵.
- Altenberg L

- ↵
- ↵.
- Kirkland S,
- C-KLi,
- Schreiber SJ

- ↵.
- Hutson V,
- López-Gómez J,
- Mischaikow K,
- Vickers GT

- ↵.
- Dockery J,
- Hutson V,
- Mischaikow K,
- Pernarowski M

- ↵.
- Cantrell R,
- Cosner C,
- Lou Y

- ↵
- ↵.
- Altenberg L

- ↵
- ↵.
- Maynard Smith J

- ↵
- ↵
- ↵.
- Charlesworth B

- ↵
- ↵
- ↵.
- Gandon S,
- Otto SP

- ↵
- ↵.
- Ishii K,
- Matsuda H,
- Iwasa Y,
- Sasaki A

- ↵
- ↵.
- Thattai M,
- Van Oudenaarden A

- ↵.
- Kussell E,
- Leibler S

- ↵.
- Gaál B,
- Pitchford JW,
- Wood AJ

- ↵.
- Liberman U,
- Van Cleve J,
- Feldman MW

- ↵.
- Salathé M,
- Van Cleve J,
- Feldman MW

- ↵.
- Carja O,
- Liberman U,
- Feldman MW

- ↵Schreiber SJ, C.-K. Li (2011) Evolution of unconditional dispersal in periodic environments..
*J Biol Dyn*5(2):120–134. - ↵.
- Holsinger KE,
- Feldman MW

- ↵.
- Holsinger KE,
- Feldman MW

- ↵.
- Holsinger K,
- Feldman MW,
- Altenberg L

- ↵
- ↵
- ↵
- ↵.
- May R

- ↵
- ↵
- ↵.
- Rosenbloom DI,
- Allen B