# The transition between the niche and neutral regimes in ecology

See allHide authors and affiliations

Edited by David Tilman, University of Minnesota, St. Paul, MN, and approved July 10, 2014 (received for review March 26, 2014)

## Significance

In recent years, there has been a vigorous debate among ecologists over the merits of two contrasting models of biodiversity: the niche and neutral theories of ecology. Using two different theoretical models of ecological dynamics, we show that there is a transition between a selection-dominated regime (the niche phase) and a drift-dominated regime (the neutral phase). This is analogous to the phase diagram of water, which can be in the solid, liquid, or gas phase, depending on the temperature and pressure. Our results demonstrate how the niche and neutral theories both emerge from the same underlying ecological principles.

## Abstract

An ongoing debate in ecology concerns the impacts of ecological drift and selection on community assembly. Here, we show that there is a transition in diverse ecological communities between a selection-dominated regime (the niche phase) and a drift-dominated regime (the neutral phase). Simulations and analytic arguments show that the niche phase is favored in communities with large population sizes and relatively constant environments, whereas the neutral phase is favored in communities with small population sizes and fluctuating environments. Our results demonstrate how apparently neutral populations may arise even in communities inhabited by species with varying traits.

The success of the neutral theory of biodiversity and biogeography (1, 2) at explaining patterns in biodiversity has resulted in a vigorous debate on the processes underlying the assembly, dynamics, and structure of ecological communities (1, 3⇓⇓⇓⇓⇓⇓⇓⇓–12). Starting with the pioneering work of MacArthur (13⇓–15), ecologists have emphasized the roles of interspecific competition and environmental interactions in community assembly and dynamics. These niche-based models emphasize ecological selection as the driving force of community assembly, whereas neutral models of biodiversity assume a functional equivalence between species and emphasize the role of ecological drift (i.e., stochasticity) in community dynamics (1, 2, 16, 17). The success of both types of models at explaining ecological data highlights the crucial need for understanding the impacts of ecological drift and selection in community ecology (18).

## Hypothesis

We begin with a hypothesis that a diverse ecological community with many species can be either neutral or nonneutral, depending on the state of its environment. We call the regime in which a community is well described using neutral models the “neutral phase” and the regime in which the community behaviors are inconsistent with neutrality the “niche phase.” The dynamics in the neutral phase are dominated by stochasticity whereas the dynamics in the niche phase are dominated by selection. Our goal in this paper is to demonstrate that these two phases naturally emerge from simple probabilistic models of ecological dynamics and that a community may transition from one phase to the other as its environment is altered (Fig. 1).

Historically, ecological neutrality is based on the assumption of functional equivalence, which states that trophically similar species are essentially identical in terms of their vital characteristics, such as birth and death rates (19). Ecological neutrality, however, is generally not a measurable feature of a community. Therefore, we adopt a pragmatic definition of neutrality: We say that a community is “statistically neutral” if its multivariate distribution of species abundances cannot be distinguished from a distribution constructed under the assumption of ecological neutrality. In other words, the multivariate species abundance distributions of statistically neutral communities are indistinguishable from those of communities of functionally identical species. Note that ecological neutrality implies statistical neutrality, but statistical neutrality does not necessarily imply ecological neutrality. We can now restate our hypothesis more precisely: As the characteristics of an ecosystem change (e.g., carrying capacity, immigration rate), there will be a transition between a neutral phase where the ecosystem behaves as if it is effectively neutral and a niche phase where the multivariate species abundance distribution is inconsistent with statistical neutrality.

## Background on Phase Transitions in Disordered Systems

Our hypothesis that ecological systems are likely to exhibit multiple phases is based on an analogy with disordered systems in physics. For this reason, we briefly provide some background on phase transitions in disordered systems. A phase transition refers to an abrupt change in the qualitative behavior of a system as one of its characteristics, or a characteristic of its environment, is altered (20). The most well-known example may be the behavior of water, which can be found as a solid, liquid, or gas depending on temperature and pressure. Disordered systems often display a more complicated type of phase transition, labeled the freezing transition, where the system configuration gets “frozen” into a particular state.

One illustrative example of a disordered physical system is a protein (21). A protein can be thought of as a disordered system in which the different amino acids along the protein chain interact heterogeneously. The diversity of interactions in a protein distinguishes natural proteins from homopolymers and is what allows some proteins to fold to a stable native structure, while causing others (like prions or amyloids) to misfold. In ecology, this is analogous to the observation that the diversity of interactions between the species in a community distinguishes niche-like communities from neutral communities. To continue our analogy, at high temperatures, a typical polypeptide sequence will be in an unfolded phase where it samples different configurations randomly. If the temperature is lowered below a critical value, the polypeptide will freeze into a single structure (the folded state). This phase transition occurs when the stochasticity impacting the dynamics (i.e., the temperature) is smaller than the energetic differences in the interactions of the amino acids along the chain. If we take this analogy seriously, we should expect to find a critical amount of stochasticity, compared with the diversity of species traits, that separates neutral and niche communities.

## Theoretical Models for Studying Community Assembly

To test our hypothesis regarding the niche-to-neutral transition, we analyzed two models of ecological dynamics: (*i*) a generalized Lotka–Volterra (LV) system including immigration and stochasticity and (*ii*) a binary model for the presence/absence (PA) of the species in a community. Each of these models has advantages and disadvantages. The LV model is a widely used and interpretable model of many ecological phenomena. However, in general, it is intractable to perform analytic calculations using the LV model and one must rely on numerical simulations. In contrast to the LV system, the PA model is amenable to analytical arguments but this comes at the expense of ignoring species abundances. Both models assume well-mixed populations, although relaxing this assumption is an important avenue of future research. These two models correspond to extreme cases of functional responses (22, 23). The functional response in the PA model is essentially a step function in which species interact only when their abundances are above a threshold. By contrast, the LV model corresponds to linear functional responses. We expect that real communities lie somewhere in between these models.

## Parameterizing Ecosystem Characteristics

To construct ecological phase diagrams, it is necessary to parameterize ecosystem characteristics. Because we are interested in stochastic community assembly, we must introduce parameters that reflect the impact of stochasticity as well as parameters that capture variation in species traits. Due to the similarity of the two models, we use the same symbols for analogous parameters with an added tilde for parameters in the PA model (e.g., *K* denotes carrying capacity in the LV model and

There are two potential sources of stochasticity in the ecological dynamics: “demographic stochasticity” resulting from random births and deaths in small populations and “environmental stochasticity” caused by random variations in the environmental conditions. Although there is no doubt that the origin of the stochasticity is important for making quantitative ecological predictions (24), extensive numerical simulations suggest that the qualitative phase diagrams are insensitive to these details (*SI Appendix*). For this reason, we parameterize the amount of stochasticity by a single parameter, the noise strength ω

We must also introduce parameters describing species traits. In principle, each species in the community has a unique immigration rate, a unique carrying capacity, and some set of parameters that describes how it interacts with other species. In the main text, we restrict ourselves to the case where all species have the same immigration rate, *λ* *K* *SI Appendix* for relaxation of these assumptions). We assume that the regional species pool is large so that there is a separation of timescales between the dynamics of the local community and those of the regional species pool. As a result our model does not explicitly include speciation, even though speciation is ultimately required to maintain diversity over longer timescales relevant to the species pool. Following May’s seminal work (25), we randomly draw symmetric interaction coefficients from a probability distribution and focus on describing the average behavior of ecosystems. Specifically, the interaction matrix *C* *c*_{ij} *i* and *j*—is drawn from a Gamma distribution with mean *μ*/*S* *σ*^{2}/*S* *S* is the number of species (see *SI Appendix* for results with other distributions).

## Stochastic Lotka–Volterra Dynamics

The first model that we analyze is a system of stochastic LV equations including immigration. Niche-based models of community assembly frequently use LV equations as a simplified description of ecological dynamics within a well-mixed community (13, 26⇓–28). Here, we study a system of LV equations incorporating immigration and multiplicative noise (i.e., stochasticity). The rate of change in the abundance (*x*_{i}) of species *i* = 1, … , *S* is*λ*) is the rate that species *i* immigrates into the local community from an infinitely large regional species pool. The second term (*x*_{i}(*K* − *x*_{i})) limits the population of species *i* to its carrying capacity (*K*) in the absence of immigration and species interactions. The third term *i* according to their interaction coefficients (*c*_{ij}). All of these deterministic terms (i.e., *λ*, *K*, and *c*_{ij}) collectively represent the effects of ecological selection on the abundance of species *i*. Ecological drift is incorporated into our model through the last term *η*_{i}(*t*), with mean 〈*η*_{i}(*t*)〉 = 0, variance 〈*η*_{i}(*t*)*η*_{j}(*t*′)〉 = *δ*_{ij}*δ*(*t* − *t*′), and strength

## Dynamics of Presence/Absence Model

In the PA model, a species *i* is described by a binary variable *s*_{i} with *s*_{i} = 1 if species *i* is present in a community and *s*_{i} = 0 if it is absent. The stochastic dynamics of species PA are defined by two rates: the rate at which a species immigrates into a community (i.e., the rate that *s*_{i} = 0 becomes *s*_{i} = 1) and the rate at which a species becomes extinct once it is in the community (i.e., the rate that *s*_{i} = 1 becomes *s*_{i} = 0). Thus, a species immigrates into a community and lives there for some time before it dies out, only to reimmigrate back into the community later, and so on. We assume that the rate of immigration is simply *SI Appendix*. After an initial transient period, the community reaches a steady state where the immigration and extinction processes are balanced. Due to the simplicity of this model, we can derive an analytic expression for the steady-state probability distribution:

## Measuring the Neutrality of a Community

To test our hypothesis that communities can exhibit a niche-to-neutral transition, it is necessary to define “order parameters” that distinguish the niche and neutral phases. By convention, an order parameter is chosen so that it is zero in one phase and greater than zero in the other. Recall that the dynamics in the neutral phase are dominated by stochasticity and multivariate species abundance distributions in this phase are indistinguishable from those obtained from a neutral model with functionally equivalent species. By contrast, the niche phase is dominated by interactions and multivariate species abundance distributions are peaked around the equilibrium value they would have in the absence of stochasticity.

Using these intuitions we can define order parameters for both the LV model and the PA model. In the LV model, we define an order parameter that measures the distance (i.e., Kullback–Leibler divergence) between the multivariate species abundance distribution resulting from LV dynamics and the multivariate species abundance distribution resulting from purely neutral dynamics (see below). This order parameter is zero in the neutral phase and nonzero in the niche phase. For the PA model, it is convenient to consider a different order parameter, the Shannon entropy, of the steady-state PA probability distribution. The Shannon entropy is zero in the niche phase and nonzero in the neutral phase. We now discuss both of these order parameters in more detail.

### Measuring Neutrality in LV Models.

Early studies attempting to quantify the neutrality of a community focused on the shape of the marginal species abundance distribution, i.e., a histogram indicating the number of species with 10 individuals in the community, the number of species with 20 individuals in the community, and so on. However, recent studies have shown that both and neutral and nonneutral ecological models give rise to similar marginal species distributions (12). For this reason, to measure neutrality in the LV model we use the multivariate species abundance distribution. In contrast to previous studies on marginal species abundance distributions in niche and neutral communities (e.g., ref. 12), using the multivariate species abundance distribution allows us to study the effects of different species interactions on correlations in species abundances.

In particular, we quantify statistical neutrality in our LV simulations by measuring the distance between the steady-state distributions of species abundances obtained from the LV model *P*_{LV}∥*P*_{N}) is defined as the amount of information about the true multivariate species abundance distribution [i.e., *P*_{LV}∥*P*_{N}) = 0 implying that the simulated distribution is identical to the distribution obtained under the assumption of neutrality. We study the average of the KL divergence over many random realizations of the species interactions, i.e., 〈KL(*P*_{LV}∥*P*_{N})〉. We expect 〈KL(*P*_{LV}∥*P*_{N})〉 ∼ 0 in the neutral phase, whereas 〈KL(*P*_{LV}∥*P*_{N})〉 ≫ 0 in the niche phase. Similar results are obtained with distance measures other than the KL divergence (*SI Appendix*).

In principle, it is possible to use an explicit formula for *SI Appendix*. With this caveat in mind, we observe that ecologically neutral models are also statistical neutral. Namely, the time-averaged moments of the abundance of species *i* are the same as the time-averaged moments of the abundance of species *j*. Moreover, the correlation in the abundances of species *i* and *j* is the same as the correlation in the abundances of species *k* and *l* (*SI Appendix*). Simulations shown in Fig. 2*A* demonstrate that this is the case, at least for Hubbell’s neutral model (*Materials and Methods*), where the KL divergence equals zero for all positive immigration rates. Finally, we note that although ecological neutrality implies statistical neutrality, statistical neutrality does not necessarily imply ecological neutrality. Thus, our use of statistical neutrality is consistent with the interpretation of ecological neutrality as a type of null model that allows one to identify communities in which selection is important.

### Measuring Neutrality in the Presence/Absence Model.

In the PA model, we do not have access to species abundances. For this reason, it is convenient to define a different order parameter that measures the fluctuations in the binary vector of community composition, *H*[*P*_{PA}] = 0. In contrast, if the dynamics are entirely random, then each species will randomly flip between being absent and present in the community and *H*[*P*_{PA}] = *S* ln 2. For diverse ecosystems with *S* ≫ 1, we can define the boundary between the neutral phase and the niche phase as the points where 〈*H*[*P*_{PA}]/*S*〉 = 0, with angular brackets denoting averaging over random realizations of interaction coefficients.

## Phase Diagrams for Ecological Dynamics

### Phase Diagrams.

Armed with the order parameters discussed in the previous section, we can construct phase diagrams for both the LV and PA models. Fig. 2 shows the KL divergence and entropy as a function of stochasticity and interaction diversity for the two models. First, we note that the phase diagram determined using LV simulations is remarkably similar to the phase diagram calculated using our PA model (compare Fig. 2 *B* and *C*), which suggests that our results are fairly robust to model details. Fig. 2*B* shows that there is a large neutral regime in which 〈KL(*P*_{LV}∥*P*_{N})〉 ∼ 0 in the LV simulations. The distance from neutrality rises once the stochasticity is lowered below a critical value. That is, 〈KL(*P*_{LV}∥*P*_{N})〉 increases for small *ω*/*K*; note that the colors in Fig. 2*B* represent exponential growth in 〈KL(*P*_{LV}∥*P*_{N})〉.

Fig. 2*C* shows the phase diagram for the PA model. In the limit the number of species *S* becomes large, and the entropy is strictly zero in the niche phase (blue shaded area) and different from zero in the neutral phase (white area). In particular, we find that the PA of the species in a community freezes into a small number of configurations determined by the species traits if the stochasticity

It is important to recall that the neutral phase refers to a regime in which the multivariate species abundance distribution is well described by a neutral model, even if the underlying community is not ecologically neutral. For example, one special case that illustrates this relation is a community of species that have become so differentiated that they do not interact at all; i.e., *μ* = 0 and *σ* = 0. Because there is no disorder, this community will reside in the statistically neutral phase even though the species are all highly differentiated. Nevertheless, the species in a community must be differentiated for the community to reside in the niche phase.

### Scaling Relation for the Niche–Neutral Phase Boundary.

We can explicitly calculate the phase boundary separating the niche and neutral phases, using the PA model. For diverse ecosystems with many species *S* ≫ 1, the relation defining the phase boundary can be derived by mapping the problem to the random energy model in physics (30, 31) (*SI Appendix*). Using this mapping we can derive a simple scaling relation that indicates when an ecological community will transition between the niche and neutral phases (*SI Appendix*):

### On the Nature of the Transition.

The transition between the niche and neutral phases in the PA model is sharp (Fig. 3). In the LV model the distance from neutrality (〈KL(*P*_{LV}∥*P*_{N})〉) increases when the stochasticity is lowered below the critical value. However, in the PA model, the derivative of the entropy with respect to stochasticity is undefined along the phase boundary, the signature of a freezing phase transition in the theory of disordered systems. Comparing the two models, the niche-to-neutral transition in the PA model appears to be sharper than in the LV model. Technically, a phase transition occurs at a point with an undefined derivative, whereas the term “crossover” is used to describe transitions between qualitative regimes without this feature. Our numerical results do not distinguish whether the transition in the LV model is a crossover or a true phase transition. This difference arises due to the differences in the functional responses of the two models. These two models were chosen, in part, because they represent the two extremes of possible species functional responses (linear vs. step function). We expect the functional responses of real ecological communities lie somewhere in between these two models. For this reason, we expect that real ecological communities will also exhibit a transition between the niche and neutral phases.

## Ecological Implications

### The Prevalence of Neutral and Niche Communities.

Our model suggests that neutral communities and niche-like communities both correspond to large volumes of the ecological phase diagram. Moreover, our model is such that species always have real differences in traits, but these differences in species traits leave no trace on the equilibrium multivariate species abundance distributions in the statistically neutral regime. This does not preclude the possibility that one could observe the effects of species trait variation on other types of observations. This may explain the success of neutral models at explaining many large-scale patterns in ecology, even though selective forces are well documented, and ubiquitous, on local scales. Furthermore, the crossover region surrounding the phase boundary corresponding to “nearly neutral” communities occupies only a small volume of the phase diagram. As a result, we predict that nearly neutral communities should actually be quite rare, as long as there is not an external force (e.g., group selection) driving communities toward the niche–neutral boundary.

### Ecological Disturbances.

One of our main predictions is that the apparent neutrality of an ecological community is a function of both the inhabiting species and the environment. As a result, it is possible to drive a community between the niche and neutral phases by changing the environmental conditions. As an example, we consider the effects of selective logging on a population of butterflies in a tropical forest on Buru, Indonesia (32). Through habitat destruction, logging essentially moves the butterfly community from a position with high *K*/*ω* to one with low *K*/*ω*, tracing a path along the stochasticity axis in the phase diagram (Fig. 4*A*). Our model predicts that when a diverse community within the niche phase is placed under a stress that lowers *K*/*ω* to the critical value, it will undergo a transition to the neutral regime. LV simulations show that this transition results in a collapse of biodiversity and leads to an increase in the skewness of the species abundance distribution (*SI Appendix*). The increase in skewness of the species abundance distribution calculated from LV simulations is evident in a steeper curve in the rank-abundance plot for low *K*/*ω* compared with high *K*/*ω* (Fig. 4*B*). Similarly, the observed data display an increase in the skewness of the rank-abundance curve of the logged forest relative to the unlogged forest, consistent with a loss of biodiversity accompanying a niche-to-neutral transition (Fig. 4*C*). As our model predicts a transition in the shape of the rank-abundance curve as a function of increasing stochasticity (or decreasing carrying capacity), observations of rank-abundance curves as a function of deforestation, i.e., where the amount of deforestation varies continuously, could provide a more stringent test of our model than only the two endpoints discussed here. Nevertheless, this example demonstrates the potential of ecological phase diagrams for predicting the qualitative effects of community-wide disturbances and for capturing the characteristics that contribute to community resilience.

## Conclusion

In summary, we have argued that the niche and neutral perspectives of ecology naturally emerge from stochastic models for the dynamics of diverse populations as distinct phases of an ecological community. Population dynamics in the niche phase are dominated by ecological selection, whereas population dynamics in the neutral phase are dominated by ecological drift. Furthermore, we have derived a simple scaling relation for determining whether an ecological community will be well described by neutral models.

Our hypothesis can be experimentally tested using synthetic microbial communities in which the immigration rates, carrying capacities, and interaction coefficients can be controlled to search for a transition as one moves from one region of the phase diagram to another (33). Alternatively, connections to island biogeography discussed in *SI Appendix* suggest that our hypothesis could be tested by calculating the KL divergence from the multivariate species abundance distributions on a chain of islands as a function of their distance to the mainland (34). Observation of a transition in the shape of the rank-abundance curve of a community along a disturbance gradient would also provide evidence of the niche–neutral transition.

In this work, we made some simplifications that are unrealistic for natural ecological communities. For example, we restricted our analysis to well-mixed communities with purely competitive interactions. It will be necessary to generalize our results to include the effects of dispersal, mutualism, predator–prey interactions, etc., to obtain a more quantitative model of natural communities. Nevertheless, we conjecture that the presence of a niche–neutral phase transition is robust to these model perturbations. However, disordered systems with complex interactions display additional phases (35), which suggests that more complex ecological communities may also exhibit additional phases with novel characteristics.

## Materials and Methods

We simulated Hubbell’s neutral model with a local community of *J* individuals connected to an infinitely large metacommunity containing *S* = 50 equally abundant species. In each time step, with probability *λ*, an individual randomly drawn from the metacommunity replaced a randomly chosen individual in the local community, or with probability 1 − *λ*, one randomly chosen individual in the local community replaced another randomly chosen individual in the local community. The simulations were run for 5 × 10^{7} steps. Ten simulations were run for each set of parameters, and the results were averaged.

LV simulations with *S* = 50 species were performed over the parameter ranges specified in Figs. 1–4. In each case, the competition coefficients were sampled randomly, and then the stochastic Lotka–Volterra equations (Eq. 1) were forward integrated for 5 × 10^{7} steps of size *δt* = 0.005, using the Milstein method. Ten simulations were run for each set of parameters, and the results were averaged.

## Acknowledgments

We thank Jeff Gore, Alex Lang, Javad Noorbakhsh, Daniel Segre, and Les Kauffman for useful discussion and the Alfred Sloan Foundation for funding.

## Footnotes

- ↵
^{1}To whom correspondence should be addressed. Email: pankajm{at}bu.edu.

Author contributions: C.K.F. and P.M. designed research; C.K.F. performed research; C.K.F. analyzed data; and C.K.F. and P.M. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1405637111/-/DCSupplemental.

## References

- ↵
- ↵
- Hubbell SP

- ↵
- Ricklefs RE,
- Renner SS

- ↵
- ↵
- ↵
- ↵
- ↵
- Jeraldo P,
- et al.

- ↵
- Tilman D

- ↵
- Volkov I,
- Banavar JR,
- Hubbell SP,
- Maritan A

- ↵
- ↵
- Chisholm RA,
- Pacala SW

- ↵
- ↵
- Chase JM,
- Leibold MA

- ↵
- Tilman D

- ↵
- ↵
- ↵
- ↵
- ↵
- Kadanoff LP

- ↵
- Bryngelson JD,
- Wolynes PG

- ↵
- ↵
- ↵
- Lande R,
- Engen S,
- Sæther BE

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵

## Citation Manager Formats

## Sign up for Article Alerts

## Jump to section

- Article
- Abstract
- Hypothesis
- Background on Phase Transitions in Disordered Systems
- Theoretical Models for Studying Community Assembly
- Parameterizing Ecosystem Characteristics
- Stochastic Lotka–Volterra Dynamics
- Dynamics of Presence/Absence Model
- Measuring the Neutrality of a Community
- Phase Diagrams for Ecological Dynamics
- Ecological Implications
- Conclusion
- Materials and Methods
- Acknowledgments
- Footnotes
- References

- Figures & SI
- Info & Metrics