# How market ecology explains market malfunction

^{a}Institute for New Economic Thinking, Oxford Martin School, University of Oxford, Oxford OX1 3QD, United Kingdom;^{b}Computer Science Department, University of Oxford, Oxford OX1 3QD, United Kingdom;^{c}Mathematical Institute, University of Oxford, Oxford OX1 3QD, United Kingdom;^{d}Santa Fe Institute, Santa Fe, NM 87501

See allHide authors and affiliations

Edited by Simon Asher Levin, Princeton University, Princeton, NJ, and approved March 24, 2021 (received for review September 16, 2020)

## Significance

We develop the mathematical analogy between financial trading strategies and biological species and show how to apply standard concepts from ecology to financial markets. We analyze the interactions of stereotypical trading strategies in ecological terms, showing that they can be competitive, predator–prey, or mutualistic, depending on the wealth invested in each strategy. The deterministic dynamics suggest that the system should evolve toward an efficient state where all strategies make the same average returns. However, this happens slowly and the evolution is so noisy that there are large fluctuations away from the efficient state, causing bursts of volatility and extended periods where prices deviate from fundamental values. This provides a conceptual framework that gives insight into the reasons why markets malfunction.

## Abstract

Standard approaches to the theory of financial markets are based on equilibrium and efficiency. Here we develop an alternative based on concepts and methods developed by biologists, in which the wealth invested in a financial strategy is like the abundance of a species. We study a toy model of a market consisting of value investors, trend followers, and noise traders. We show that the average returns of strategies are strongly density dependent; that is, they depend on the wealth invested in each strategy at any given time. In the absence of noise, the market would slowly evolve toward an efficient equilibrium, but the statistical uncertainty in profitability (which is calibrated to match real markets) makes this noisy and uncertain. Even in the long term, the market spends extended periods of time away from perfect efficiency. We show how core concepts from ecology, such as the community matrix and food webs, give insight into market behavior. For example, at the efficient equilibrium, all three strategies have a mutualistic relationship, meaning that an increase in the wealth of one increases the returns of the others. The wealth dynamics of the market ecosystem explain how market inefficiencies spontaneously occur and gives insight into the origins of excess price volatility and deviations of prices from fundamental values.

Why do markets malfunction? According to the theory of market efficiency, markets always function perfectly. Prices always reflect fundamental values and change only when new information affects fundamental values. Thus, by definition, any problems with price setting are caused by factors outside the market. Empirical evidence suggests otherwise. Large price movements occur even when there is very little new information (1), and prices often deviate substantially from fundamental values (2). This means that we need to go beyond the theory of market efficiency to understand how and why markets malfunction.

Here we build on earlier work (3⇓⇓⇓–7)* and develop the theory of market ecology, which provides the necessary alternative. This approach borrows concepts and methods from biology and applies them to financial markets. Financial trading strategies are analogous to biological species. Plants and animals are specialists that evolve to fill niches that provide food; similarly, financial trading strategies are specialists that evolve to exploit market inefficiencies. Trading strategies can be classified into distinct categories, such as technical trading, value investing, market making, statistical arbitrage, and many others. The capital invested in a strategy is like the population of a species. Trading strategies interact with one another via price setting, and the market evolves as the wealth invested in each strategy changes through time, as regulations change, and as old strategies fail and new strategies appear.

The theory of market ecology emerges from the inherent contradictions in the theory of market efficiency. A standard argument used to justify market efficiency is that competition for profits by arbitrageurs should cause markets to rapidly evolve to an equilibrium where it is not possible to make excess profits based on publicly available information. But, if there are no profits to be made, there are no incentives for arbitrageurs, so there is no mechanism to make markets efficient. This paradox suggests that, while markets may be efficient in some approximate sense, they cannot be perfectly efficient (8). In contrast, under the theory of market ecology, trading strategies exploit market inefficiencies, but, as new strategies appear and as the wealth invested in each strategy changes, the inefficiencies change as well. To understand how the market functions, it is necessary to understand how each strategy affects the market and how the interactions between strategies cause market inefficiencies to change with time. The theory of market ecology naturally addresses a different set of problems than the theory of market efficiency and can be viewed as a complement rather than a substitute.

Our study here builds on a large body of work on agent-based models of financial markets (e.g. refs. 9⇓⇓–12). The theory of market ecology provides a conceptual framework for understanding such models. Our goal is not to construct a better model of financial markets, but rather to show how ideas from ecology can be used to interpret market phenomena and predict market behavior.

Here we study a stylized toy market model with three trading strategies. We approach the problem in the same way that an ecologist would study three interacting species. We study how the average returns of the strategies depend on the wealth invested in each strategy, how their wealth evolves through time under reinvestment, and how their endogenous time evolution causes the market to malfunction.

We show that, with realistic parameters, evolution toward market efficiency is very slow. The expected deviations from efficiency are, in some sense, small, but they persist even in the long term, and cause extended deviations from fundamental values and excess volatility (which, in extreme cases, becomes market instability). Our study provides a simple example of how analyzing markets in these terms and tracking market ecosystems through time could give regulators and practitioners better insight into market behavior.

## Model Description

The structure of the model is schematically summarized in Fig. 1. There are two assets, a stock and a bond. The bond trades at a fixed price and yields

We use market clearing to set prices. The stock has a fixed supply Q, but the excess demand *Materials and Methods*.

The size of a trading strategy is given by its wealth

A trading strategy is defined by its trading signal

### Investment Strategies.

We study three typical trading strategies, which we call value investors, trend followers, and noise traders. We intentionally make all strategies boundedly rational, that is, they work from limited information, and their strategies are not optimal. We use a representative agent hypothesis, treating each strategy as though it were only used by a single fund; however, these should be thought of as representing all investors using these strategies. We now describe each strategy in turn.

### Value investors.

Value investors observe the dividend process and use a model to derive the value of the stock. They seek to hold more of the stock when it is undervalued and hold more of the bond when the stock is overvalued. The parameters of their model are estimated based on historical dividends.

The fundamental value *required rate of return*, with

The valuation

We define the trading signal for the value investor as the difference in log prices between the estimated fundamental value

### Trend followers.

Trend followers expect that historical trends in returns continue into the short-term future. Several variants exist in the literature, including the archetypal trend follower that we use here (3, 10, 18⇓–20). There is evidence to suggest that trend-based investment strategies are profitable over long time horizons, and ref. 21 argues that investors earn a premium for the liquidity risk associated with stocks with high momentum (momentum trading is a synonym for trend following).

The trend-following strategy extrapolates the trend in price between recently realized prices

### Noise traders.

Noise traders represent nonprofessional investors who do not track the market closely. Their transactions are mostly for liquidity, but they are also somewhat aware of value, so they are slightly more likely to buy when the market is undervalued and slightly more likely to sell when the market is overvalued (this is necessary so that their positions do not diverge over long periods of time).

The signal function of our noise traders contains the product of the value estimate

The parameters of the model are summarized later (see Table 5). We have chosen them for an appropriate trade-off between realism and conceptual interest, for example, so that each strategy has a region in the wealth landscape where it is profitable. We will see that there are a few ways in which the properties of these strategies do not match those observed in real markets (the trend strategy is very short term, the value investor uses somewhat high leverage, and, at equilibrium, the noise trader has a surprisingly low return-to-risk ratio). Nonetheless, they are realistic enough to make our key points.

## Results

### Density Dependence.

An ecosystem is said to have density dependence if its characteristics depend on the population sizes of the species, as is typically the case. Similarly, a market ecosystem is density dependent if its characteristics depend on the wealths of the strategies (both its own wealth and that of other strategies). The toy market ecosystem that we study here is strongly density dependent.

When the core ideas in this paper were originally introduced in ref. 3, prices were formed using a market impact function, which translates the aggregate trade imbalance at any time into a shift in prices. This can be viewed as a local linearization of market clearing. The use of a market impact function suppresses density dependence and neglects nonlinearities that are important for understanding market ecology.

With market clearing, there is strong density dependence. This is evident in Fig. 2, which shows which strategy makes the highest profits as a function of the relative size of each of the three strategies. To control the size of each strategy, we turn off reinvestment, and instead replenish the wealth of each strategy at each step as needed to hold wealth constant. We then systematically vary the wealth vector

Roughly speaking, the profitability of the dominant strategy divides the wealth landscape into four distinct regions. Trend followers dominate at the bottom of the diagram, where their wealth is small. Value investors dominate on the left side of the diagram, where their wealth is small, and noise traders dominate on the right side of the diagram, where their wealth is small. There is an intersection point near the center where the returns of all three strategies are the same. In addition, there is a complicated region at the top of the diagram, where no single strategy dominates. The turbulent behavior in this region comes about because the wealth invested by trend followers is large, and the price dynamics are unstable. We do not regard this region as realistic, except perhaps in rare extreme market conditions.

A quantitative snapshot of the average returns and volatility is given in Fig. 2*B*, where we hold the size of the noise traders constant at

### Adaptation and the Slow Approach to Market Efficiency.

We now investigate the dynamics of the market ecosystem. To understand how the wealth of the strategies evolves through time, we allow reinvestment and plot trajectories corresponding to the average return at each wealth vector w. This is done by averaging over many different runs. The result is shown in Fig. 3*A*. Most of the wealth trajectories in the diagram evolve toward a fixed point where the wealths of the strategies no longer change. There is also a region at the top of the diagram where the dynamics are more complicated (due to instabilities) and a region in the lower left corner where the ecosystem evolves toward the boundary of the simplex.

At the fixed point, the annual returns to the three strategies are all equal to *efficient equilibrium*. We say “loosely” because the volatilities of the strategies are

The large central region of initial conditions that are attracted to the efficient equilibrium gives a misleading impression of a smooth evolution toward a state of market efficiency. In fact, the dynamics are noisy and stray far from the deterministic dynamics shown in Fig. 3*A*. Tracking a few individual trajectories, as we do in Fig. 3*B*, demonstrates that the dynamics are dominated by noise, due to the statistical uncertainty in the performance of the strategies. The typical trajectories bear little correspondence to the deterministic trajectories of Fig. 3*A*, and the convergence to the efficient equilibrium is weak.

To get a feeling for the asymptotic wealth distribution, we sample the space of initial wealth uniformly, simulate the ecosystem dynamics under reinvestment with *C*. The deviations from the efficient equilibrium point are substantial, often more than 20%. Furthermore, the evolution toward the asymptotic distribution is exceedingly slow: Each trajectory in Fig. 3*B* spans 200 y of simulated time. There are substantial changes in the relative wealth taking place over time scales that are longer than a century.

The long time scale to reach efficiency observed here matches with the estimate made by Farmer in ref. 3. In the ideal case of a stationary market and independent and identically distributed (I.I.D.) normally distributed returns, the time required to detect excess performance

To demonstrate that statistical uncertainty is the dominant factor determining the approach to efficiency, we did a series of experiments that are described in detail in *SI Appendix*, section 4. As the reinvestment rate f in Eq. **2** varies in the range

The absence of autocorrelation in price returns is an indicator of market efficiency. Efficient price returns should have an autocorrelation that is reasonably close to zero (close enough that it is not possible to make statistically significant excess profits). In Fig. 3*D*, we plot the one-time-step autocorrelation of returns across the wealth landscape. There is a striking white band across the center of the simplex, corresponding to zero autocorrelation. This happens when trend followers invest about 40% of the total wealth, thereby eliminating the autocorrelation coming from the dividend process. The wealth of trend followers fluctuates, even at very long times, and, consequently, the degree of autocorrelation in price returns fluctuates as well.

### Community Matrix.

The community matrix is a tool used in ecology to describe the pairwise effects of the population of species j on the population growth rate of species i (24, 25). As originally pointed out by Farmer (3), who called it the gain matrix, an analogous quantity is also useful for interpreting the behavior of market ecosystems.

Let

Because we do not have a differentiable model for our toy market ecosystem, we compute the community matrix numerically using finite differences (see *Materials and Methods*). The community matrix is strongly density dependent. If we compute the community matrix near the equilibrium point in the center of the simplex, we get the result shown in Table 1.

The diagonal entries of the community matrix are all negative, indicating that the strategies are competitive with themselves. This means that their average returns diminish as the strategy gets larger, causing what is called *crowding* in financial markets. We already observed this in Fig. 2. Interestingly, however, the size of the diagonal terms varies considerably, from

All of the other entries of the community matrix are positive, indicating mutualism. This implies that every strategy benefits from an increase in the wealth of any of the other strategies. While we initially found it surprising that all of the strategies could have mutualistic interactions with each other, on reflection, this makes sense: The ecosystem is, by definition, efficient at the equilibrium, and driving any of the strategies away from equilibrium creates an inefficiency that provides a profit opportunity for the other two strategies.

The community matrix is density dependent. If we compute the community matrix at the wealth vector given in Table 2, where the value investors are dominant, there is a shift in the pairwise community relations. As before, all of the terms in the row corresponding to the noise traders are small, indicating that the noise traders are not strongly affected by other strategies, and that they compete only weakly with themselves. This should not be surprising—the noise traders’ strategy is mostly random, and is less influenced by prices than the other two strategies. Value investors, who have the majority of the wealth in this case, still strongly benefit from an increase in the wealth of noise traders (although less so than at the equilibrium). However, there is now a negative

The Lotka–Volterra equations, which describe how the populations in an idealized predator–prey system evolve through time, are perhaps the most famous equations in population biology. Their surprising result is that, at some parameter values, they have solutions that oscillate indefinitely. Using the assumption of no density dependence, Farmer derived Lotka–Volterra equations for market ecology (3). Our results here indicate that the density dependence in this system is so strong that simple Lotka–Volterra equations are a poor approximation, at least for this system. The existence of oscillating solutions in financial ecosystems remains an open question.

### Food Webs and Trophic Level.

The food web provides an important conceptual framework for understanding the interactions between species. If lions eat zebras, and zebras eat grass, then the population of lions is strongly affected by the density of grass, and, similarly, the density of grass depends on the population of lions, even though lions have no direct interactions with grass. The trophic level of a species is, by definition, one level higher than what it eats, so, in this idealized system, grass has trophic level one, zebras have trophic level two, and lions have trophic level three.

The existence of animals with more-complicated diets, such as omnivores and detritivores, means that real food webs are never this simple. If we let

We can also compute trophic levels for the strategies in an ecosystem. We define the analogous quantity

Eqs. **10** and **11** allow us to compute trophic levels. At the efficient equilibrium, the trophic levels are (noise trader = 1.00, value investor = 2.00, trend follower = 2.99). The proximity to integer values is because there are only three strategies with similar wealth levels: The noise trader cannot profit from either the value investor or the trend follower, the value investor profits from the noise trader, and the trend follower profits just a little from the noise trader but quite a lot from the value investor—see the discussion in *SI Appendix*. However, away from the equilibrium, where the wealths of the three strategies are substantially unequal, this changes. In order to better understand the density dependence, we compute trophic levels at each point in the wealth landscape. For three strategies, there are

The computation of trophic levels is complicated by the fact that, for some wealth vectors, there are cycles in the food web. For example, for **9** does not converge, and the trophic levels become undefined. Cycles in the trophic web are not unique to markets—they can also occur in biology, for example, due to cannibalism or detritovores.

A comparison of Figs. 4 and 3*C* makes it clear that, at long times (after transients have died out), the system divides its time between the region in which the trophic levels are ordered as (noise trader, value investor, trend follower), as they are at the equilibrium point, and the region where there are cycles, where the trophic levels are undefined.

### How Ecosystem Dynamics Cause Market Malfunction.

The wealth dynamics of the market ecosystem help explain why the market malfunctions and illuminate the origins of excess volatility and mispricing, that is, deviations of prices from fundamental values. Both in real markets (2) and in the agent-based models mentioned earlier (including our model here), volatility and mispricings change endogenously with time—there are eras where they are large and eras where they are small. Volatility varies intermittently, with periods of low volatility punctuated by bursts of high volatility, called *clustered volatility*. The standard explanations for clustered volatility are fluctuating agent populations (9, 10, 27) and leverage (28). We focus here on the first explanation; while we also observe that clustered volatility increases with increasing leverage, we have not investigated this, in detail, here.

Fig. 5*A* presents the variation of the volatility across the wealth landscape. The landscape can roughly be divided into two regions. On the lower right, there is a flat “low-volatility plain” occupying most of the landscape. On the upper left, there is a high-volatility region, with a sharp boundary between the two. As we will now show, excursions into the high-volatility region cause clustered volatility. A similar story holds for mispricing.

Fig. 5*A* shows a sample trajectory that begins at the efficient equilibrium and spans 200 y. This sample is representative of trajectories that fluctuate around the equilibrium point indefinitely. We choose a 200-y time span to show the scale at which spikes in volatility and mispricing occur. The statistical fluctuations in the performance of the three strategies act as noise, causing large excursions away from equilibrium. The trajectory mostly remains on the volatility plain, but there are several epochs where it ventures into the high-volatility region, causing bursts of high volatility.

The wealth dynamics have strong explanatory power for both mispricing and volatility. This is illustrated in Table 3, where we perform regressions of the strategies’ wealth against volatility using daily values for the time series shown in Fig. 5*A*. For volatility, *t* statistics are more than an order of magnitude smaller. In Fig. 5 *B* and *C*, we compare a time series of the predicted volatility and predicted mispricing against the actual values. The predictions are very good.

Nonetheless, due to their effect on the population of value investors, the net effect of the trend followers on market malfunctions is not obvious. In Fig. 5*D*, we plot the wealth of value investors and trend followers. The strong mutualism predicted by the community matrix is clearly evident from the fact that the wealth of trend followers and value investors rise and fall together. However, their dynamics are quite different—there are several precipitous drops in the value investors’ wealth, whereas the trend followers tend to take more-gradual losses. As predicted, the highest-volatility episodes happen when the value investors’ wealth drops sharply while the trend followers’ wealth is high.

## Discussion

Our analysis here demonstrates how understanding fluctuations of the wealth of the strategies in the ecosystem can help us predict market malfunctions such as mispricings and endogenously generated clustered volatility. The toy model that we study here is simple and highly stylized, but it illustrates how one can import ideas from ecology to better understand financial markets. Our analysis of this model illustrates several properties of market ecosystems that we hypothesize are likely to be true in more general settings.

Concepts from ecology give important insights into how deviations from market efficiency occur and how they affect prices. While the market may be close to efficiency in the sense that the excess returns to any given strategy are small, nonetheless, there can be substantial deviations in the wealth of different strategies, that can cause excess volatility and market instability.

Market ecology is a complement rather than a substitute for the theory of market efficiency. There are circumstances, such as pricing options, where market efficiency is a useful hypothesis. Market ecology, in contrast, provides insight into how and why markets deviate from efficiency, and what the consequences of this are. It can be used to explain the time dependence in the returns of trading strategies, and, in some cases, it can be used to explain market malfunctions. One of our main innovations here is to demonstrate how to compute the community matrix and the trophic web, which provide insight into the interactions of strategies. Surprisingly, at the efficient equilibrium, we find that all three strategies have mutualistic relationships to one another.

There are, so far, only a few examples of empirical studies of market ecology (29, 30). This is because such a study requires counterparty identifiers on transactions in order to know who traded with whom. Trying to study a market ecosystem without such data is like trying to study a biological ecosystem in which one can observe that an animal ate another animal, without any information about the types of animals involved. Unfortunately, for markets, such data are difficult for most researchers to obtain.

Regulators potentially have access to the balance sheets of all market participants, which can allow them to track, in detail, the ecology of the markets they regulate. Ideas such as those presented here could provide valuable insight into when markets are in danger of failure, and make it possible to construct models for the ecological effect of innovations, for example, the introduction of new types of assets such as mortgage-backed securities.

One of our most striking results is that the approach to efficiency is highly uncertain and exceedingly slow. As already pointed out, this should be obvious from a straightforward statistical analysis, but it is not widely appreciated. Our results demonstrate this dramatically, and they indicate that, even in the long term, we should expect large deviations from efficiency.

There are many possible extensions to this work. An obvious follow-up is to explore a larger space of strategies, or to let new strategies evolve in an open-ended way through time. Does the process of strategy innovation tend to stabilize or destabilize markets? Another follow-up is to construct a model that is empirically validated against data with counterparty identifiers. Our analysis here provides concepts and methods that could be used to interpret the behavior of real world examples.

## Materials and Methods

### Accounting and Balance Sheets.

The funds in our model use a stylized balance sheet that is presented in Table 4. Funds are endowed with equity capital *leverage*, meaning using borrowed funds to purchase additional risky assets, by borrowing cash L. Public regulatory filings of US institutional fund managers indicate common leverage ratios between 1 and 10.^{†} The interest rate that applies to cash holdings, loans, and margin is the same as the risk-free rate from holding the bond.

Wealth is calculated as

### Software.

The simulation in this paper builds on the Economic Simulation Library.^{‡}

## Data Availability

The model and code to run the experiments have been deposited in GitHub (https://github.com/INET-Complexity/market-ecology).

## Acknowledgments

We thank Klaus Schenk-Hoppé, Robert MacKay, Michael Wooldridge, Rama Cont, Alissa Kleinnijenhuis, and Cephas Svosve for enlightening discussion during the development of this work. We acknowledge funding by J.P. Morgan AI Faculty Awards, Baillie-Gifford, and the Rebuilding Macroeconomics program, funded by the Economic and Social Research Council. This research was partially supported by Trustworthy AI - Integrating Learning, Optimisation and Reasoning (TAILOR) (https://tailor-network.eu/), a project funded by European Union Horizon 2020 research and innovation program under Grant Agreement 952215.

## Footnotes

- ↵
^{1}To whom correspondence may be addressed. Email: maarten.scholl{at}cs.ox.ac.uk.

Author contributions: M.P.S., A.C., and J.D.F. designed research, performed research, contributed new analytic tools, and wrote the paper.

The authors declare no competing interest.

↵*B. LeBaron, Building the Santa Fe artificial stock market. www2.econ.iastate.edu/tesfatsi/blake.sfisum.pdf (2002).

↵

^{†}The Electronic Data Gathering, Analysis, and Retrieval system (EDGAR), publicly available at https://www.sec.gov/edgar.shtml, Securities and Exchange Commission.↵

^{‡}An open-source library for agent-based modeling which is publicly accessible at https://github.com/INET-Complexity/ESL.This article is a PNAS Direct Submission.

This article contains supporting information online at https://www.pnas.org/lookup/suppl/doi:10.1073/pnas.2015574118/-/DCSupplemental.

Published under the PNAS license.

## References

- ↵
- D. M. Cutler,
- J. M. Poterba,
- L. H. Summers

- ↵
- R. J. Shiller

- ↵
- ↵
- A. Lo

- ↵
- T. Hens,
- K. R. Schenk-Hoppé

- ↵
- ↵
- N. Beale,
- R. Gunton,
- K. L. Bashe,
- H. S. Battey,
- R. S. MacKay

- ↵
- S. J. Grossman,
- J. E. Stiglitz

- ↵
- W. B. Arthur,
- S. N. Durlauf,
- D. Lane

- W. B. Arthur,
- J. H. Holland,
- B. D. LeBaron,
- R. G. Palmer,
- P. Tayler

- ↵
- ↵
- ↵
- C. Chiarella

- ↵
- B. LeBaron

- ↵
- E. F. Fama,
- K. R. French

- ↵
- L. Tesfatsion,
- K. L. Judd

- B. LeBaron

- ↵
- M. J. Gordon,
- E. Shapiro

- ↵
- F. Black,
- M. Scholes

- ↵
- W. A. Brock,
- C. H. Hommes

- ↵
- J. D. Farmer,
- S. Joshi

- ↵
- B. LeBaron

- ↵
- ↵
- J.-P. P. Bouchaud et al.

- ↵
- D. Cherkashin,
- J. D. Farmer,
- S. Lloyd

- ↵
- ↵
- M. Novak et al.

- ↵
- J. L. Bronstein

- ↵
- R. Franke,
- F. Westerhoff

- ↵
- S. Thurner,
- J. D. Farmer,
- J. Geanakoplos

- ↵
- A. Kirilenko,
- A. S. Kyle,
- M. Samadi,
- T. Tuzun

- ↵
- F. Musciotto,
- L. Marotta,
- J. Piilo,
- R. N. Mantegna

## Citation Manager Formats

## Article Classifications

- Social Sciences
- Economic Sciences