# Subjective expectation of rewards can change the behavior of smart but impatient foragers

^{a}Departamento de Física, Universidade Federal do Paraná, Curitiba-PR, 81531-980, Brazil;^{b}Laboratório de Física Teórica e Computacional, Departamento de Física, Universidade Federal de Pernambuco, Recife-PE, 50670-901, Brazil;^{c}Department of Physics and National Institute of Science and Technology of Complex Systems, Universidade Federal do Rio Grande do Norte, Natal-RN, 59078-970, Brazil

See allHide authors and affiliations

Search efficiency can be a matter of life and death in biological encounter processes. Animals that fail to explore their environments efficiently suffer heightened risk of earlier death (1). Some theoretical aspects of foraging are closely related to the random search problem of determining the best strategy for optimizing the search when (*i*) targets are distributed in unknown spatial locations and (*ii*) learning and other skills (such as long-range ability to detect prey) do not significantly alter the choice of long-term strategy. Also, it is well known (1) that foragers that learn usually outperform foragers that do not. However, it is not yet well established how foraging patterns change when foragers are cognitively complex. In PNAS, Namboodiri et al. (2) propose an interesting framework to understand possible changes in such spatial patterns.

Their plausible premise, assuming a smart forager, is that the searcher can attempt to improve future outcomes by determining, through exploratory trial and error, a good strategy. The forager searches by sampling the environment and using “different step lengths so as to maximize the ability to detect differences in reward distributions associated with each step length.” The model by Namboodiri et al. (2) predicts that animals attempting to learn optimally from their landscapes will have asymptotically power law-distributed move lengths *x*. Essentially, such a probability density function (PDF) *t* is the delay time for access to the reward. The case *i* and *ii* above hold true.

## The Virtues of Impatience

Temporal discounting is ubiquitous (5, 6). It is the preference for a reward now or soon rather than the same (or even a somewhat higher) reward later. In economics, temporal discounting is conventionally assumed to be exponential, which is the basis of compound interest (6). For example, if the rate of inflation is fixed, cash devalues exponentially. Empirical studies of discounting choices made by real animals showed that exponential discounting is not necessarily standard (7⇓⇓⇓⇓⇓–13). Many animals seem to use hyperbolic discounting, humans included.

Specifically, for a discounting function *μ* and smaller

Hyperbolic functions are power law-tailed. Further extensions of the hyperbolic function have been discussed, such as the *q*-exponential time discount model (15), in which one considers

Namboodiri et al. (2) exploit the subjective value *r*, which, in the case of constant error in the perception of time for each flight (noiseless temporal representation), is given by

Suppose that individual step lengths’ *x*’s (along straight lines and of constant speed *x* in the more general case, for traveling a distance *x* (since

Let us reduce the inherent randomness in *M* fixed. Define then *m*-th value obtained for *r* when

Now, the question is how to minimize the global (i.e., taking into account all of the *M*. The minimization is accomplished by imposing that *n*, with λ being the Lagrange multiplier. Thus,

Finally, because, by definition, **1** that*x*, Eq. **2** results in the PDF:

The proposed framework of Namboodiri et al. (2), summarized in Eq. **3**, is, in principle, able to account for the powerlaw tails in the move lengths of foraging animals (usually taken as *x*’s has received overwhelmingly empirical support in the past decade or so (e.g., see refs. 3, 18, and 19). Interestingly, combinations of memory skills and learning from the environment have also been explored in foraging models, leading to optimal Lévy statistics (20⇓⇓⇓–24). This issue, however, still remains somewhat controversial, including the debate in the context of the Lévy flight foraging hypothesis (24⇓⇓–27).

An initial technical observation concerns the quantity *r*, the reward. The assumption that the related function *x* is, of course, plausible in different situations. But certainly, a more general *c* are in order. First, for large enough *x*, the PDF for step lengths given in Eq. **3** does behave like a pure power law **3** experimentally, one should be able to test (with significant statistics) relatively short steps. A particularly interesting and curious result is the one in figure 3C of ref. 2. The hyperbolic-like function (with two free parameters, *μ* and *c*) seems to be a better fit model than a truncated power law for data of single shark individuals, specially for the long steps, when, in principle, both distributions should closer agree. This finding may indicate that learning skills for sharks are indeed important. Second, power laws are the asymptotic limit of Lévy α-stable distributions, which constitute the theoretical support for the Lévy flight hypothesis (3). For optimization purposes, long steps matter the most (4), hence a common practice (but hardly highlighted in the literature) is to suppose power laws as the basic PDFs (3). Nevertheless, if one assumes the actual full expressions for the Lévy distributions, a saturation parameter like *c* in Eq. **3** would be present. [For example, the Cauchy, which is a Lévy stable distribution with **3** for *c* is needed to test whether or not it is compatible with a *c* here.

## A New Explanation for Lévy Flights

The findings reported by Namboodiri et al. (2) fit into the broader context of prior studies. There have thus far been three basic approaches to explain power law-tailed distributions of move lengths in foraging.

Perhaps the simplest explanation, now a minority view among specialists, is that power law-tailed distributions are due either to experimental error or to faulty data analysis (29⇓–31). However, since the seminal works by Sims et al. (18) and Humphries et al. (19), which applied improved statistical methods of data analysis to more than

The second explanation is the Lévy flight foraging hypothesis that we proposed in 1999 with S. Buldyrev, S. Havlin, and H. E. Stanley (4). According to this hypothesis, organisms must have evolved via natural selection to exploit the optimal search strategies provided by truncated power law Lévy-like distributions of move lengths under scarce information on the environment and sparse conditions. Given a clear ecological scenario, Lévy flights not only optimize (33), but the resulting power law behavior is not already contained in the assumptions of the hypothesis (i.e., it is not a case of “power law in, power law out”).

The third explanation is emergence (32, 33). The movement of foragers in disordered environments can mimic a Lévy flight pattern due to the interaction between the searcher and the landscape. It is known that such emergence does occur in specific geometries (34) or in some kinds of fractal landscapes (18). Moreover, if the foraging strategy has two search modes, of intensive searches that alternate with relocation events, then it is possible for the move lengths to reflect more than one characteristic scale (32). Multiscale behavior is similar to scale-free behavior, and both are related (to differing extents) to power laws (35).

The results of Namboodiri et al. (2) raise a new possibility to explain the observed power law tails in move length distributions. Factors that rely on memory and learning can influence the subjective expectation of rewards during the search. These psychological aspects can lead to the best (or just good enough) strategy to follow while foraging. In fact, a too-low expectation can even trigger drastic changes in the usual hunting habits, as recently reported for big felines (36).

Surely, we can eliminate temporal discounting as the correct explanation for cognitively too simple foragers and when the resources in the environment are so dynamic that past information cannot be of any help for future exploration. However, this proposed mechanism for animals that move very intelligently probably should play a relevant role in the searching behavior. [Experiments with humans (2) point to the importance of temporal costs while performing spatial exploration tasks. However, Namboodiri et al. (2) prudently mention the necessity of more tests, generating larger datasets (e.g., the cumulative distribution function in figure 2C of ref. 2 clearly cannot be fitted by either a power law or a hyperbolic distribution).]

Finally, in a wider view, one might argue that these results do not contradict the Lévy flight foraging hypothesis. Power law distribution of step lengths results in optimal search outcomes (3). So, individuals possessing learning expertise and whose original function

## Acknowledgments

We thank the CNPq and FACEPE for funding.

## References

- ↵.
- Méndez V,
- Campos D,
- Batumeus F

- ↵.
- Namboodiri VMK,
- Levy JM,
- Mihalas S,
- Sims DW,
- Shuler MGH

- ↵.
- Viswanathan GM,
- da Luz MGE,
- Raposo EP,
- Stanley HE

- ↵
- ↵.
- Stephens DW,
- Krebs JR

- ↵.
- Glimcher PW,
- Fehr E

- ↵
- ↵.
- Green L,
- Myerson J

- ↵
- ↵.
- Simpson CA,
- Vuchinich RE

- ↵.
- Murphy JG,
- Vuchinich RE,
- Simpson CA

- ↵
- ↵
- ↵.
- Sozou PD

- ↵
- ↵.
- Loewenstein G,
- Prelec D

- ↵.
- Green L,
- Fry AF,
- Myerson J

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Boyer D,
- Walsh PD

- ↵
- ↵.
- James A,
- Plank MJ,
- Edwards AM

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵.
- Reynolds AM

- ↵

## Citation Manager Formats

## Article Classifications

- Biological Sciences
- Ecology

## See related content: