Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology
Letter

Does model-free forecasting really outperform the true model?

Florian Hartig and Carsten F. Dormann
PNAS October 15, 2013 110 (42) E3975; https://doi.org/10.1073/pnas.1308603110
Florian Hartig
Department of Biometry and Environmental System Analysis, University of Freiburg, 79106 Freiburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: florian.hartig@biom.uni-freiburg.de
Carsten F. Dormann
Department of Biometry and Environmental System Analysis, University of Freiburg, 79106 Freiburg, Germany
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site

This Letter has a Reply and related content. Please see:

  • Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data
  • Reply to Hartig and Dormann: The true model myth
  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

Estimating population models from uncertain observations is an important problem in ecology. Perretti et al. observed that standard Bayesian state–space solutions to this problem may provide biased parameter estimates when the underlying dynamics are chaotic (1). Consequently, forecasts based on these estimates showed poor predictive accuracy compared with simple “model-free” methods, which lead Perretti et al. to conclude that “Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data.” However, a simple modification of the statistical methods also suffices to remove the bias and reverse their results.

The instability of both maximum-likelihood and Bayesian inference for chaotic models has been recognized before (2). Deterministic chaos produces quasirandom trajectories that are extremely sensitive to changes in parameters and initial conditions. Likelihoods are therefore often highly irregular in shape (3). Moreover, if there is sufficient noise in either process or observation, “true” chaotic parameters may have lower likelihood than alternative parameters with stable trajectories, effectively devaluating maximum-likelihood as a consistent estimator for chaotic dynamical systems (4). The reason is that, for chaotic models, the smallest amount of noise leads to diverging population trajectories, so that simulations from the same parameters may be further apart from each other in the long run than from a stable trajectory at the time-series mean (Fig. 1B).

Fig. 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Fig. 1.

Comparison of median posterior parameter estimates for different values of the growth rate r (A) and predictive error for r = 3.7 (B) for the estimation method used by Perretti et al. (1) and our alternative estimation method. For B, we additionally show the predictive error of the true parameters. Predictive error is measured by standardized root mean-square error (RMSE) (1). The shaded area depicts the 95% confidence interval. Note that biased, but stable r values estimated by Perretti et al. have larger short-term but smaller long-term error than the true values.

There are a number of known methods to bypass these problems. Using summary statistics, potentially in an approximate Bayesian framework, is one of them (3, 5). In the case of the chaotic models presented by Perretti et al. (1), however, there is a simpler solution. Because the bias arises from the long-term divergence of the chaotic population dynamics, a simple solution is to divide the time series in smaller subsets and fit the model to those individually (2). We applied this method to the example of the logistic model used in Perretti et al. (1) and obtained parameter estimates that are virtually unbiased. Technical details and code are available at http://arxiv.org/abs/1305.3544. The resulting median parameter estimates show similar dynamics and predictive uncertainty as the “true” model, with lower short-term error than the statistical alternatives (Fig. 1). It seems likely to us that similar results could also be obtained for the other model types examined in Perretti et al. (1).

Perretti et al. (1) highlight a statistical problem in inferring parameters of chaotic dynamics, which is very important, as it is conceivable that the described bias may have gone unnoticed when working with empirical data only. However, our simulations question Perretti et al.’s conclusion that these problems fundamentally render “model-free” approaches superior. As we show, using a simple modification of the statistical method provides a better solution to the problem, without having to give up other advantages of mechanistic models that might also benefit forecasting in the long run, such as transferability and theoretical understanding.

Footnotes

  • ↵1To whom correspondence should be addressed. E-mail: florian.hartig{at}biom.uni-freiburg.de.
  • Author contributions: F.H. designed research; F.H. performed research; F.H. and C.F.D. analyzed data; and F.H. and C.F.D. wrote the paper.

  • The authors declare no conflict of interest.

References

  1. ↵
    1. Perretti CT,
    2. Munch SB,
    3. Sugihara G
    (2013) Model-free forecasting outperforms the correct mechanistic model for simulated and experimental data. Proc Natl Acad Sci USA 110(13):5253–5257.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Pisarenko VF,
    2. Sornette D
    (2004) Statistical methods of parameter estimation for deterministically chaotic time series. Phys Rev E Stat Nonlin Soft Matter Phys 69(3 Pt 2):036122.
    OpenUrlCrossRefPubMed
  3. ↵
    1. Wood SN
    (2010) Statistical inference for noisy nonlinear ecological dynamic systems. Nature 466(7310):1102–1104.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Judd K
    (2007) Failure of maximum likelihood methods for chaotic dynamical systems. Phys Rev E Stat Nonlin Soft Matter Phys 75(3 Pt 2):036210.
    OpenUrlCrossRefPubMed
  5. ↵
    1. Hartig F,
    2. Calabrese JM,
    3. Reineking B,
    4. Wiegand T,
    5. Huth A
    (2011) Statistical inference for stochastic simulation models—Theory and application. Ecol Lett 14(8):816–827.
    OpenUrlCrossRefPubMed
View Abstract
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Does model-free forecasting really outperform the true model?
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
Citation Tools
True model outperforms model-free forecasting
Florian Hartig, Carsten F. Dormann
Proceedings of the National Academy of Sciences Oct 2013, 110 (42) E3975; DOI: 10.1073/pnas.1308603110

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
True model outperforms model-free forecasting
Florian Hartig, Carsten F. Dormann
Proceedings of the National Academy of Sciences Oct 2013, 110 (42) E3975; DOI: 10.1073/pnas.1308603110
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 116 (49)
Current Issue

Submit

Sign up for Article Alerts

Article Classifications

  • Biological Sciences
  • Population Biology

Jump to section

  • Article
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

Modulating the body's networks could become mainstream therapy for many health issues. Image credit: The Feinstein Institutes for Medicine Research.
Core Concept: The rise of bioelectric medicine sparks interest among researchers, patients, and industry
Modulating the body's networks could become mainstream therapy for many health issues.
Image credit: The Feinstein Institutes for Medicine Research.
Adaptations in heart structure and function likely enabled endurance and survival in preindustrial humans. Image courtesy of Pixabay/Skeeze.
Human heart evolved for endurance
Adaptations in heart structure and function likely enabled endurance and survival in preindustrial humans.
Image courtesy of Pixabay/Skeeze.
Viscoelastic carrier fluids enhance retention of fire retardants on wildfire-prone vegetation. Image courtesy of Jesse D. Acosta.
Viscoelastic fluids and wildfire prevention
Viscoelastic carrier fluids enhance retention of fire retardants on wildfire-prone vegetation.
Image courtesy of Jesse D. Acosta.
Water requirements may make desert bird declines more likely in a warming climate. Image courtesy of Sean Peterson (photographer).
Climate change and desert bird collapse
Water requirements may make desert bird declines more likely in a warming climate.
Image courtesy of Sean Peterson (photographer).
QnAs with NAS member and plant biologist Sheng Yang He. Image courtesy of Sheng Yang He.
Featured QnAs
QnAs with NAS member and plant biologist Sheng Yang He
Image courtesy of Sheng Yang He.

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Classics
  • Front Matter
  • Teaching Resources
  • Anthropology
  • Chemistry
  • Physics
  • Sustainability Science

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Press
  • Site Map
  • PNAS Updates

Feedback    Privacy/Legal

Copyright © 2019 National Academy of Sciences. Online ISSN 1091-6490