Irreducible imprecision in atmospheric and oceanic simulations
See allHide authors and affiliations

Contributed by James C. McWilliams, April 4, 2007 (received for review March 1, 2007)
Abstract
Atmospheric and oceanic computational simulation models often successfully depict chaotic space–time patterns, flow phenomena, dynamical balances, and equilibrium distributions that mimic nature. This success is accomplished through necessary but nonunique choices for discrete algorithms, parameterizations, and coupled contributing processes that introduce structural instability into the model. Therefore, we should expect a degree of irreducible imprecision in quantitative correspondences with nature, even with plausibly formulated models and careful calibration (tuning) to several empirical measures. Where precision is an issue (e.g., in a climate forecast), only simulation ensembles made across systematically designed model families allow an estimate of the level of relevant irreducible imprecision.
Extensive experience over several decades shows that computational atmospheric and oceanic simulation (AOS) models can be devised to plausibly mimic the space–time patterns and system functioning in nature. Such simulations provide fuller depictions than those provided by deductive mathematical analysis and measurement (because of limitations in technique and instrumentalsampling capability, respectively), albeit with less certainty about their truth.
AOS models are widely used for weather, general circulation, and climate, as well as for many more isolated or idealized phenomena: flow instabilities, vortices, internal gravity waves, clouds, turbulence, and biogeochemical and other material processes. However, their solutions are rarely demonstrated to be quantitatively accurate compared with nature. Because AOS models are intended to yield multifaceted depictions of natural regimes, their partial inaccuracies occur even after deliberate tuning of discretionary parameters to force model accuracy in a few particular measures (e.g., radiative balance for the top of the atmosphere; horizontal mass flux in the Antarctic Circumpolar Current).
Weather forecasts have both demonstrable skill and appreciable error (1). Climate predictions for anthropogenic global warming are both broadly credible yet mutually inconsistent at a level of tens of percent in such primary quantities as the expected centennial change in largescale, surface air temperature or precipitation (2, 3). Slow, steady progress in model formulations continues to expand the range of plausibly simulated behaviors and thus provides an extremely important means for scientific understanding and discovery. Nevertheless, there is a persistent degree of irreproducibility in results among plausibly formulated AOS models. I believe this is best understood as an intrinsic, irreducible level of imprecision in their ability to simulate nature.
AOS Models
The central component of AOS models is fluid dynamics manifested as turbulence explicitly in the space–time structure of a solution, implicitly in the formulation of turbulent effects (e.g., material mixing), or often both. Turbulence is the epitome of chaos in its evident disorder but also in the partial order of its recurrent patterns. Additional physical, chemical, and biological components are included as appropriate to the target problem. Global, equilibrium target problems include the influences of Earth's rotation, gravity, fluid compositions, and solidsurface configuration, as well as forcings by solar radiation, gravitational tides, and air–sea–land interface fluxes of momentum, heat, and materials. Local and limitedtime target problems include boundary and initial conditions extracted or abstracted from the equilibrium global air–sea dynamical system.
The fruits of AOS are the many forms of intrinsic variability that spontaneously arise through instability of directly forced circulations and have important feedbacks on largescale, lowfrequency fields. Their varieties include coherent atmospheric storms and oceanic eddies, gravitational and rotational waves emitted in internal adjustments, turbulent transports between different locations, and cascades of variance and energy across the space–time spectrum that effect the mixing and dissipation essential for evolution toward balance with the forcing. An AOS can provide reliable realizations for idealized processes. AOS solutions expose structural and dynamical relations among different measurable quantities. They yield space–time patterns reminiscent of nature (e.g., visible in semiquantitative, highresolution satellite images), thus passing a meaningful kind of Turing test between the artificial and the actual. They exhibit emergent behaviors that are not (yet) mathematically deducible from known dynamical equations for fluids, such as a tornado, a Gulf Stream path, or a decadal “teleconnection” relation between western tropical Pacific cumulus convection and a nearly hemispheric standingeddy pattern in surface air pressure.
Atmospheric and oceanic forcings are strongest at global equilibrium scales of 10^{7} m and seasons to millennia. Fluid mixing and dissipation occur at microscales of 10^{−3} m and 10^{−3} s, and cloud particulate transformations happen at 10^{−6} m or smaller. Observed intrinsic variability is spectrally broad band across all intermediate scales. A full representation for all dynamical degrees of freedom in different quantities and scales is uncomputable even with optimistically foreseeable computer technology. No fundamentally reliable reduction of the size of the AOS dynamical system (i.e., a statistical mechanics analogous to the transition between molecular kinetics and fluid dynamics) is yet envisioned.
This reality in nature and computers has given rise to two pervasive AOS practices: (i) AOS solution fields are nonsmooth near the space–time discretization scales (i.e., the “resolution” of the model) imposed on the known governing principles expressed mostly as partial differential equations. (ii) AOS models contain essential parameterizations for unresolved or highly simplified processes whose specifications are not at a fundamental level of known governing principles.
Both practices are coping strategies to span as large a subrange as feasible for the uncomputably broad scale in the most fundamentally grounded fluid dynamical problem. Nonsmoothness is a consequence of trying to separate the resolved scales and processes from the parameterized ones. Among their other roles, parameterizations regularize the solutions on the grid scale by limiting finescale variance (also known as computational noise). This practice makes the choices of discrete algorithms quite influential on the results, and it removes the simulation from the mathematically preferable realm of asymptotic convergence with resolution, in which the results are independent of resolution and all well conceived algorithms yield the same answer. The fruits of AOS are generally considered more satisfactory when the resolution is increased and simultaneously the parameterization schemes are adjusted to maintain an acceptable degree of nonsmoothness, thus maximizing the breadth of the resolved scale range for the available computational capability.
Parameterizations are nonfundamental model elements that represent important aspects of AOS system functioning. Examples are cloud and aerosol microphysics, radiative transfer in heterogeneous media, watershed hydrological routing, subgridscale boundary form stress by topographic roughness, and spatial transport and downscale variance cascade in turbulent boundary layers near the air–sea–land interfaces (also known as eddy diffusion). Parameterization schemes are typically formulated by asserting the desired qualitative effects, devising a mathematical representation to achieve them, and parametrically fitting the rate constants either to independent measurements or to some aspect of the AOS results they control. Within this methodology there is room for plausible alternative parameterization schemes, i.e., nonuniqueness. Some useful parameterizations are also nondifferentiable (e.g., in the transition from a convective to a stably stratified boundary layer), and this compounds AOS nonsmoothness in ways unrelated to resolution convergence.
In a scientific problem as potentially complicated as climate, there is another modeling practice that is increasingly important: AOS models are openended in their scope for including and dynamically coupling different physical, chemical, biological, and even societal processes.
The rationales for coupling are to investigate potentially significant feedbacks (e.g., radiative properties for different airborne crystalline ice structures, changes in air and water inertia due to suspended dust and sediments, and water and other material exchanges with plants and biome evolution) and to achieve ever fuller depictions of Earth's fluid envelope. Besides adding to the overall complexity of AOS models, coupling increases the number of processes with a nonfundamental representation (i.e., similar to a parameterization), because, for the most part, the governing equations are not well determined for the model components other than fluid dynamics. When adding a new coupling link, there is no a priori guarantee of seeing only modest consequences in the AOS solution behavior.
Of course, models can be formulated that eschew these practices. They are mathematically safer to use, but they are less plausibly similar to nature, with suppressed intrinsic variability, important missing effects, and excessive mixing and dissipation rates.
AOS models are therefore to be judged by their degree of plausibility, not whether they are correct or best. This perspective extends to the component discrete algorithms, parameterizations, and coupling breadth: There are better or worse choices (some seemingly satisfactory for their purpose or others needing repair) but not correct or best ones. The bases for judging are a priori formulation, representing the relevant natural processes and choosing the discrete algorithms, and a posteriori solution behavior. Plausibility criteria are qualitative and loosely quantitative, because there are many relevant measures of plausibility that cannot all be specified or fit precisely. Results that are clearly discrepant with measurements or between different models provide a valid basis for model rejection or modification, but moderate levels of mismatch or misfit usually cannot disqualify a model. Often, a particular misfit can be tuned away by adjusting some model parameter, but this should not be viewed as certification of model correctness.
Sensitive Dependence and Structural Instability
AOS models are members of the broader class of deterministic chaotic dynamical systems, which provides several expectations about their properties (Fig. 1). In the context of weather prediction, the generic property of sensitive dependence is well understood (4, 5). For a particular model, small differences in initial state (indistinguishable within the sampling uncertainty for atmospheric measurements) amplify with time at an exponential rate until saturating at a magnitude comparable to the range of intrinsic variability. Model differences are another source of sensitive dependence. Thus, a deterministic weather forecast cannot be accurate after a period of a few weeks, and the time interval for skillful modern forecasts is only somewhat shorter than the estimate for this theoretical limit. In the context of equilibrium climate dynamics, there is another generic property that is also relevant for AOS, namely structural instability (6). Small changes in model formulation, either its equation set or parameter values, induce significant differences in the longtime distribution functions for the dependent variables (i.e., the phasespace attractor). The character of the changes can be either metrical (e.g., different means or variances) or topological (different attractor shapes). Structural instability is the norm for broad classes of chaotic dynamical systems that can be so assessed (e.g., see ref. 7). Obviously, among the options for discrete algorithms and parameterization schemes, and perhaps especially for coupling to nonfluid processes, there are many ways that AOS model equation sets can and will change and hence will be vulnerable to structurally unstable behavior.
A seminal, atmospherically motivated example of chaos is Lorenz's loworder, Galerkin truncation of midlatitude jet and weather dynamical equations (8). It exhibits several bifurcations with respect to changes in the steady forcing amplitude F as a control parameter. The chaotic regime is the quasiperiodic “strange attractor” as a paradigm for sensitive dependence and limited predictability. Its attractor has the phasespace portrait of a butterfly. This model is also structurally unstable in several ways. Transitions between the strange attractor regime and periodic limit cycles are densely intermixed for slightly different F values, and the mostly accurate “balance” approximations to Lorenz's equations have different transitional F values (9). The attractor is even more substantially altered by changing the truncation order of the model (10).
Although we may expect a chaotic AOS model to be structurally unstable, it is difficult to explicitly make this determination. The attractor cannot be fully visualized or measured because the phase space has such a high dimension (i.e., high order). Probability distribution functions (PDFs) (Fig. 1) give at least a rough view of an AOS attractor. There are many aspects to the equation set for a model, most notably in the choices of discrete algorithms, parameterizations, and coupling scope, and these are usually not systematically explored in AOS practices. To do so requires formulating multiple models for a given problem. Even systematic scans in the parameter values of a complicated AOS model are rarely published, although parameter variations are commonly made while tuning a model to improve its plausibility. ^{†}
Nevertheless, I advocate the hypothesis that plausible, chaotic AOS models have important levels of irreducible imprecision due to structural instability resulting from choices among a set of modeling options that cannot be clearly excluded. The level of irreducible imprecision will depend on the context, and this level is likely to be greater the more chaotic and multiply coupled the targeted flow regime is.
Examples of Irreducible Imprecision
To illustrate the issues, first consider a problem in pure fluid dynamics with the incompressible Navier–Stokes equations at large Reynolds number Re (i.e., a parameter for the ratio of advective and viscous rates). Vortices permeate atmospheric and oceanic flows and spontaneously emerge in AOSs. From random, intermediatescale initial conditions, a twodimensional flow evolves without forcing in a spatially periodic domain with a small viscosity coefficient effective only on smaller scales than the initial ones. This type of flow has emergent coherent vortices whose mutual interactions control the longtime evolution (12). The flow is chaotic, and it exhibits sensitive dependence with respect to individual vortices. If one takes the narrow view that the governing equations are nonnegotiable and follows a conservative computational practice by limiting the size of Re to make the viscous dissipation scale (also known as the Kolmogorov scale) much larger than the grid scale, then this system is widely believed, but not proven, to be structurally stable with respect to the only available discretionary modeling choices: the value of Re, the choice of discrete algorithm, the grid resolution, and the particular initialstate realization. Call this the fundamental formulation for the problem.
The fundamentally formulated problem is far from being an AOS to compare with nature. ^{‡} An important mismatch is that its computable versions all have much smaller Re values than is plausible for large and mesoscale atmospheric and oceanic vortices. The mismatch can be addressed partially by following the practices discussed in the second section, namely by allowing solutions to be less smooth on the grid scale and parameterizing the viscous diffusion and dissipation. One approach is to use the class of “quasimonotone” discrete advection operators and dispense with the viscous diffusion component in the model equations. For the fundamental advection–diffusion dynamics, isolated vorticity extrema are provable to be nonincreasing with time. Monotone advection operators are designed to strongly limit or prohibit the occurrence of false extrema in the advected field (here the vorticity, the velocity curl) by using an upstreambias in the finite differencing or otherwise constraining the shape of the field. Monotone advection operators are widely used in AOS models, especially for material concentrations for which spurious negative values are both chemically and biologically impossible and computationally dangerous. The character of the discretization error for monotone operators is a smoothing of the advected field near the grid scale (the opposite of introducing spurious extrema). So they provide a parameterization for mixing and dissipation that automatically changes when the resolution is changed.
In ref. 15, many different monotone operators are used for twodimensional flow to guide the choice of the discrete advection algorithm in an oceanic simulation model. Their set of solutions can also be used as a model ensemble to illustrate structural instability and irreducible imprecision. All of the plausible operator options give solutions with the correct phenomenology (Fig. 2), and all of them give solutions in close agreement at relatively early times, with even better convergence with higher resolution. Sensitive dependence for each model formulation is expressed in the variable number, positions, and amplitudes of individual vortices at a particular later time as a consequence of different initial conditions or model formulation (data not shown). Furthermore, after a sufficient evolution time, the average vortex number, amplitude set, and characteristic shape for the radial decay of vorticity away from central extrema all vary substantially among the different model formulations; e.g., the alternatives in Fig. 2 show that the top row has relatively weak and broad vortex profiles, the middle row has blunt shapes for the extrema, and the bottom row has strong cuspy extrema. For each model, there are important (and dynamically interesting) trends with increasing resolution toward a population with stronger, more abundant, and incrementally smaller vortices. These trends are therefore a meaningful surrogate for what can be inferred to occur with increasing Re in the more physically grounded and structurally stable form of the problem, but the solutions with monotone operators are much farther along in these measures than in the fundamental problem at the same resolution and size of the computation (data not shown). For each model formulation in Fig. 2, the solutions show a plausible consistency with different resolutions (accounting for the trends), but between the formulations, the differences do not strongly diminish with resolution. Expressed in terms of distribution functions for the vorticity field, the different formulations yield similar shapes but with persistent differences with increasing Re (as sketched for a metrical structural instability in Fig. 1). The same conclusions hold for the broader class of operators examined in ref. 15, in which more quantitative measures of operator differences are presented. Insofar as many, if not all, of these operator choices are plausible from the perspectives of algorithmic design and solution behavior, then their differences demonstrate an irreducible imprecision compared with the practically inaccessible truth about fluid dynamics with appropriately large values of Re. And if that truth standard were somehow accessible, the motivations for relevance to nature would push the AOS even higher in its effectively equivalent Re value, again into an inaccessible realm for the fundamental problem.
Pure fluid dynamics may be at the core of the AOS modeling problem, but nature combines fluid physics with other processes, and we must look to more comprehensive model formulations to be able to assess simulation accuracy against the relevant empirical reality. Thus, we can consider the many comparison studies that show a substantial spread among the results from AOS models created by different groups, as well as in the degree of correspondence with observations. Because each of the models is created independently, such model ensembles are more opportunistically assembled than systematically designed. Furthermore, the compared models are typically being reformulated by their creators faster than they can be compared with each other. So the comparisons are more like snapshots of model differences than careful, enduring assessments.
An example is large eddy simulations for cloudtopped surface atmospheric boundary layers. In ref. 16, 15 different models are compared for a regime with trade wind cumuli. Vertical profiles for heat, water, and velocity differ among the models by tens of percent in their means and variances and by similar amounts compared with measurements. Within a single participating model, changes in the advection algorithm and subgridscale turbulence parameterization scheme show qualitative differences in the simulated cloud patterns. And differences between two of the models increase as the resolution increases. Similar characteristics are found in a regime of nocturnal stratus clouds, in which measurements show quantitative discrepancies with the model ensemble at a level comparable to, but not entirely within, the ensemble spread (17).
AOS model comparisons have also been made for the oceanic and atmospheric general circulations. Two separate ensemble comparisons are reported in refs. 18 and 19 for the timeaverage North Atlantic oceanic circulation with commonly specified surface forcing fields. Ref. 20 describes a frequently repeated type of comparison among global atmospheric models by using observationally specified oceanic surfacetemperature fields with seasonal and interannual changes. Again, the ensemble spreads are substantial, and measures of their realism have mixed success in matching measurements.
More famously, the Intergovernmental Panel on Climate Change (IPCC) report (21) shows the spread among climate models for global warming predictions. One of its results is an ensemblemean prediction of ≈3°C increase in global mean surface temperature for doubled atmospheric CO_{2} concentration with an ensemble spread of ≈50% on either side. The predicted value for the climate sensitivity and its intermodel spread have remained remarkably stable throughout the modern assessment era from the National Research Counsel (NRC) in 1979 (22) to the anticipated results in the IPCC Fourth Assessment Report (foreshadowed, e.g., in ref. 3) despite diligent tuning and after great research effort and progress in many aspects of simulation plausibility. An even broader distribution function for the increase in mean surface air temperature is the solution ensemble for a standard atmospheric climate model produced by Internetshared computations (23), but there is a question about how carefully the former ensemble members were selected for their plausibility.
In each of these model–ensemble comparison studies, there are important but difficult questions: How well selected are the models for their plausibility? How much of the ensemble spread is reducible by further model improvements? How well can the spread can be explained by analysis of model differences? How much is irreducible imprecision in an AOS?
Simplistically, despite the opportunistic assemblage of the various AOS model ensembles, we can view the spreads in their results as upper bounds on their irreducible imprecision. Optimistically, we might think this upper bound is a substantial overestimate because AOS models are evolving and improving. Pessimistically, we can worry that the ensembles contain insufficient samples of possible plausible models, so the spreads may underestimate the true level of irreducible imprecision (cf., ref. 23). Realistically, we do not yet know how to make this assessment with confidence.
Implications for AOS Practices and Expectations
An appreciation of the AOS property of sensitive dependence has led to the practice of ensemble weather forecasting on the basis of a set of solutions using initial conditions perturbed around the estimated atmospheric state (24, 25). The evolving spread among the individual forecasts exposes the degree of reliability in the ensemblemean forecast. Similarly an appreciation of the property of structural instability ought to lead to the practice of ensemble AOS modeling on the basis of a set of deliberately varied model formulations, to expose the reliability and precision of the simulated behaviors. ^{§} It can even be useful in a forecast ensemble to include variations in the model formulation if the structure of the longtime attractor manifests in limitedtime integrations relative to the initialstate influence (27). Weather ensembles have a great practical advantage over general circulation and climate ensembles in their repeatable comparability with nature.
Thus, there is a clear imperative to better understand the consequences of the choices that are commonly made in constructing a plausible AOS model. This would allow a clearer assessment of its precision compared with nature. This in turn would better pose the question of whether or not model reformulations might be devised to reduce the imprecision. However, attempting to do these things will substantially increase the task load for AOS modeling. Varying the model formulations in a systematic way is much more difficult than varying the initial conditions with random realizations from a specified spectrum. Assessing the outcome from systematic model reformulations is likely to be much more useful than the present practice of simply comparing available AOS model solutions from different groups. To explore structural instability, fuller measures are needed for the attractor beyond the usual practices of analyzing mean and variance fields, covariance eigenmodes (also known as empirical orthogonal functions), and singlevariate probability distribution functions (PDFs). It is quite laborious to make an AOS model, both in the code construction and in the necessary testing and tuning, but it would not be too difficult to routinely incorporate some alternative algorithms and parameterizations. To further overcome the hurdle of model building, some new techniques for automated, algorithmic procedures are needed to enable many more models to be built and tested. Recipes are needed for the types of model variations to incorporate. Some of the likely ingredients are grid resolution, forcing fields, and parameters; already, these are often explored and sometimes publicly reported by careful modelers. More problematic ingredients are the feasible means of varying discrete algorithms and parameterization schemes systematically, rather than merely opportunistically. ^{¶} Once included in an AOS model, the consequences of various couplings are straightforward to assess by systematically disabling the elements of an AOS model. It is much more daunting, if not impossible, to know a priori the consequences of coupling some new process into dynamical systems as chaotic as general circulation and climate. The AOS agenda is to push ahead and explore plausible additional couplings, again leading to more models.
In the context of a systematically designed AOS model ensemble for a given problem, there can be inspiration, if not guidance, for further model development by examining the nature and causes of the spread among the answers. Outlier results are of particular interest, negatively in identifying candidates for model rejection and positively in implicating high sensitivities in the model formulation. There is also great value in establishing a model hierarchy to accompany the ensemble: deliberate and successive simplifications in the model formulation that retain the essential phenomenon in their answers as a basis both for developing a mechanistic understanding and for further demonstrating robustness (11, 29).
For many purposes that are well demonstrated with present practices, AOS models are very useful even without the necessity of carefully determining their precision compared with nature. These models are structurally unstable in various ways that are not yet well explored, and this implies a level of irreducible imprecision in their answers that is not yet well estimated. Their value as scientific tools is undeniable, and the theoretical limitations in their precision can become better understood even as their plausibility and practical utility continue to improve. Whether or not the irreducible imprecision proves to be a substantial fraction of present AOS discrepancies with nature, it seems imperative to determine what the magnitude of this type of imprecision is.
Acknowledgments
I thank Annalisa Bracco, Michael Ghil, Alex Hall, David Neelin, Leonard Smith, and Bjorn Stevens for valuable discussions. A workshop reported in ref. 30 helped provoke some of the ideas in this essay.
Footnotes
 *Email: jcm{at}atmos.ucla.edu

Author contributions: J.C.M. wrote the paper.

The author declares no conflict of interest.

↵ † Common but mostly anecdotal experiences are that such parameter scans can indicate quite rough modelfitness landscapes that might be seen as an indication and measure of structural instability. (This analogy with ecological fitness landscapes was expressed in a preliminary version of ref. 11.)

↵ ‡ This perspective is expressed in the characterization of an AOS as a “pseudofluid” (13). It is also implicit in the traditional distinction between direct numerical simulation and largeeddy simulation for turbulent flows (e.g., see ref. 14).

↵ § Sensitive dependence and structural instability are humbling twin properties for chaotic dynamical systems, indicating limits about which kinds of questions are theoretically answerable. They echo other famous limitations on scientist's expectations, namely the undecidability of some propositions within axiomatic mathematical systems (Gödel's theorem) and the uncomputability of some algorithms due to excessive size of the calculation (see ref. 26).

↵ ¶ Simple stochastic representations for model variations are probably not germane to the actual effects of alternative discretization schemes and parameterizations, and they have not always been successful in encompassing nature within the modelensemble spread; e.g., ref. 28.
 Abbreviation:
 AOS,
 atmospheric and oceanic simulation.

Freely available online through the PNAS open access option.
 © 2007 by The National Academy of Sciences of the USA
References
 ↵
 ↵
 ↵
 ↵
 ↵

↵
 Andronov AA ,
 Pontryagin LS
 ↵
 ↵
 ↵
 ↵
 ↵
 ↵
 ↵

↵
 Pope SB
 ↵
 ↵
 ↵
 ↵
 ↵
 ↵

↵
 Houghton JT ,
 Ding Y ,
 Griggs DJ ,
 Noguer M ,
 van der Linden PJ ,
 Dai X ,
 Maskell K ,
 Johnson CA

↵
 National Research Council/Ad Hoc Study Group on Carbon Dioxide and Climate
 ↵
 ↵

↵
 Toth Z ,
 Kalnay E

↵
 Penrose R
 ↵
 ↵

↵
 Ghil M

↵
 Emanuel KA ,
 Wyngaard JC ,
 McWilliams JC ,
 Randall DA ,
 Yung YL