Invariance and universality in social agent-based simulations
Abstract
Agent-based simulation models have a promising future in the social sciences, from political science to anthropology, economics, and sociology. To realize their full scientific potential, however, these models must address a set of key problems, such as the number of interacting agents and their geometry, network topology, time calibration, phenomenological calibration, structural stability, power laws, and other substantive and methodological issues. This paper discusses and highlights these problems and outlines some solutions.
Agent-based models are in their infancy in the social sciences (1–3), especially when compared with earlier modeling methods based on classical mathematical analysis (formal deductive models) and statistical or econometric tools (inductive models). Despite their infancy, however, several agent-based simulation projects are beginning to address important research problems across diverse domains of the social sciences, often proposing insightful solutions (refs. 4–8; ref. 8 reprinted in ref. 9). Given their untapped potential, it now seems likely that these new methods will soon start yielding a set of significant contributions in numerous areas of social science as “a third way of doing science” (5).
To realize their promising scientific potential, however, it should be possible to demonstrate that findings obtained from agent-based model simulations are invariant with respect to changes in a critical set of assumptions, parameters, or dimensions in a particular simulation or set of simulation runs. Otherwise, inferences based on these methods can be invalid, or range from weak to unwarranted; they may even be outright erroneous and misleading. These are nontrivial issues that should be raised at an early stage of investigation; they concern the invariance and universality of results obtained through agent-based model simulations. Several specific aspects and examples of this general problem are identified below, with a view toward highlighting potential pitfalls and suggesting possible solutions. This catalog of potential problems and possible solutions is meant to be heuristic, not definitive or exhaustive; no doubt other aspects will be encountered and hopefully resolved as experience is gained in the use of these new methods. Some isolated awareness of some of these issues already exists (e.g., ref. 5), but the emphasis here is on the class of problems and solutions.
System Size
Agent-based model simulations differ by the number of agents or sites (compare refs. 10 and 11), although system size is rarely changed within a given model run. How does variation in the number of interacting units (grid size) affect the main results of an agent-based simulation? Empirically, we know that size is time-dependent for numerous social systems (e.g., urban centers, protesters at a demonstration, international systems) and that group or system size matters for systems and processes involving collective action (12, 13), war and peace patterns (for example, refs. 14 and 15), and similarly significant social phenomena.
Sensitivity analyses of the main results with respect to system or group size are necessary to ensure that simulated results (synthetic outcomes) are not purely local for a given system size or are not idiosyncratically determined. In many instances, nonlinear effects of system size S on resulting behavior f are expected on theoretical and empirical grounds (∂2ƒ/∂S 2 ≠ 0), so a better understanding of agent-model size is clearly necessary.
Agent Geometry
With few exceptions (16, 17), the standard geometry of agents in most agent-based model simulations is square, which is an odd social shape except for city blocks in downtown urban areas or similarly structured social systems. How does this feature affect main results in an agent-based model simulation with adjacent agents? For example, empirically we know that the average number of borders of real territorial political units is closer to six than it is to four (18). It seems difficult to believe that a property as basic as the number of interaction opportunities, which is determined by agent geometry, would have no effect on resulting behavior (19, 20).
A feasible and worthwhile solution to this problem is to design agent geometry in such a way as to come closer to the referent geometry, but without loss of parsimony. In the case of territorial political units, hexagons should provide a closer empirical fit than squares, and at the same time maintain parsimony (16). Not surprisingly, hexagons are also commonly used in military battlefield simulations and war games (21). Whether or not agent geometry will make a difference in the results of agent-based model simulations remains an open question, as long as this plausible conjecture remains untested. If hexagonal geometry does make a difference, then this specific type of invariance may raise further issues, given the real-world distribution of political boundaries.
Network Topology
Besides site or agent geometry, most agent-based simulations select the basic network interaction topology (e.g., a von Neumann/orthogonal or Morgan/diagonal rule) in arbitrary ways, or perhaps in terms of computational criteria, not substantive aspects of the problem under investigation. How does the interaction structure among agents affect the results? Should Morgan or von Neumann rules apply? This is not always a trivial problem.
Again, it seems likely that network topology should have some (and as yet unknown) effect on resulting processes and emergent behavior. For example, in the case of international systems, this problem is avoided altogether by a network topology consisting of hexagons, with six adjacent interactions per site or political agent and no need to chose between Morgan or von Neumann rules. In other cases a solution would be to alter the network topology rule and explore the effects.
Time Calibration
Agent-based simulations typically run for “many iterations,” just as previous simulation models. How many hours, days, months, or years equal one iteration in simulated “time”? Unless the iteration duration is known in terms of some physical calendar time, there is no way to comprehend the time scale of the emergent phenomena. This problem does not arise in some classical models; for example, in extensive form games, punctuated by well-defined decision episodes, or in arms race models, punctuated by budgetary cycles or cycles of weapons appropriation.
A solution to this problem is to use emergent features in the simulation process to calibrate the time scale. For example, in an agent-based simulation where wars occur as a result of agent interactions, it is possible to calibrate the iteration units in the time domain based on the known empirical frequency of wars of various types and magnitudes (Table 3.1 in ref. 22). Similarly, the time domain of a simulation can be calibrated based on empirical statistics of polity duration and related phenomena (23, 24). The underlying principle is the same in both cases: use empirical observations to estimate the proportionality factor φ for simulated time, such that τsimulation = φτreferent.
Phenomenological Calibration
Besides time, the size or magnitude of various phenomena in agent-based simulation results is also often uncalibrated. For example, the territorial size of units in a landscape, or the intensity of warfare, are generally obtained as dimensionless quantities without any known relation to empirically observable (physical) quantities such as hectares or fatalities, respectively. Moreover, the relationship between a dynamic set of state variables X(τ) and the time-domain φ cannot be arbitrary (a magnitude 7.5 earthquake cannot occur in 3 s; just as an empire cannot form in 2 days). Intensity and temporal dimensions are correlated in the real world (space-time allometry).
Here again, the known record from empirical observation should be used to calibrate simulated dimensions, similar to the temporal domain. A reliable, albeit decentralized, database of empirical magnitudes for conflicts of various types (22), territorial units (23), and other important magnitudes is now available and should be used for calibrating simulated phenomena.
Structural Stability
Agent-based simulations typically produce patterns of emergent phenomena expressed in terms of state trajectories, phase portraits, distributions of behavior, and similar aggregate representations of “long-term” evolution. Are such long-term results stationary in the sense of showing convergence toward a steady state? Can structural stability be demonstrated, or must it be simply assumed?
Let 〈z〉(τ) denote some aggregate (average) evolutionary trajectory in a simulation run or set of simulation runs. Then, “the temporal behavior of 〈z〉(τ) offers only a rough check on whether the asymptotic regime has been reached. To be careful, one would check the temporal evolution of the distribution functions” (26). These time-dependent diagnostics are rare in the social sciences, but nevertheless essential to address issues of structural stability or invariance in the time domain (14, 15, 27).
Power Laws
Finally, numerous empirical social processes are known to organize themselves as power laws, or patterns of the form ƒ(x) ∼ x −b, where b is a dimensional exponent characteristic of the underlying process (unpublished data). Pareto's law of income distribution, Zipf's rank-size rule, and Richardson's law of war magnitudes are examples of power laws. To what extent are power laws present in the emergent behavior of agent-based simulations? Are such synthetic power laws, when they do emerge, comparable to empirically observed power laws?
Empirically validated power laws can be used to calibrate key dimensions in agent-based simulation models. Moreover, the exact value of the synthetic parameter value b can shed new light on underlying processes, such as self-organized criticality (27).
Conclusions
The preceding inventory of potential pitfalls and possible solutions is minimal, and mostly based on past experience with earlier pre-agent-based modeling and simulation methods. For example, this repertoire omits other potential issues that may be similarly highlighted (e.g., sensitivity of results to decision rules, types of random distributions, selection criteria for initial conditions, and so on). No doubt other potential problems will be discovered and hopefully corrected as experience with agent-based models develops.
The issues raised in this brief paper seem particularly timely because agent-based simulation methods are relatively new and many standards are still unsettled. Users are discussing and selecting “best practices,” but already new substantive applications are being closely scrutinized by others who use traditional methods and are understandably suspicious of the new agent-based projects as “a third way of doing science” (5). Some of these pitfalls should be identified and addressed early on, lest they cause greater problems later, especially in terms of threatening the long-term contribution that agent-based simulations can make in the social sciences.
Acknowledgments
I thank B. J. L. Berry, L. D. Kiel, and E. Elliott for the invitation to the 2001 National Academy of Sciences Sackler Colloquium on “Adaptive Agents, Intelligence, and Emergent Human Organization: Capturing Complexity through Agent-Based Modeling,” at the Arnold and Mabel Beckman Center, Irvine, CA, October 4–6, 2001. I also thank H. Alker, Jr., R. Bhavnani, J. Bennett, L.-E. Cederman, N. Collier, C. Dupont, U. Luterbacher, M. Macey, S. Majeski, F. Santerre, and D. Sylvan for comments during the Agent-Based Modeling Conference, Graduate Institute of International Studies, Geneva, Switzerland, 8–11 November, 2001, and P. Bak, K. Carley, D. Campbell, R. Conte, V. J. Gupta, M. Hoffmann, P. E. Johnson, S. Lohmann, J. F. Padgett, J. B. Rundle, and D. Sallach for helpful discussions.
Footnotes
-
↵ * E-mail: cioffi{at}colorado.edu.
-
This paper results from the Arthur M. Sackler Colloquium of the National Academy of Sciences, “Adaptive Agents, Intelligence, and Emergent Human Organization: Capturing Complexity through Agent-Based Modeling,” held October 4–6, 2001, at the Arnold and Mabel Beckman Center of the National Academies of Science and Engineering in Irvine, CA.
- Copyright © 2002, The National Academy of Sciences
