# Optimal control of aging in complex networks

See allHide authors and affiliations

Edited by John A. Rogers, Northwestern University, Evanston, IL, and approved July 7, 2020 (received for review April 5, 2020)

## Significance

Aging in many complex systems composed of interacting components leads to decay and eventual collapse/death. Repair delays this process at a cost, suggesting a trade-off between the cost of repair and the benefit of health and longevity. Using an interdependent network model of a complex system, we introduce a control theoretic and learning framework for maximizing longevity at minimal repair cost and determine the optimal maintenance schedule for the system. Our approach should be relevant to determining checking schedules for complex engineered and living systems.

## Abstract

Many complex systems experience damage accumulation, which leads to aging, manifest as an increasing probability of system collapse with time. This naturally raises the question of how to maximize health and longevity in an aging system at minimal cost of maintenance and intervention. Here, we pose this question in the context of a simple interdependent network model of aging in complex systems and show that it exhibits cascading failures. We then use both optimal control theory and reinforcement learning alongside a combination of analysis and simulation to determine optimal maintenance protocols. These protocols may motivate the rational design of strategies for promoting longevity in aging complex systems with potential applications in therapeutic schedules and engineered system maintenance.

Aging is the process of damage accumulation with time that is responsible for an increasing susceptibility to death or decay (1). Many complex systems that consist of multiple interacting components (2) (e.g., biological organisms and artificially engineered systems) experience aging. Indeed, models of the interdependence between components of a system implemented in a network (3) show aspects of aging and eventual system-wide catastrophe and death. This is because when components are interdependent, the failure of one component may adversely affect its dependents. The dynamics of these processes have been the focus of many recent studies (4⇓⇓–7), exhibit temporal scaling (8, 9) and failure cascades, and reproduce empirical survivorship curves for many biological organisms and technological devices (4).

Understanding the onset of aging in network models points toward a central question in the field (10): how can one control aging in complex systems through interventions associated with repair and maintenance, with the eventual goal of designing strategies for increasing longevity? Available control strategies in networks are primarily for single nodes (11) and sets of driver nodes (12, 13), and they largely fall into three classes: network design (14, 15), edge and node removal at onset of cascade (16), and time-dependent edge weight distribution (17, 18). Complementing these approaches, in reliability engineering, there are maintenance policies for deteriorating multiunit systems (19⇓⇓–22) that include opportunistic repair (23) and group and block replacement (24) for systems with economic and structural dependencies between components (20, 21, 25). However, aging systems are primarily characterized by failure dependencies between components. Only very special repair policies have been optimized for failure-dependent complex systems (26), and most are restricted to systems composed of few units (20, 21), or with strong assumptions about the underlying failure distribution without consideration for the dynamics of individual network components from which they emerge (22, 27).

Here, we deploy an optimal control framework to determine strategies to delay aging in a tractable and realistic model for the failure of interdependent networks (4). A combination of numerical simulations and analysis shows how the microscopic dynamics of individual network components that are capable of stochastic failure determine how system-level macroscopic dynamics of decreasing vitality and failure cascades follow. We then introduce the notion of repair in such a network that can lead to a delay in aging, but at a cost. This can be couched in the framework of optimal control theory (28) and allows us to determine strategies to delay aging in these interdependent networks analytically in the linearized regime and computationally in the nonlinear regime. To understand the implications of our results, we then deploy a simple reinforcement learning scheme (29) to determine the parameters associated with explicit temporal repair protocols that determine the efficacy of drugs on the longevity of a classic model organism used in aging studies—the nematode *Caenorhabditis elegans*.

## Network Model of Aging and Repair

### Computational Model.

Our computational model of aging starts with the consideration of a network with N nodes representing the individual components of the complex system and edges between nodes representing interdependencies between the individual components (Fig. 1*A*). The main network structure used in this study is the Gilbert *SI Appendix*, Fig. S6). In the model, each node is assigned an initial state of binary value

The network is then allowed to age via a simple iterative algorithm (*SI Appendix*, Algorithm 1) through the following actions: 1) each node fails with probability f; 2) nodes are repaired with probability r; 3) a node fails if the fraction of vital providers (i.e., functional neighboring nodes) is less than I; 4) the network vitality is calculated using the expression *A*). Here, I is a measure of the interdependence between the system components and denotes the threshold fraction of vital providers required for a node to stay alive.

Our model reproduces the characteristic cascading failures that are present in the computational models of breakdown of complex systems (4, 14, 16, 33). In a representative simulation, the vitality *B*). The cascading failure is observed in all three graph structures (*SI Appendix*, Fig. S6). This sudden decrease in system vitality is similar to the compression of morbidity that is observed during late life for humans and many other biological organisms (34). Movie S1 shows a two-dimensional visualization of network failure. Our simulations allow us to go beyond the mean-field theory and look at the probability distribution of failure events, defined in Fig. 1*D* in terms of the mean time for failure μ and the standard deviation in the failure time σ. In Fig. 1*E*, we show how these parameters vary with the failure rate f, and in Fig. 1*F*, we see that the standard deviation is linearly correlated with the mean failure time, consistent with a Weibull distribution (see *SI Appendix*, section S3).

### Nonlinear Theory of Network Aging.

To complement our computational model of aging networks, we also construct an effective equation for the average network vitality measured over several realizations, *SI Appendix*, Fig. S1, we compare the mean-field model Eq. **4** and the network simulations (Fig. 1*B*). Analytically, we see that the solution to Eq. **4** describes an average vitality that decreases slowly at early times. In the limit when the system is away from collapse (**4** can be linearized and approximated to leading order as*B*). At later times, the average vitality exhibits failure cascade and rapid collapse after a critical vitality value *B*). This effect originates when the denominator in the first term on the right-hand side of Eq. **4** becomes small, which causes the effective failure rate to blow up; thus, an estimate for the critical fraction for failure cascade can be obtained by maximizing *SI Appendix*, section S1).

## Optimal Control of Network Aging

Having a qualitative understanding of the forward problem of how aging arises in interdependent networks, we now turn to the problem of controlling the progressive aging of a network by varying the repair rate, subject to some constraints.

### Optimal Repair Protocols.

For an interdependent network that ages according to Eq. **4**, our goal is to design optimal repair protocols [i.e., replace the constant repair frequency r in Eq. **4** by a time-dependent unknown repair rate **4** and **6** is to find the repair protocol **6** while satisfying the evolution equation for vitality Eq. **4** (*SI Appendix*, section S8 has a discussion of alternative cost functions and the effect of nonlinear cost functionals).

We solve this optimal control problem for a network with initial vitality *SI Appendix*, section S2 has details). Since the optimal control problem is linear in the repair rate *SI Appendix*, Eq. **S23** (*SI Appendix*, section S2.2 has a derivation, and *SI Appendix*, section S2.4 has a discussion on singular arcs). Eq. **7** states that the optimal decision to repair depends on two parameters: 1) the repairable fraction of nodes,

### Linear Control Theory.

To gain an understanding of how the optimal repair protocol depends on the physical parameters, we focus first on the linearized limit, Eq. **5**, valid away from vitality collapse. In fact, explicit analytical expressions for optimal protocols can be obtained in this case. Condition Eq. **7** leads to nonmonotonic optimal repair protocols characterized by a waiting time for repair, followed by an intermediate period where repair is preferable and a terminal phase where the repair rate is set again to zero (*SI Appendix*, Fig. S2):*SI Appendix*, section S2 has a derivation)**9**, which for *SI Appendix*, section S2.3.1). We also characterized longevity gain resulting from optimal protocols as a function of α and γ (*SI Appendix*, Fig. S3).

In the infinite horizon limit

Thus far, we have focused on the simple linear cost function. Exploring nonlinear cost functions leads to optimal repair protocols that are no longer of bang-bang type but are still nonmonotonic in time (*SI Appendix*, section S8), with initial and terminal phases of low repair and an intermediate region of higher repair (*SI Appendix*, Fig. S8). Additional extensions may be motivated by future experiments and might involve considering a terminal cost for vitality, including nonlinearities in vitality and/or repair rate (*SI Appendix*, section S8), or introducing additional variables, such as node checking and associated cost (*SI Appendix*, section S9).

### Phase Diagram for Repair.

A question of some interest is the determination of the conditions under which a repair protocol is advisable. From Eq. **9**, it follows that since **10** results in a phase diagram separating a region of “repair” from a region of “no repair,” where repair is too costly, as a function of two relevant dimensionless parameters **10** predicts the existence of regions of low (*B*). This behavior follows intuition; when failure rate is low, vitality remains high over the interval **10** predicts a rapidly shrinking window of repair due to the combined effect of increasing the effectiveness of and associated cost (*C*). As the cost of repair α increases, Eq. **10** similarly predicts a decreasing window of repair (Fig. 2*D*) that results from an increasing cost burden. There exists a critical value for the repair cost,

### Interdependent Networks.

For networks with interdependent components, the optimal protocols are still bang bang, and the switching times can be calculated using Eq. **7**. Notably, increasing the interdependence (*SI Appendix*, section S4). The results shown in Fig. 2*E* agree with the optimal policies calculated using Eq. **7** (solid lines).

### Fluctuations in Failure Times.

Using our framework, we have measured the extent of fluctuations in failure times with and without repair (Fig. 1*D*). The resulting distribution of failure times is described by the Weibull distribution (*SI Appendix*, section S3). Interestingly, we find that, in addition to increasing the average failure time, application of the finite horizon optimal repair protocol results in a broader distribution of network failure times (Fig. 1 *C*–*E*). This effect emerges naturally from the Weibull distribution, in which the mean failure time is linearly correlated with the SD in failure times for different values of f (Fig. 1*F*).

### Role of Network Topology.

We have also studied optimal protocols numerically for Erdős–Rényi *SI Appendix*, Fig. S6 *A–C*). For all random and scale-free networks, we observe no significant qualitative differences in the optimal repair protocols (*SI Appendix*, Fig. S6 *D* and *E*), indicating that our protocols are robust and may be applicable to a diverse range of complex systems.

## A Generalization and an Application

Our computational and theoretical model of aging and its control in a complex network naturally raises the question of whether the optimal control policy that we determined can be iteratively learned and applied to a real system.

### Reinforcement Learning Approach to Interdependent Network Aging Control.

Optimal control strategies rely on knowledge of the model and a cost function, both of which are hard to crystallize into quantitative form in many biological systems. An alternative strategy is to ask whether the system is able to learn the optimal repair protocol for aging via an iterative procedure. This is tantamount to direct adaptive optimal control (36), embodied in reinforcement learning, a process by which a system is able to optimize its actions by interacting with its environment. Optimization occurs iteratively on a trial and error basis by reinforcing actions that maximize reward and/or minimize punishment. We use a relatively simple version of this algorithm known as *Q* learning (Fig. 3*A* and *SI Appendix*, Fig. S5) (29), which is a model-free alternative to dynamic programming models of the Bellman equation (37). This method consists of creating a *Q* matrix, *Q* matrix according to the rule (29)*Q*-learning discount factor that is related to the optimal control through *Q*-learning model iterates through learning episodes until qualitative convergence of the *Q* matrix is achieved. The optimal protocol is defined as the maximal *Q*-valued trajectory traveled by a network through

Using this method, the *Q*-learned repair policy converges to optimal repair protocols that are bang bang (Fig. 3*B*, *Inset*) and closely match the predicted switching time *B*). These results suggest that the optimal protocols for repair can be obtained through simple iterative learning and highlight the potential of *Q* learning as a method to approximate optimal repair protocols for complicated systems in which no analytic description of the aging dynamics is available.

### Quantifying the Protocols for Extending Longevity of *C. elegans*.

We now deploy this approach using data from a model organism in aging research, the nematode *C. elegans*. We used *C. elegans* life span data (38⇓⇓–41) in the presence of a metabolite α-ketoglutarate that significantly extend *C. elegans* life span from which we estimate the parameters in our minimal model corresponding to the failure rate f, interdependency I, and α-ketoglutarate repair rate r (*SI Appendix*, section S7). Using *Q* learning with these fitted parameters, we predict optimal protocols for this system as a function of the cost of repair α (Fig. 3*C*). Consistent with our model, the switching time for α-ketoglutarate treatment increases with the cost of repair α. These results suggest that life-extending treatments that work through repair or replacement may be determined through measurable parameters. However, understanding the dependence of these strategies on the choice of the cost remains an outstanding question.

## Discussion

Although aging in real biological and technological systems is a consequence of complex biochemical and mechanical processes, here we have abstracted a minimal model designed to capture the essential ingredients that give rise to aging in a complex system—modular units (nodes) that are linked to each other via a set of edges modeled as an interdependent network subject to nodal failure and repair. Our model shows the emergence of failure cascades, a hallmark of such systems, which we can understand using an analytic approach that accounts for both the observed mean and extreme value statistics of failure. We then deployed a simple scheme for repair in such interdependent systems, couched as an optimal control problem to slow down aging. We first used a model-dependent strategy to determine explicit optimal repair protocols for aging interdependent systems characterized by a failure rate f, repair rate r, and interdependency I. We also demonstrated that a model-free approach using reinforcement learning converges to these optimal repair protocols and can therefore be leveraged to approximate optimal repair strategies in an iterative manner. This allowed us to estimate model parameters for the efficacy of a drug used to increase the longevity of a model organism in aging studies—the nematode *C. elegans*.

We conclude with some implications of our work to the biological problem of determining optimal protocols for life-extending treatments such as the clearance of senescent cells. Senescent cells enter a permanent, nondividing state and adopt an altered secretory profile, which has been implicated in inflammation, tumorigenesis, and aging (10, 42). The presence of these cells has been shown to promote senescence in surrounding tissue (43), similar to how node failure in a network can spread due to interdependence. In contrast, the selective clearance of senescent cells (i.e., via the use of senolytic cocktails) improves physical function and survival (43, 44), without reducing either the total cell count (in human tissue) or body weight (in mouse models) (43). This suggests a rapid replacement of cleared senescent cells by healthy dividing cells, so that the application of senolytic treatments becomes analogous to node repair in an aging network. Like many life-extending compounds, senolytic cocktails may include toxicity (45, 46), which can be modeled in the cost function. The relative cost of repair α can be determined by separately measuring the loss of viability caused by senescence and the toxicity that results from treatment on an ensemble of healthy cells. This would allow for the deployment of our optimal repair protocols and design treatment schedules for senescent cell inhibitors and other therapeutics that target general aging processes.

## Materials and Methods

There are no data associated with this paper.

## Acknowledgments

We acknowledge support from the Swiss National Science Foundation and the Amgen Scholars Program.

## Footnotes

↵

^{1}E.D.S. and T.C.T.M. contributed equally to this work.- ↵
^{2}To whom correspondence may be addressed. Email: lmahadev{at}g.harvard.edu.

Author contributions: E.D.S., T.C.T.M., and L.M. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

The authors declare no competing interest.

This article is a PNAS Direct Submission.

This article contains supporting information online at https://www.pnas.org/lookup/suppl/doi:10.1073/pnas.2006375117/-/DCSupplemental.

Published under the PNAS license.

## References

- ↵
- D. Harman

- ↵
- Y. Bar-Yam

- ↵
- ↵
- D. C. Vural,
- G. Morrison,
- L. Mahadevan

- ↵
- S. Taneja,
- A. B. Mitnitski,
- K. Rockwood,
- A. D. Rutenberg

- ↵
- S. G. Farrell,
- A. B. Mitnitski,
- K. Rockwood,
- A. D. Rutenberg

- ↵
- A. B. Mitnitski,
- A. D. Rutenberg,
- S. Farrell,
- K. Rockwood

- ↵
- ↵
- N. Stroustrup

- ↵
- ↵
- ↵
- ↵
- ↵
- L. Zhao,
- K. Park,
- Y. C. Lai,
- N. Ye

- ↵
- T. Tanizawa,
- G. Paul,
- R. Cohen,
- S. Havlin,
- H. E. Stanley

- ↵
- ↵
- W. X. Wang,
- G. Chen

- ↵
- B. Mirzasoleiman,
- M. Babaei,
- M. Jalili,
- M. Safari

- ↵
- ↵
- L. C. Thomas

- ↵
- W. P. Pierskalla,
- J. A. Voelker

- ↵
- ↵
- R. Radner,
- D. W. Jorgenson

- ↵
- R. E. Barlow,
- F. Proschan

- ↵
- Y. S. Sherif,
- M. L. Smith

- ↵
- ↵
- ↵
- L. M. Hocking

- ↵
- ↵
- E. N. Gilbert

- ↵
- Erdős P.,
- Rényi A.

- ↵
- A. L. Barabási,
- R. Albert

- ↵
- P. Crucitti,
- V. Latora,
- M. Marchiori

- ↵
- ↵
- ↵
- R. Sutton,
- A. Barto,
- R. Williams

- ↵
- S. Bhasin

- ↵
- ↵
- ↵
- I. Alfaras et al.

- ↵
- ↵
- ↵
- ↵
- ↵
- M. Serrano,
- N. Barzilai

- ↵
- J. L. Kirkland,
- T. Tchkonia

## Citation Manager Formats

## Article Classifications

- Physical Sciences
- Applied Physical Sciences