# Stabilized supralinear network can give rise to bistable, oscillatory, and persistent activity

See allHide authors and affiliations

Edited by William Bialek, Princeton University, Princeton, NJ, and approved February 11, 2018 (received for review January 12, 2017)

## Significance

Many fundamental neural computations from normalization to rhythm generation emerge from the same cortical hardware, but they often require dedicated models to explain each phenomenon. Recently, the stabilized supralinear network (SSN) model has been used to explain a variety of nonlinear integration phenomena such as normalization, surround suppression, and contrast invariance. However, cortical circuits are also capable of implementing working memory and oscillations which are often associated with distinct model classes. Here, we show that the SSN motif can serve as a universal circuit model that is sufficient to support not only stimulus integration phenomena but also persistent states, self-sustained network-wide oscillations along with two coexisting stable states that have been linked with working memory.

## Abstract

A hallmark of cortical circuits is their versatility. They can perform multiple fundamental computations such as normalization, memory storage, and rhythm generation. Yet it is far from clear how such versatility can be achieved in a single circuit, given that specialized models are often needed to replicate each computation. Here, we show that the stabilized supralinear network (SSN) model, which was originally proposed for sensory integration phenomena such as contrast invariance, normalization, and surround suppression, can give rise to dynamic cortical features of working memory, persistent activity, and rhythm generation. We study the SSN model analytically and uncover regimes where it can provide a substrate for working memory by supporting two stable steady states. Furthermore, we prove that the SSN model can sustain finite firing rates following input withdrawal and present an exact connectivity condition for such persistent activity. In addition, we show that the SSN model can undergo a supercritical Hopf bifurcation and generate global oscillations. Based on the SSN model, we outline the synaptic and neuronal mechanisms underlying computational versatility of cortical circuits. Our work shows that the SSN is an exactly solvable nonlinear recurrent neural network model that could pave the way for a unified theory of cortical function.

Understanding the mechanisms underlying cortical function is the main challenge facing theoretical neuroscience. Nonlinear responses such as normalization, contrast invariance, and surround suppression are often encountered in the cortex, particularly in the visual cortex (1, 2). At the same time, working memory about a transient stimulus, persistent activity in decision-making tasks, and generation of global rhythms represent fundamental computations which have been linked to cognitive functions (3⇓–5). Whether these computations arise from common or different mechanisms is an open question.

Previous work showed that the response of in vivo cortical neurons is well described by a power law (6) (Fig. 1). Experiments and theory indicate that low firing rates and high variability reported in the cortex are associated with a power-law activation function and noise-driven firing (6⇓⇓–9). Furthermore, a power law is mathematically the only function that permits precise contrast invariance of tuning given input that has contrast-invariant tuning (8, 9). Recent work (1, 2, 10) proposed the stabilized supralinear network (SSN) as a mechanism for normalization, contrast invariance, and surround suppression. While the SSN model is powerful in the context of stimulus integration, its role for other cortical computations remains to be clarified (11).

Working memory is the ability to remember a transient input long enough for it to be processed (3). It is often implemented as a stimulus-induced switch in a bistable system (12). Persistent activity is thought to be critical for neural implementation of decision making (3, 4). Mechanistically, it is firing observed in the absence of any stimulus between the cue and the go signal in decision-making tasks. Persistent activity was first recorded in the early 1970s in the prefrontal cortex of behaving monkeys (13) and has since been found in other cortical areas (3). Finally, generation of synchronous network-wide rhythms is also a fundamental cortical feature that has been suggested to mediate communication between cortical areas and may be needed for attention, feature integration, and motor control (5).

Here, we investigate whether the SSN motif is sufficient to give rise to network multistability, persistent activity, and global rhythms. Seminal articles have proposed that working memory is a result of network multistability (12). To explain multistability, mechanisms that involve either S-shaped neuronal activation function in combination with a suitably strong excitatory feedback or S-shaped synaptic nonlinearity have been proposed (14⇓–16). Likewise, a persistent state which occurs in the decision-making process has been modeled using synaptic nonlinearities (17). The SSN framework, however, lacks both ingredients: it does not exhibit an S-shaped neural activation function and lacks synaptic plasticity. It is also currently unclear whether the SSN motif alone supports the emergence of stable global oscillations. If oscillations can emerge in the SSN, it would be important to determine the conditions for connectivity and neural power-law exponents when this occurs. On the other hand, if the SSN motif is not sufficient for the emergence of oscillations, it would be necessary to understand what sets it apart from models that can support global rhythms (14, 18, 19).

To investigate whether and when the SSN model can support multistability and persistent and oscillatory activity we need to understand its dynamics systematically. Since the SSN is a nonlinear model, mathematical methods to obtain exact solutions are challenging. Previous studies used the functional shape of nullclines to determine their possible crossings and derive the number of steady states in the Wilson–Cowan model with an S-shaped activation function (14, 15). Here, we aim to explain how location and multiplicity of steady states are influenced by specific parameter choices for which information on the shape of nullclines is not always mathematically tractable. To this end, we developed an analytical method that allows us to map the steady states of the 2D SSN model to the zero crossings of a 1D characteristic function. We determine multiplicity and stability of the SSN steady states for all possible parameter regimes. In particular, we show that the SSN model supports the following three effects. First, we outline network configurations corresponding to a bistable regime that could serve as a substrate for working memory. Second, we show that the SSN model can lead to dynamically stable persistent activity and derive the corresponding connectivity requirements. Third, we prove the occurrence of stable global oscillations in the SSN and provide methods for how to tune the oscillatory frequency to a desired value. Our article is organized as follows. First, we analyze the multiplicity and stability of the SSN steady states and show the existence of a unique mapping between zero crossings of a characteristic function and steady states. Second, we prove for equal-integer power-law exponents that the model has at most four coexisting steady states and at most two steady states can be stable. Finally, we present parameter regimes for which a stable persistent state and global oscillations exist.

## Results

We consider a recurrent network as illustrated in Fig. 1 and model the activity of excitatory and inhibitory populations *Rescaling the Connectivity and Input Constants to Match a Desired Firing-Rate Range* for details on the rescaling procedure.

To characterize the SSN activity states, it is crucial to understand how location, multiplicity, and dynamical stability of the SSN steady states depend on connectivity, inputs, and activation functions of neuronal populations. The steady-state firing rates **1**. We show that the 2D system of nonlinear equations in Eq. **2** can be reduced to a single 1D equation for a nonlinear characteristic function. Specifically, we have identified a one-to-one correspondence between the 2D steady states and the 1D zero crossings of a characteristic function such that the steady-state search algorithm is reduced to a lower-dimensional problem.

### Reducing the Dimensionality via a Characteristic Function.

To solve the two nonlinear equations presented in Eq. **2**, we take the following steps. We first apply a variable substitution replacing *Materials and Methods*. Now, we can simplify the problem by eliminating the variable **2** are uniquely mapped to the zero crossings of the characteristic function F which depends on z. For *Materials and Methods* and *Characteristic Function*). In the following, we refer to the function F as a characteristic function. Mathematically, the nullcline crossings in Eq. **2** that define the steady states are now mapped to the zero crossings of a 1D function F. This transformation significantly enhances the mathematical tractability because steady states are now intersection points of a 1D function with a coordinate axis and are no longer intersections of two complex nonlinear nullclines. In Fig. 2, we depict the zero crossings of F and the corresponding steady states for four representative classes of solutions. Additionally, in *Location of SSN Steady States Is Preserved in the Presence of Firing-Rate Saturation* and Fig. S2 we show that these steady states remain at the same location if activation functions saturate at high firing rates. Saturation of activation functions provides boundedness of all solution trajectories; however, it can introduce additional steady states.

### Stability Conditions.

The characteristic functions in Fig. 2 exhibit zero crossings with positive derivatives which correspond to saddles, while those with negative derivatives correspond to either stable or repelling steady states. To support this finding mathematically, we consider the eigenvalues **4** is similar to the relation derived in ref. 20 where a 2D system of equations corresponding to a conductance-based model was reduced to a 1D function whose derivative coincided with the determinant of the Jacobian. A steady state is a saddle if the eigenvalues are real and have opposite signs. This can occur if and only if **5** is fulfilled. We note that if **5** can always be fulfilled by choosing **5** is automatically fulfilled and the steady state is stable. If *Stability Conditions*).

### Determining the Number of Coexisting Steady States.

We now determine how many steady states can coexist for a given synaptic connectivity and inputs. First, let us note that previous work (14, 15) reported up to five steady states in the Wilson–Cowan model with an S-shaped activation for positive

To study the multiplicity of the SSN steady states, we partition the parameter space of the SSN model into nine classes. Such partition makes the problem more accessible for mathematical analysis. For details we refer to *Materials and Methods*. First, we identify three classes according to the sign of *Materials and Methods*). To determine the multiplicity of steady states for each of the nine parameter classes presented in Table 1, we follow the steps outlined in *Materials and Methods* and *Multiplicity of Steady States*. We find that there can be at most four steady states and this number does not grow with n. For *Multiplicity of Steady States*). For

Taken together, our results indicate that the power-law–type networks with equal integer exponents can support two coexisting stable states, but no more than two stable states can occur at a time. The ability of a circuit to support bistability has been suggested as a substrate for working memory where the circuit transitions from the lower to the upper state follow the presentation of a brief stimulus. The emergence of bistability has so far been confirmed in neural networks with S-shaped synaptic or neuronal nonlinearities (16). However, our results show that neither synaptic nonlinearities nor S-shaped activation functions are prerequisites for bistability in a network. Two coexisting steady states can simply emerge in a network with a globally convex power-law activation.

### Persistent State.

One of the hallmarks of nonlinear systems is the ability to sustain a response after the input is switched off. Such response is referred to as a persistent state. We ask whether it is possible to observe a persistent state in a system described by Eq. **1** with the exponents *Materials and Methods* and *Existence of Persistent States*). The inequality on the left side of Eq. **6** implies that a persistent state in the network described by Eq. **1** is possible only when the cross-population connectivity **6**. We note that a stable persistent state does not exist outside the parameter set specified by the inequality Eq. **6** (see *Materials and Methods* for further details). To guarantee that a given persistent steady state is stable and can thus be reached dynamically, the time constants **5** presented above. Fig. 3 *A* and *B* shows examples of a stable and repelling persistent state. If the existence of a positive zero crossing with negative derivative in absence of inputs has been established, **5**.

### Emergence of Oscillations.

Periodic fluctuations of neural activity are a common occurrence across the cortex and have been associated with many information-processing tasks (5). Some cognitive disorders are also reportedly accompanied by a change in the oscillatory cortical activity (21). In some instances, the oscillatory activity has been explained by the amplification of selected input frequencies (22) while in others, it has been proposed to arise from intrinsic, self-sustained oscillations originating for example from a Hopf bifurcation (18). Here, we study whether the SSN can support global oscillations arising from a Hopf bifurcation and if so which synaptic or neuronal parameters can lead to oscillations and which ones control the oscillatory frequency. An extensive analysis of the Hopf bifurcation for the Wilson–Cowan model with an S-shaped activation function can be found in ref. 23. A Hopf bifurcation occurs when the complex conjugate eigenvalues corresponding to a stable steady state cross the imaginary axis when the input current or other parameter values are varied. In this way, the initial frequency of the global oscillation corresponds to the imaginary part of the eigenvalues (18). To determine when the steady-state eigenvalues in the SSN can become complex and when the limit cycle is stable, we have followed the steps outlined in *Existence of Global Oscillations* and in ref. 24. If *Existence of Global Oscillations* and the code accompanying this article. In Fig. 4 we present the example of a stable limit cycle emerging from a supercritical Hopf transition.

## Discussion

Multipurpose models that explain more than just the functions they are designed for are currently in short supply. Here, we show that the SSN can serve as a universal circuit model that can support rhythm generation and working-memory features such as bistability and persistent states along with many stimulus integration rules as well as contrast invariance, normalization, and surround suppression (2, 6, 8, 9). Importantly, we provide a unique analytical method to obtain exact solutions for the SSN model that are valid for all parameter regimes. This makes the SSN model a nonlinear, yet exactly solvable, recurrent neural network model. Remarkably, the computational versatility of the SSN model emerges from the nonlinear activation function and does not require synaptic plasticity, delay tuning, or specific ion channel models. The characteristic function method we derived here allows us to show that the connectivity regime where the product of cross-connectivities **6**). Our work thus provides an important step toward a unified theory of cortical function by identifying connectivity regimes supporting working memory and network oscillations and uniting these fundamental computations via the SSN.

## Materials and Methods

### Correspondence Between the Steady States and the Zero Crossings of a Characteristic Function.

We show how the SSN steady states can be unambiguously mapped to the zero crossings of a characteristic function F, a correspondence which we illustrate in Fig. 2. Here, we derive F for *Characteristic Function* for **15**.

As a first step toward F for **2** can be written as**7**. We obtain**8** into Eq. **9** and obtain**10** and **11** and reduce the system to 1D. To this end, we introduce an abbreviation for the left side of Eq. **10**:**10** can be written as **11** and obtain**13**,**12**. We thereby have shown that if **12** and **14**, respectively. Next, we show that no spurious zeros of F exist that do not correspond to steady states. To this end, we consider a real number **14** for **12**, and insert these variables into **2**. Next, we use the relation **2**. This shows that our definition of F establishes a bijective mapping between

In summary, the steps outlined above for *Characteristic Function* for

Notably, we obtain a bijective mapping such that if **1**, then **1** is**15** is necessary for calculating the number of zero crossings of F later (it is critical to the Descartes rule of signs and other statements) (*Multiplicity of Steady States*). When applying the same P and F for all values of

### Stability of Steady States.

In the previous paragraph, we presented the method of finding the steady states using the characteristic function in Eq. **15**. Next, we derive exact stability conditions for the steady states. Our goal is to relate stability of steady states to the behavior of the characteristic function in the vicinity of its zero crossings. To this end, we first move the time constants to the right side of Eq. **1** and consider the resulting dynamical system **4** and **5**.

For completeness, we note that a previous study (10) showed that if **5**. Here, we significantly expanded this result and revealed the full SSN stability repertoire.

### Multiplicity of Steady States.

In the previous two paragraphs, we demonstrated how to determine the location of steady states and to characterize their stability using the characteristic function. Here, we build on these findings and use the characteristic function to derive the multiplicity of possible steady states. To provide exact results, we limit our analysis to integer power-law exponents *Multiplicity of Steady States*, we detailed why the maximal number of coexistent steady states with equal integer exponents is four and that at most two can be stable simultaneously. This proves that bistability is supported by the SSN model.

To determine the multiplicity of steady states, we need to find the number of F zero crossings. To this end, we separate the parameter space into nine distinct classes as indicated in Table 1 and determine possible multiplicity configurations of zero crossings in each class. The goal of parameter separation is to simplify the equation for the characteristic function Eq. **15** in each class and make it more accessible for mathematical analysis. We distinguish three subsets of the parameter space according to the sign of **15**. Further, we separate parameter space based on the sign of

To understand why, consider the expressions *Multiplicity of Steady States*, we determine possible stability and multiplicity configurations of steady states in each class. The parameter classes in which two stable steady states can coexist are indicated in boldface type in Table 1. The constants

### Persistent Stable State.

A persistent state is a stable positive steady state for zero input. To guarantee its existence, it is necessary to show two features. First, we show the existence of a positive zero-crossing z with **5**). Using the steps outlined in *Existence of Persistent States*, we show that a required positive zero crossing of the characteristic function with the negative derivative exists in absence of inputs if and only if

Based on this result, we derived an easy-to-verify necessary condition Eq. **6** for the existence of a stable persistent state. For **18** can be directly inserted in the inequality Eq. **17**; see *Existence of Persistent States* for more details.

## Acknowledgments

We thank K. Miller, H. Cuntz, B. Ermentrout, A. Nold, and members of our group for comments and discussions. We thank K. Miller and an anonymous reviewer for contributing to the persistent-state condition Eq. **S11** and Eq. **6**. This work is supported by the Max Planck Society (T.T.) and a grant from the German Research Foundation via Collaborative Research Center 1080 (to T.T.).

## Footnotes

- ↵
^{1}To whom correspondence may be addressed. Email: tatjana.tchumatchenko{at}brain.mpg.de or nataliya.kraynyukova{at}brain.mpg.de.

Author contributions: N.K. and T.T. designed research, performed research, contributed new reagents/analytic tools, analyzed data, and wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

Data deposition: The code for Figs. 2–4 and the Lyapunov coefficient has been deposited in GitHub, https://github.molgen.mpg.de/MPIBR/CodeSSN.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1700080115/-/DCSupplemental.

- Copyright © 2018 the Author(s). Published by PNAS.

This open access article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND).

## References

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Hansel D,
- Van Vreeswijk C

- ↵
- ↵
- Wolf F,
- Engelken R,
- Puelma-Touzel M,
- Weidinger JDF,
- Neef A

- ↵
- ↵
- Fuster JM,
- Alexander GE

- ↵
- ↵
- Ermentrout GB,
- Terman DH

- ↵
- ↵
- ↵
- Izhikevich EM

- ↵
- ↵
- Rinzel J,
- Ermentrout B

- ↵
- ↵
- Burns SP,
- Xing D,
- Shapley RM

- ↵
- Hoppensteadt FC,
- Izhikevich EM

- ↵
- Kuznetsov Y

## Citation Manager Formats

## Article Classifications

- Biological Sciences
- Neuroscience

- Physical Sciences
- Biophysics and Computational Biology