## New Research In

### Physical Sciences

### Social Sciences

#### Featured Portals

#### Articles by Topic

### Biological Sciences

#### Featured Portals

#### Articles by Topic

- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology

# Efficient associative memory storage in cortical circuits of inhibitory and excitatory neurons

Edited by Terrence J. Sejnowski, Salk Institute for Biological Studies, La Jolla, CA, and approved November 2, 2012 (received for review July 10, 2012)

## Abstract

Many features of synaptic connectivity are ubiquitous among cortical systems. Cortical networks are dominated by excitatory neurons and synapses, are sparsely connected, and function with stereotypically distributed connection weights. We show that these basic structural and functional features of synaptic connectivity arise readily from the requirement of efficient associative memory storage. Our theory makes two fundamental predictions. First, we predict that, despite a large number of neuron classes, functional connections between potentially connected cells must be realized with <50% probability if the presynaptic cell is excitatory and >50% probability if the presynaptic cell is inhibitory. Second, we establish a unique relation between probability of connection and coefficient of variation in connection weights. These predictions are consistent with a dataset of 74 published experiments reporting connection probabilities and distributions of postsynaptic potential amplitudes in various cortical systems. What is more, our theory explains the shapes of the distributions obtained in these experiments.

Fundamental functions of the brain, such as learning and memory storage, are mediated by many mechanisms of excitatory (1⇓⇓–4) and inhibitory (5⇓⇓⇓–9) synaptic plasticity. Working together with the genetically encoded developmental mechanisms of circuit formation, synaptic plasticity shapes neural circuits by creating, modifying, and eliminating individual synaptic connections in an experience-dependent manner. It is, therefore, reasonable to hypothesize that many stereotypic features of adult synaptic connectivity, whether established through evolution or the developmental learning process, have arisen to facilitate memory storage.

In this study, we focus on three such features of cortical connectivity. Cortical connectivity is predominantly excitatory; it is mediated by two major classes of neurons—excitatory glutamatergic and inhibitory GABAergic cells. Chemical synapses made by the axons of inhibitory cells in the adult brain are believed to be all inhibitory, whereas those synapses made by the axons of excitatory neurons are believed to be all excitatory (10). The resulting connectivity is largely excitatory, with only about 15–20% of inhibitory neurons and inhibitory synapses (11). The second stereotypic feature of cortical connectivity is sparseness. Networks in the cortex are thought to be organized into relatively small units ranging from hundreds to tens of thousands of neurons in size. Such units may include mini columns (12, 13), structural columns (14, 15), and a variety of functional columns (16, 17). Analysis of neuron morphology (14, 18⇓⇓–21) has shown that cells within such units have the potential of being connected by structural synaptic plasticity (22⇓–24). However, despite this potential, synaptic connectivity within the units is sparse. For example, nearby excitatory neurons in the neocortex are synaptically coupled with less than 50% probability (25⇓⇓⇓⇓–30), and this probability decays with the increase in lateral distance between the neurons beyond the 100-µm range (29). Although connections between inhibitory and excitatory neurons are comparatively more frequent (29, 31), their probabilities also are well below 100%. Lastly, we note that the distributions of excitatory and inhibitory connection weights, measured in terms of the amplitudes of unitary postsynaptic potentials (PSPs), have a stereotypic shape (28⇓–30, 32⇓–34). This shape can be described as a truncated Gaussian, except for a somewhat heavier tail. In this study, we set out to show that the above basic structural and functional features of cortical circuits could have arisen from the hypothesized requirement of efficient memory storage.

Over the years, there has been a great deal of theoretical interest in the problem of associative memory storage. In particular, the perceptron (35, 36) and networks composed of McCulloch and Pitts neurons (37) received much attention (32, 38⇓⇓⇓⇓⇓⇓⇓–46). This is due in part to the existence of a theoretical framework for solving such problems, which was initially developed in the context of statistical physics. Remarkably, most of the models considered thus far do not explicitly constrain the neurons in the network to be either excitatory or inhibitory, and thus, these models are not biologically plausible. Notable exceptions are the studies by Brunel et al. (32) and Barbour et al. (34) that analyzed associative memory storage in a model of a Purkinje cell receiving excitatory parallel fiber inputs. These works show that, at the maximum (critical) storage capacity, the probability of finding a functional parallel fiber to Purkinje cell connection is less than 50% (32, 34). In other words, the majority of parallel fibers that are potentially connected to a given Purkinje cell does not establish functional connections (i.e., connections are either silent or absent). What is more, the shape of the PSP amplitude distribution is consistent with the idea that the parallel fiber Purkinje cell system is functioning at its critical memory storage capacity.

In this study, we extend the framework in the works by Brunel et al. (32) and Barbour et al. (34), making it applicable to cortical neurons receiving both excitatory and inhibitory inputs. We show that the critical memory storage capacity and the shape of the connection weight distribution in the biologically constrained model considered here exhibit complex dependence on network architecture (i.e., connection probabilities and the fraction of inhibitory neurons). By comparing the theoretical results with the connection probabilities and the distributions of unitary PSP amplitudes measured in numerous cortical systems, we substantiate the hypothesis of efficient memory storage and put forward experimentally verifiable predictions regarding cortical connectivity.

## Results

### Associative Memory Storage in Local Cortical Circuits.

We analyze a recurrent McCulloch and Pitts neural network (37) of *N*_{inh} inhibitory and *N*_{exc} = *N* − *N*_{inh} excitatory neurons (Fig. 1*A*, *Left*). This network represents a small cortical unit (e.g., structural or functional column), in which neurons are all to all connected in terms of potential synapses (14, 19). This restriction on the geometric size of the network is necessary to ensure that, if needed, any connectivity diagram can be realized during learning. The weights, *J*, of inhibitory and excitatory connections in the model represent the amplitudes of inhibitory and excitatory postsynaptic potentials (IPSP and EPSP). Zero-weight connections correspond to unconnected neurons or neurons connected only with silent synapses. Our theoretical model needs not distinguish between these two possibilities, because both cases are classified as an absence of a connection in electrophysiological experiments. The activity of neurons in the network, where zero represents the resting state and one is the firing state, is described by the probability of firing, *f*. This activity is determined at every time step by comparing the neuron’s total input with its firing threshold, *h* (Fig. 1*A*, *Right*). Both *f* and *h* may differ among neuron classes.

The state of the network in our model is described by a set of binary activities of all neurons. The neurons can learn to associate certain subsequent network states by modifying their connection weights. Such learning may be important for processing of sensory information and recall of memories. A number of constraints must be imposed on the learning process to build a biologically plausible model. First, because all synapses made by inhibitory neurons in the adult brain are inhibitory and all synapses made by excitatory neurons are excitatory, the signs of the connection weights in the network should remain fixed throughout learning. Such a network is referred to as sign-constrained (43), whereas in the absence of this constraint, the network is termed unconstrained. Second, our analysis of published electrophysiological data (*Experimental Procedures* and Tables S1 and S2) shows that the variability in the neurons’ firing thresholds is significantly smaller than the variability in the weights of connections that they receive. This observation motivated us to keep the firing thresholds of all model neurons fixed throughout learning. Third, it is commonly assumed that, similar to many biological networks (47, 48), cortical networks need to be robust (32, 40). Robustness may be required to withstand failures in generation or propagation of presynaptic action potentials, spontaneous neural activity, synaptic failure, and fluctuations in synaptic weight. The requirement of robustness is imposed in the model by the parameter *κ* ≥ 0 (*Experimental Procedures* and *SI Text*). For nonzero *κ*, small fluctuations within the network do not necessarily change its dynamics.

In this work, we assume that the network must learn to associate pairs of subsequent states that are random and independent. With this assumption, the problem of learning by the network becomes equivalent to the problem of learning by the *N* independent inhibitory and excitatory neurons that it contains. Consequently, it is sufficient to analyze learning in a single biologically constrained neuron (Fig. 1*A*, *Right*). The ratio between the number of associations (*m*) that such a neuron can learn and the total number of potential inputs (*N*) that it receives is referred to as the neuron’s capacity, *α* = *m*/*N*. Fig. 1*B* shows how the probability of perfect learning depends on the number of presented associations. With increasing *m*, it becomes progressively more difficult to find connection weights that fulfill all associations, and therefore, the probability of successful learning is a decaying function of *m*. The capacity at which the associations can be learned with 50% success probability is known as the critical capacity, *α*_{c} (40). Fig. 1*B* illustrates that *α*_{c} = 2 for a simple unconstrained perceptron learning random unbiased (*f* = 0.5) associations (38) and that *α*_{c} reduces to one if a sign constraint is applied (42⇓–44). The red curve in Fig. 1*B* shows that the success probability in our model exhibits a similar critical dependence on the number of presented associations.

The biologically constrained neuron is faced with the learning task of finding connection weights *J* that fulfill the presented set of associations given the above-mentioned constraints (second set of inequalities in Eq. **1**). This task is illustrated geometrically in Fig. 1*C* for a neuron receiving only two potential inputs. The axes in Fig. 1*C* parse the space of the possible connection weights into positive (excitatory) and negative (inhibitory) regions. The solution has to be confined to one of four quadrants as prescribed by the inhibitory/excitatory nature of the individual inputs (sign constraints). The dashed lines in Fig. 1*C* represent the individual associations that the neuron must learn. Robust learning restricts the solutions to be on a particular side of the margin surrounding each dashed line (first set of inequalities in Eq. **1**). Combined, the two sets of requirements constrain the solutions (if a solution exists) to a single convex region (one of the colored regions in Fig. 1*C*).

In the limit of large *N* and *m*, the total input received by the biologically constrained neuron is near threshold (*SI Text*), and the solution region is guaranteed to be on the hyperplane described by Eq. **2** (gray region in Fig. 1*C*). As the neuron learns a progressively increasing number of associations, the solution volume decreases, and at the critical number of associations, the typical volume of solutions shrinks to zero. This number is the maximum number of associations that the neuron can learn. Any additional learning is not possible without forgetting some of the previously learned associations. Next, we examine the properties of the biologically constrained neuron at its critical capacity.

### Properties of the Biologically Constrained Neuron at Critical Capacity.

The properties of the biologically constrained neuron at critical capacity depend on the four model parameters: the fraction of potential inhibitory inputs, *N*_{inh}*/N*; the firing probabilities of inhibitory and excitatory neurons, *f*_{inh} and *f*_{exc}; and rescaled robustness parameter, . Fig. 2 illustrates some aspects of this dependence. For networks of mostly excitatory neurons (*N*_{inh}*/N* < 0.5), the critical capacity is a decreasing function of robustness (Fig. 2*A*). This trend can be explained by the fact that some of the solution regions close as increases (Fig. 1*C*), lowering the solution probability. In the limit of large , all interior solution regions close, and *α*_{c} converges to its minimum value (black line in Fig. 2*A*). These theoretical results were confirmed by numerical simulations (dotted lines in Fig. 2*A*).

A characteristic feature of sign-constrained neural networks at critical capacity is a finite fraction of zero-weight connections (32, 43). The zero-weight connections result from the clear delineation of synapses into excitatory and inhibitory. In the course of learning, the weights of many inhibitory and excitatory connections will approach zero, where many of these weights will remain, because they are unable to cross to the other side. This feature is essential for biological feasibility, because a finite fraction of zero-weight connections (or less than 100% probability of connection between neurons) is a ubiquitous feature of neural circuits in the brain. Fig. 2*B* illustrates how the connection probability, *P*_{con}, depends on the value of robustness and the fraction of inhibitory neurons. For , both excitatory and inhibitory connection probabilities remain at 50% for all values of *N*_{inh}*/N* < 0.5. In robust networks (), excitatory connection probabilities decrease, whereas the probabilities for inhibitory connections increase with increasing *N*_{inh}*/N* or . This trend is accompanied with the overall strengthening of inhibitory and excitatory connections (Fig. 2*C*) and a decline in the neuron’s ability to recall learned associations reliably in the presence of noisy synaptic transmission (Fig. 2*D*). The reliability of recall is captured by the reliability parameter *ρ* (32), which is defined as the ratio of robustness to noise fluctuations in the input (*SI Text*). For some combinations of *N*_{inh}*/N*, *f*_{inh}, *f*_{exc}, and (Fig. 2, ♦), the average absolute connection weight diverges, and at the same time, the reliability parameter reduces to zero, rendering such network configurations metabolically and functionally unfavorable.

### Comparison of Theoretical Predictions with Experimental Measurements.

In this section, we test the predictive power of the theory by making comparisons with the available synaptic connectivity data. To this end, we examined experimental connection probabilities and shapes of unitary PSP amplitude distributions for inhibitory and excitatory inputs to principle cortical neurons (e.g., pyramidal cell in the cerebral cortex or Purkinje cell in the cerebellum). Although our theoretical results may hold well for nonprinciple postsynaptic cells, this idea is not examined here in detail because of insufficient experimental data (*Discussion*). To facilitate the comparison, the theoretical results are expressed only in terms of experimentally measurable parameters *N*_{inh}*/N*, , and (*SI Text*, section 6). Fig. 3*A* shows that the volume of biologically plausible solutions, expressed in terms of these parameters, is very limited. It amounts to only 4.9% of the total volume of the parameter space (unit cube in Fig. 3*A*). Any combination of *N*_{inh}*/N*, , and outside the 4.9% volume corresponds to a subcritical network or a completely unreliable network (*ρ* = 0) with prohibitively large connection weights.

Fig. 3*B* shows experimental distributions of inhibitory and excitatory connection probabilities gathered from a large number of studies (*Experimental Procedures* and Table S2). In agreement with the theoretical predictions, probabilities of excitatory connections are entirely within the region of biologically plausible solutions (green square in Fig. 3*B*); inhibitory connection probabilities, although not entirely within the region, have a significant overlap. It should be noted that connection probabilities obtained in tissue-slice recording experiments may underestimate the in vivo probabilities (49). This artifact is expected to primarily affect projections with low connection probabilities. Therefore, the low connection probabilities in Fig. 3*B* may need to be scaled up, which would only improve the agreement between the theory and experiment. Two of the studies from Table S2 (29, 50) measured both inhibitory and excitatory connection probabilities. These measurements from rat visual and somatosensory cortices (Fig. 3*B*, ○ and □) are well within the boundaries of the biologically plausible solution region.

Is there an optimal neural network configuration within the biologically plausible solution volume of Fig. 3*A*? To examine this question, we looked at the behavior of critical capacity, average absolute connection weight, and recall reliability of learned associations on three cross-sections of the solution volume (Fig. 3 *C–E*). In an all-excitatory network (32), there is a clear tradeoff between capacity and reliability (Fig. 3 *C1* and *E1*). With the addition of inhibitory neurons, the tradeoff becomes more complicated, because the average absolute connection weight comes into play. Because connection weight is correlated with the presynaptic bouton volume, the number of synaptic vesicles, the area of the postsynaptic density, and the spine head volume (51⇓⇓⇓⇓⇓⇓⇓–59), stronger connections entail a larger metabolic cost, and they are detrimental to the organism. The three arrows in Fig. 3 *C2*, *D2*, and *E2* illustrate the general directions of increasing capacity and reliability and decreasing connection weight—factors that may be important for efficient memory storage. However, another consideration should be mentioned. At high fractions of inhibitory neurons (greater than 50%), the solution volume is extremely small (0.65% of the total volume of the parameter space), and memory storage may become unstable. For example, small fluctuations in the inhibitory connection probability could push the network out of the solution volume and into the subcritical regimen. Balance of the above four functional forces (white arrows in Fig. 3) is expected to play a role in determining the optimal network configuration, whereas the result is likely to be cortical area-dependent.

Next, we compared the shapes of the theoretical and experimental distributions of connection weights. Similar to the study by Brunel et al. (32) performed for all-excitatory connections, our theory predicts that the distributions of inhibitory and excitatory connection weights must consist of finite fractions of zero-weight connections and Gaussians shifted to the left and truncated at zero (Eq. **3**). This theoretical finding is validated with a numerical simulation in Fig. 4*A*. For the first point of comparison, we derived a unique theoretical expression relating (with no free parameters) the coefficient of variation (CV) in connection weight and the connection probability. Fig. 4*B* shows the experimental data (red/blue points and histograms) overlaid on the theoretical curve (green line). The histograms represent all of the CV and connection probability data from Table S2, whereas the points represent studies where CV and the connection probability were measured in the same experiment. As expected (Fig. 3*B*), there is a clear clustering between inhibitory and excitatory connection probabilities, with the former being significantly higher. There is also a reasonably good agreement between the data points and the theoretical prediction (*R*^{2} = 0.72, *n* = 38 points). However, CVs for many excitatory projections seem to be significantly greater than predicted. For example, the agreement between theory and experiment improves substantially if data points corresponding to excitatory projections from mouse and rat barrel cortices (Fig. 4*B*, blue open circles) (33, 60) are excluded from the analysis (adjusted *R*^{2} = 0.83, *n* = 26). Unexpectedly large CVs result from a small fraction of very large-amplitude EPSPs (5- to 10-fold larger than the mean) recorded in many cortical systems (30, 33, 60). Although not very numerous (see for example Fig. 5*G*), these strong connections seem to be a characteristic feature of cortical connectivity. We postpone the discussion of the tail of the connection weight distribution for the moment, and instead, we continue with the examination of the overall distribution shape.

The nine largest count distributions of unitary PSP amplitudes (Table 1) were selected from the dataset for comparison with the theoretical distribution of connection weights. One-parameter fits of these distributions with Eq. **3** are shown in Fig. 5. The fitting parameter was required to match the dimensionless scale of the theoretical connection weights to the scale of unitary PSP amplitudes recorded in millivolts. Gray bins in all of the histograms in Fig. 5 were deemed unreliable and excluded from fitting (*Experimental Procedures*). The goodness of all fits, as captured by the adjusted *R*^{2} coefficients 0.92 ± 0.07 (mean ± SD), was high. As expected, a closer examination of the tails of the distributions revealed the deviations from the theoretical model.

To examine this deviation in more detail, we replotted the two highest count distributions (Fig. 5 *F* and *G*) along with their fits on a logarithmic scale by using logarithmic binning. Fig. 6 *A* and *B* shows that the theoretical fits appear to be nearly perfect on a semilog scale (adjusted *R*^{2} = 0.995 and 0.989). However, on a log–log scale (Fig. 6*C*), the deviation between the tails of the theoretical Gaussian distributions (Fig. 6, dashed lines) and the experimental distributions (Fig. 6, solid lines) becomes apparent. Remarkably, the two experimental distributions of connection weights obtained from rat visual and mouse barrel cortices (Fig. 6*C*, solid blue and red lines) have nearly identical shapes on a log–log scale. In fact, the remaining seven distributions from Fig. 5 exhibit very similar nonexponential decays (not confirmed statistically because of the low counts), suggesting that this feature may be inherent to many cortical areas.

Subcriticality of the network and nonlinearity in the input summation have been proposed as possible explanations for the heavy tails of the experimental distributions of connection weights (34). Here, we examine an alternative possibility. Because the theoretical results were obtained in the limit of large network size (*N* → ∞), we wondered whether the heavy tail of the experimental connection weight distribution could be a result of the finite network size. Fig. 6*D* shows the result of a numerical simulation performed for a model neuron receiving 75 inhibitory and 425 excitatory potential connections. The tail of this finite-size numerical distribution (Fig. 6*D*, solid line) deviates from the infinite-size theoretical distribution (Fig. 6*D*, dashed line) in a manner similar to the manner observed in Fig. 6*C* for the experimental distributions. Hence, the heavy tails of unitary PSP amplitude distributions may be explained by the fact that cortical neurons receive large (*N* ∼ 1,000) but finite numbers of potential inputs.

## Discussion

In this study, we examined the hypotheses that cortical circuits are designed for efficient associative memory storage and that many stereotypic features of adult synaptic connectivity originate from this requirement. To test the hypothesis, we theoretically solved the problem of associative memory storage in a model network of biologically constrained inhibitory and excitatory neurons. The results quantitatively explain many basic features of cortical connectivity, providing support for the efficient memory storage hypothesis.

Our theoretical analysis builds on the work by Brunel et al. (32), which considers an excitatory feed-forward system consisting of a cerebellar Purkinje cell receiving excitatory parallel fiber inputs. The work by Brunel et al. (32) shows that such a system is analogous to a perceptron functioning at critical memory storage capacity. Motivated by recent observations of inhibitory neuron plasticity (5⇓⇓⇓–9), we developed a more general theory, which can be applied to networks containing not only excitatory but also inhibitory neurons. Our results suggest that both neuron classes are actively involved in efficient associative memory storage.

Specifically, our theory makes two experimentally testable predictions. First, we predict that, despite a large number of neuron classes, all inhibitory to excitatory connections between potentially connected neurons must be realized with >50% connection probability. In contrast, all-excitatory to excitatory connections must have <50% connection probability. Hence, connectivity is sparse (i.e., it contains a large fraction of zero-weight connections). A zero-weight connection between potentially connected neurons corresponds to the absence of functional synaptic contacts (unconnected neurons or neurons connected by silent synapses). Second, we predict that the CV in the connection weights is uniquely related to the probability of that connection. These fundamental predictions do not depend on any parameters and must hold for all circuits that are designed for learning. We tested these predictions on a dataset compiled from a large number of published studies, in which connection probabilities and unitary EPSP/IPSP amplitudes had been measured for postsynaptic principle cells in various cortical systems. These measurements, based on a total of 74 experiments, are in good agreement with the theory (Figs. 3*B* and 4*B*).

Do the model predictions extend on postsynaptic inhibitory cells? To examine this question, we identified eight studies in which the probabilities of excitatory and inhibitory connections onto postsynaptic inhibitory cells were determined based on recordings from at least 20 connected neuron pairs. These probabilities are 0.60 (*n* = 73 connected pairs) among inhibitory [fast spiking inhibitory (FS) and somatostatin positive inhibitory cell] cells (64) and 0.47 (*n* = 110) for excitatory (regular spiking excitatory cell) to inhibitory (FS and low-threshold spiking inhibitory cell) connections (65) in cortical layer 4 (L4) of rodent barrel cortex; 0.45 (*n* = 146) in auditory cortex (66) and 0.59 (*n* = 79) in visual and somatosensory cortices (29) for excitatory to inhibitory (FS) projections in rodent L2/3; 0.43 (*n* = 26) for pyramidal to Martinotti connections in L5 of rat somatosensory cortex (67); 0.21 (*n* = 22) for pyramidal to inhibitory connections in L2/3 of rat somatosensory, motor, and visual cortices (68); 0.88 (*n* = 36) for excitatory to inhibitory (FS) connections in L2/3 of mouse V1 (69); and 0.76 (*n* = 28) for L4 spiny stellate to L2/3 inhibitory cell connections in rat barrel cortex (70). The last two of these projections deviate significantly from the predictions of the model, indicating that either these projections are not directly involved in efficient associative memory storage or the model assumptions must be revised to extend the theory on these classes of postsynaptic inhibitory cells. More data are needed to examine these possibilities in detail.

The shape of the theoretical distribution for nonzero-weight connections, which is a truncated Gaussian, is in good agreement with the experimental distributions of unitary EPSP/IPSP amplitudes recorded from principle neurons (Fig. 5). However, a systematic deviation was observed in the tails of the distributions. As illustrated in Fig. 6*C*, a heavy, nonexponentially decaying tail is a stereotypic feature of many unitary PSP amplitude distributions. Two mechanisms have been considered to account for this feature theoretically (32). One argument is that neurons may be operating below their critical capacity. As the memory storage becomes more and more subcritical, the tail of the connection weight distribution transitions from Gaussian to exponential. This argument, however, is not sufficient to explain the tails of the experimental distributions, which appear to be superexponential (30). An alternative mechanism is the nonlinear summation of inputs, which for specific forms of nonlinearity, can result in distributions of connection weights with superexponential tails. We propose yet another explanation. Deviations from the theoretical (Gaussian) prediction, similar to those deviations shown in Fig. 6*C*, were observed in our numerical simulations performed for large, but finite, numbers of potential inputs (*N* = 100–1,000) (Fig. 6*D*). Hence, by attributing the heavy tail of the PSP amplitude distributions to the finite size of cortical neural networks, we can account for this feature without moving away from the biologically constrained neuron model considered in this study.

Our theoretical model explains how cortical networks may benefit from the presence of inhibitory neurons. In nonrobust and completely unreliable networks (), the critical capacity is independent of the inhibitory neuron fraction as long as the system operates within the biologically plausible region of solutions (*N*_{inh}/*N* < 0.5 region in Fig. 2*A*). In this case, the inhibition may, in fact, be unfavorable, because it leads to an increase in the average absolute connection weight (Fig. 2*C*). In contrast, the critical capacity of reliable networks increases with an increasing fraction of inhibitory neurons, which could be one of the reasons for the presence of inhibitory neurons in cortical circuits. Our results suggest that sparse cortical circuits, built with small fractions of inhibitory neurons and synapses, result from the tradeoff among network capacity, reliability, metabolic cost, and stability (Fig. 3).

## Experimental Procedures

In this section, we provide a summary of the model, state model assumptions, outline the main theoretical results, and give a brief description of the experimental dataset used to validate these results. A more detailed description can be found in *SI Text*.

### Formulation of the Model.

In our theoretical model, a neuron receives *N* potential connections, of which *N*_{inh} are inhibitory and *N*_{exc} are excitatory (Fig. 1*A*). The neuron is presented with a set of *m* binary (0, 1) input–output associations (), which are randomly drawn from the probability distributions of inhibitory and excitatory inputs, *X*_{inh,exc}. Here, index *µ* = 1, …, *m* enumerates different associations, and *j* = 1, …, *N* enumerates the potential inputs. The associations must be learned by the neuron subject to a number of biologically motivated constraints: (*i*) learning is mediated through changes in the connection weights, *J*_{j}, whereas the firing threshold, *h*, is held constant; (*ii*) weights of the inhibitory and excitatory connections cannot change signs throughout learning; and (*iii*) the associations must be learned robustly, which is enforced by the robustness parameter, *κ*. The model can be written as a set of inequalities—*m* inequalities to enforce robust implementation of the associations and *N* inequalities to impose the sign constraints (Eq. **1**):

Here, *f* is the probability of firing, and *g*_{j} specifies the sign of input weight *j* (−1 for inhibitory and +1 for excitatory).

In the large *N* limit, the total input received by the biologically constrained neuron is near threshold (Eq. **S3**), and as a result, all solutions of Eq. **1** must be located in the vicinity of a single hyperplane (gray region in Fig. 1*C*) (Eq. **2**):

The critical capacity of the model and the distribution of connection weights at critical capacity can be calculated in this limit by using the replica theory from statistical physics (32, 40). In result, the critical capacity *α*_{c}, connection probabilities , and probability densities of inhibitory and excitatory connection weights, *P*_{inh,exc}(*J*), are determined in terms of the four model parameters, *N*_{inh}/*N*, *f*_{inh}, *f*_{exc}, and . In particular, one can show that the probability densities of nonzero inhibitory and excitatory connection weights are Gaussian, shifted to the left by an amount dependent on connection probability, and truncated at zero (Eq. **3**):

Parameter *s* in this expression, defining the distribution scale, is the only parameter used in the fitting of the experimental distributions in Fig. 5; erfinv denotes the inverse error function.

### Model Assumptions and Approximations.

Below, we provide some biological justifications for a number of assumptions and approximations made in the theory. McCulloch and Pitts neurons with binary (0, 1) inputs and outputs were used to represent cortical cells. This simplification can be justified if one can choose an integration window that is larger than the duration of a typical unitary PSP (*τ*_{d}) but small enough not to contain PSPs from multiple action potentials from the same input. Such binning is possible if the average firing frequencies of neurons in the network are sufficiently low, *f* × *τ*_{d} < 1. This condition also ensures that the postsynaptic neurons generally produce one or zero action potentials per integration window. We examined this condition for local neocortical networks. Connections between cortical pyramidal cells produce unitary EPSPs of 20–100 ms in duration (28, 33, 63), whereas their average spontaneous firing rates in vivo are below 10 Hz (71⇓⇓–74). These numbers justify input/output binarization for the excitatory model neurons. For inhibitory to excitatory connections, IPSP durations and in vivo inhibitory firing frequencies vary depending on the class of the inhibitory cell but generally support the above assumption (71, 73, 75⇓–77). We would like to note that, because most of the results of this study (Figs. 3, 4, 5, and 6) are cast in a manner that is independent of firing probabilities, we expect them to hold even if the product *f* × *τ*_{d} was somewhat greater than one.

We assume that coincident PSPs arriving from different inputs to a neuron sum linearly, which is generally a very good approximation for cortical neurons (32, 78, 79). The neurons in the model can learn by adjusting their connection weights; however, the firing thresholds are kept constant. This approximation is based on the observation that the distributions of connection weights between neurons in different systems are much more variable than the distributions of the neurons’ firing thresholds. For example, Tables S1 and S2 show that the CVs in connection weights, 0.47–1.7 [0.94 ± 0.03 (mean ± SE), *n* = 52 systems], are significantly higher than the CVs for firing thresholds, 0.10–0.24 (0.17 ± 0.02, *n* = 9 systems).

The activities of the model neurons at different time steps are randomly drawn from their respective probability distributions, giving rise to independent network states. This approximation is motivated by the Poisson-like statistics of cortical spike trains recorded in vivo (80). Although successive interspike intervals are often negatively correlated, these correlations are small and reduce to zero at low firing frequencies (ref. 81 and references therein). With this approximation, it is possible to decouple the recurrent network into a set of independent perceptron-like units (Fig. 1*A*) (34). We also assume that these units are functioning at their critical capacities, which enables us to find a unique analytic solution, independent of the details of the learning rule. Although it is difficult to evaluate the effect of these assumptions directly, they are supported by the agreement of our theoretical results with the experimental measurements.

Throughout this study, we discuss only two types of neurons, inhibitory and excitatory; however, the theory can be easily generalized to include any number of distinct neuron classes. Given the assumptions of the model, one can show that the probabilities of inhibitory connections must be higher than 50% for all inhibitory neuron classes, whereas the connection probabilities for all-excitatory neuron types must be below 50%. Similarly, the results shown in Figs. 4, 5, and 6 will remain unaltered.

The theoretical calculations in this study were performed in the large *N* limit to use the powerful methods of statistical physics. However, the numbers of local potential connections received by cortical neurons range from thousands (in primary neocortical areas) to hundreds of thousands (for Purkinje cells). Are these numbers large enough to justify the large *N* approximation? To answer this question, we compared a number of theoretical results to the corresponding results of the finite-sized numerical simulations. Figs. 1*B*, 2*A*, and 4*A* show that most quantities that we describe already converge to their theoretical values at 100–500 potential inputs. A notable exception is that the tail of the numerical connection weight distribution, obtained for 100–500 potential inputs, deviates from the theoretically predicted Gaussian (Fig. 6). Interestingly, a similar deviation is observed experimentally.

### Experimental Dataset and Data Analysis.

The dataset used in this study was compiled from a number of published articles reporting local connectivity measurements obtained in patch-clamp or sharp-electrode recording experiments. Specifically, we targeted studies containing distributions of unitary PSP amplitudes, mean values and SDs of unitary PSP amplitudes, and/or connection probabilities. Initially, we selected about 200 articles published in the major neuroscience journals, but later, we limited our analysis to experiments in which recordings were made from at least 20 connected neuron pairs (Table S2). Furthermore, because of very sparse data regarding connections onto postsynaptic inhibitory interneurons (*Discussion*), we decided to limit our analysis to excitatory and inhibitory connections onto the principle neurons (e.g., pyramidal and spiny stellate cells in the cerebral cortex and Purkinje cells in the cerebellum). These restrictions lead to 42 articles describing 74 projections.

A custom-made MatLab (MathWorks) algorithm was used to extract accurate quantitative information from high-resolution digital images of published distributions of unitary PSP amplitudes. The distributions were fit with Eq. **3** in MatLab using the Nonlinear Least Squares method, and goodness of fits was characterized by the adjusted *R*^{2} coefficients provided by MatLab’s fit function. Because of fluctuations in baseline recordings, very weak connections between neurons cannot be detected reliably. Such connections are often missed or ignored in experiments, leading to a systematic underestimate of weak connection counts. Therefore, connections with weights below PSP detection thresholds [0.1–0.25 mV in rodent neocortex (26, 28, 82, 83) and 0.017 mV in cerebellum (32)] were deemed unreliable and ignored during fitting (gray bins in Fig. 5 and gray areas in Fig. 6).

Numerical simulations were performed to confirm the results of the theoretical calculations (Figs. 1*B*, 2*A*, and 4*A*) and examine the tail of the connection weight distribution (Fig. 6*D*). These simulations were performed by using a modified perceptron learning algorithm (*SI Text*).

## Acknowledgments

We thank Vincent Hakim, Nicolas Brunel, Boris Barbour, and Dmitri Chklovskii for constructive discussions related to the subject of this study. A portion of this work was done by A.S. while on a sabbatical leave at the Statistical Physics Laboratory of Ecole Normale Supérieure in Paris. This work was supported by National Institutes of Health Grants NS047138 and NS063494.

## Footnotes

- ↵
^{1}To whom correspondence should be addressed. E-mail: a.stepanyants{at}neu.edu.

Author contributions: A.S. designed research; J.C., T.F., D.L., and A.S. performed research; J.C., T.F., D.L., and A.S. analyzed data; and J.C., T.F., D.L., and A.S. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

See Author Summary on page 20794 (volume 109, number 51).

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1211467109/-/DCSupplemental.

## References

- ↵
- Hebb DO

- ↵
- ↵
- Markram H,
- Lübke J,
- Frotscher M,
- Sakmann B

- ↵
- ↵
- ↵
- Holmgren CD,
- Zilberter Y

- ↵
- ↵
- ↵
- ↵
- Eccles J

- ↵
- Braitenberg V,
- Schüz A

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Hubel DH,
- Wiesel TN

- ↵
- Binzegger T,
- Douglas RJ,
- Martin KA

- ↵
- ↵
- ↵
- Kalisman N,
- Silberberg G,
- Markram H

- ↵
- ↵
- ↵
- Fares T,
- Stepanyants A

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Rosenblatt M

- ↵
- Minsky ML,
- Papert S

- ↵
- McCulloch W,
- Pitts W

- ↵
- Cover TM

- ↵
- Hopfield JJ

- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- Li F,
- Long T,
- Lu Y,
- Ouyang Q,
- Tang C

- ↵
- Kauffman SA

- ↵
- Stepanyants A,
- Martinez LM,
- Ferecskó AS,
- Kisvárday ZF

- ↵
- ↵
- ↵
- Kopec CD,
- Li B,
- Wei W,
- Boehm J,
- Malinow R

- ↵
- ↵
- Yang Y,
- Wang XB,
- Frerking M,
- Zhou Q

- ↵
- ↵
- ↵
- Schikorski T,
- Stevens CF

- ↵
- ↵
- ↵
- Petersen CC,
- Sakmann B

- Isope P,
- Barbour B

- ↵
- ↵
- Hu H,
- Ma Y,
- Agmon A

- ↵
- Beierlein M,
- Gibson JR,
- Connors BW

- ↵
- Oswald AM,
- Doiron B,
- Rinzel J,
- Reyes AD

- ↵
- ↵
- ↵
- ↵
- Helmstaedter M,
- Staiger JF,
- Sakmann B,
- Feldmeyer D

- ↵
- ↵
- ↵
- Csicsvari J,
- Hirase H,
- Czurkó A,
- Mamiya A,
- Buzsáki G

- ↵
- ↵
- ↵
- González-Burgos G,
- Krimer LS,
- Povysheva NV,
- Barrionuevo G,
- Lewis DA

- ↵
- Sun QQ,
- Huguenard JR,
- Prince DA

- ↵
- Cash S,
- Yuste R

- ↵
- ↵
- Shadlen MN,
- Newsome WT

- ↵
- ↵
- ↵

## Citation Manager Formats

### More Articles of This Classification

### Related Content

- No related articles found.