Brain computation by assemblies of neurons

How do assemblies in the association cortex come about? It has been hypothesized (see, e.g., ref. 13) that an assembly imprinting, for example, a familiar face in a subject’s medial temporal lobe (MTL) is created by the projection of a neuronal population, perhaps in the inferotemporal cortex, encoding this face as a whole object. By projection of an assembly $x$, we mean the creation of an assembly $y$ in a downstream area that can be thought of as a “copy” of $x$, and such that $y$ will henceforth fire every time $x$ fires.

How is the new assembly $y$ formed in a downstream area $B$ by the repeated firing of $x$ in area $A$? The process was vividly predicted in the discussion section of ref. 14, for the case in which $A$ is the olfactory bulb and $B$ is the piriform cortex. Once $x$ has fired once, synaptic connectivity from area $A$ to area $B$ excites many neurons in area $B$. Inhibition will soon limit the firing in area $B$ to a smaller set of neurons, let us call it $y_1$, consisting, in our framework, of $k$ neurons (Fig. 2A). Next, the simultaneous firing of $x$ and $y_1$ creates a stronger excitation in area $B$ (one extra reason for this is plasticity, which has already strengthened the connections from $x$ to $y_1$), and, as a result, a new set of $k$ neurons from area $B$ will be selected to fire, call it $y_2$. One would expect $y_2$ to overlap substantially with $y_1$—this overlap can be calculated in our mathematical model to be roughly $50\%$ for a broad range of parameters. If $x$ continues firing, a sequence $\left\{y_t\right\}$ of sets of neurons of size $k$ in area $B$ will be created. For a large range of parameters and for high enough plasticity, this process can be proved to converge exponentially fast, with high probability, to create an assembly $y$, the result of the projection.

We denote the projection process described above as $\text{project}\left(x,B,y\right)$. Assembly projection has been demonstrated both analytically (16, 17) and through detailed simulations (18, 19); simulation results in our model, as well as improved analysis, are presented in Fig. 2B4. Once the $\text{project}\left(x,B,y\right)$ operation has taken place, we write $B=\text{area}\left(y\right)$ and $x=\text{parent}\left(y\right)$.

Hebb (4) hypothesized that assemblies are densely intraconnected—that is, the chance that two neurons have a synaptic connection is significantly larger when they belong to the same assembly than when they do not—and our analysis and simulations verify this hypothesis (Fig. 2C). From the point of view of computation, this is rather surprising, because the problem of finding a dense subgraph of a certain size in a sparse graph is a known difficult problem in computer science (20), and thus the very existence of an assembly may seem surprising. How can the brain deal with this difficult computational problem? The creation of an assembly through projection as outlined in the previous paragraphs provides an explanation: Since the elements of assembly $y$ were selected to have strong synaptic input from the union of $x$ and $y$, one intuitively expects the synaptic recurrent connectivity of $y$ to be higher than random. In addition, the weights of these synapses should be higher than average because of plasticity.

It can be said that assembly operations, as described here, are powered exclusively by two forces known to be crucial for life more generally: randomness and selection (in addition to plasticity). No special purpose neural circuits are required to be in place; all that is needed is random synaptic connectivity between, and recurrently within, areas, and selection, through inhibition in each area, of the $k$ out of $n$ cells currently receiving highest synaptic input. All assembly computation described here consists of applications of this operator, which we call random projection and cap (RP&C). We believe that RP&C is an important primitive of neural computation, and computational learning more generally, and can be shown to have a number of potentially useful properties. For example, we establish, analytically and through simulations, that RP&C preserves overlap of assemblies remarkably well (as first noticed empirically in ref. 21). Finally, we used RP&C in experiments as the nonlinearity in each layer of a deep neural network, in the place of the sigmoid or the rectified linear unit typically used in machine leaning, and we found that it seems to perform at a level competitive with these.

In a recent experiment (15), electrocorticography recordings of human subjects revealed that a neuron in a subject’s MTL consistently responding to the image of a particular familiar place—such as the Pyramids—starts to also respond to the image of a particular familiar person—say, the subject’s sibling—once a combined image has been shown of this person in that place. A compelling parsimonious explanation of this and many similar results is that two assemblies imprinting two different entities adapt to the cooccurrence, or other observed affinity, of the entities they imprint by increasing their overlap, with cells from each migrating to the other while other cells leave the assemblies to maintain its size to $k$; we say that the two assemblies are associating with one another. The association of two assemblies $x$ and $y$ in the same brain area is denoted associate$\left(x,y\right)$, with the common area $\text{area}\left(x\right)=\text{area}\left(y\right)$ implicit. We can show, analytically and through simulations, that the simultaneous sustained firing of the two parents of $x$ and $y$ does effect such increase in overlap, while similar results have been obtained by simulations of networks of spiking neurons through spiked timing-dependent plasticity (STDP) (18).

Assemblies are large and, in many ways, random sets of neurons, and, as a result, any two of them, if in the same area, may overlap a little by chance. If the assembly size $k$ is about less than the square root of $n$, as we often assume in our simulations, this random overlap, if any, should be very few cells. In contrast, overlap resulting from the operation $\text{associate}\left(x,y\right)$ is quite substantial: The results of ref. 15 suggest an overlap between associated assemblies in the MTL of about 8 to 10% of the size of an assembly. The association between assemblies evokes a conception of a brain area as the arena of complex association patterns between the area’s assemblies; for a discussion of certain mathematical aspects of this view, see ref. 22.

One important and well-studied phenomenon involving assemblies is pattern completion: the firing of the whole assembly $x$ in response to the firing of a small subset of its cells (8); presumably, such completion happens with a certain a priori probability depending on the particular subset firing. In our experiments, pattern completion happens in a rather striking way, with small parts of the assembly being able to complete, very accurately, the whole assembly (Fig. 2 F1 and F2).

We believe that association and pattern completion open up fascinating possibilities for a genre of probabilistic computation through assemblies, a research direction which should be further pursued.

The most sophisticated and complex operation in the repertoire of the Assembly Calculus is merge. Denoted $\text{merge}\left(x,y,A,z\right)$, this operation starts with two assemblies $x$ and $y$, in different brain areas, and creates a new assembly $z$, in a third area $A$, such that there is ample synaptic connectivity from $x$ and $y$ to $z$, and also vice versa, from $z$ to both $x$ and $y$.

Linguists had long predicted that the human brain is capable of combining, in a particularly strong sense, two separate entities to create a new entity representing this specific combination (23, 24), and that this ability is recursive in that the combined entity can, in turn, be combined with others. This is a crucial step in the creation of the hierarchies (trees of entities) that seem necessary for the syntactic processing of language, but also for hierarchical thinking more generally (deduction, discourse, planning, storytelling, etc.). Recent fMRI experiments (25) have indicated that, indeed, the completion of phrases and sentences (the completion of auditory stimuli such as “hill top” and “ship sunk”) activates parts of Broca’s area—in particular, the pars opercularis BA 44 for phrases, and the pars triangularis BA 45 for sentences. In contrast, unstructured word sequences such as “hill ship” do not seem to activate Broca’s area. Recall that Broca’s area has long been believed to be implicated in the syntactic processing of language.

A parsimonious explanation of these findings is that phrases and sentences are represented by assemblies in Broca’s area that are the results of the merge of assemblies representing their constituents (that is, assemblies for words such as “ship” and “sunk”); presumably, the word assemblies reside in Wernicke’s area implicated in word selection in language. As these hierarchies need to be traversed both in the upward and in the downward direction (e.g., in the processes of language parsing and language generation, respectively), it is natural to assume that merge must have two-way connections between the new assembly and the constituent assemblies.

Our algorithm for implementing merge, explained in Results and Methods, is by far the most complex in this work, as it involves the coordination of five different brain areas with ample reciprocal connectivity between them, and requires stronger plasticity than other operations; our simulation results are reported in Section 2; see also Fig. 2 G1 and G2.

Finally, a simpler operation with similar yet milder complexity is $\text{reciprocal.project}\left(x,A,y\right)$: It is an extension of $\text{project}\left(x,A,y\right)$, with the additional functionality that the resulting $y$ has strong backward synaptic connectivity to $x$. Reciprocal projection has been hypothesized to be instrumental for implementing variable binding in the brain—such as designating “cats” as the subject of the sentence “cats chase mice” (see ref. 19). The plausibility of $\text{reciprocal.project}$ has been experimentally verified through detailed simulations of networks of spiking neurons with STDP (19), as well as in our simplified model.

The purpose of the Assembly Calculus is to provide a formal system within which high-level cognitive behavior can be expressed. Ultimately, we want to be able to write meaningful programs—in fact, parallel programs—in this system, for example, containing segments such as

With this goal in mind, we next introduce certain additional low-level control operations, sensing and affecting the state of the system. First, a simple read operation. In an influential review paper on assemblies, Buzsáki (7) proposes that, for assemblies to be functionally useful, readout mechanisms must exist that sense the current state of the assembly system and trigger appropriate further action. Accordingly, the Assembly Calculus contains an operation $\text{read}\left(A\right)$ that identifies the assembly which has just fired in area $A$, and returns null otherwise.

Finally, the Assembly Calculus contains two simple control operations. We assume that an assembly $x$ in an area $A$ can be explicitly caused to fire by the operation $\text{fire}\left(x\right)$. That is to say, at the time an assembly $x$ is created, a mechanism is set in place for activating it; in view of the phenomenon of pattern completion discussed above, in which firing a tiny fragment of an assembly leads to the whole assembly firing, this does not seem implausible. We also assume that, by default, the excitatory cells in an area $A$ are inhibited, unless explicitly disinhibited by the operation $\mathtt{d}\mathtt{i}\mathtt{s}\mathtt{i}\mathtt{n}\mathtt{h}\mathtt{i}\mathtt{b}\mathtt{i}\mathtt{t}\left(A\right)$ for a limited period, whose end is marked by the operation $\text{inhibit}\left(A\right)$; the plausibility of the disinhibition–inhibition primitives is argued in ref. 26 in terms of vasoactive intestinal peptide cells (27).

For a simple example of a program in the Assembly Calculus, the command $\text{project}\left(x,B,y\right)$ where $\text{area}\left(x\right)=A$, is equivalent to the following:

Simulations show that, with typical parameters, a stable assembly is formed after about $T=10$ steps. An alternative version of this program, relying on the function $\text{read}$, is the following:

For another simple example, the command $\text{associate}\left(x,y\right)$, whose effect is to increase the overlap of two assemblies in the same area $A$, is tantamount to this:

A far more elaborate program in the Assembly Calculus, for a proposed implementation of the creation of syntax trees in the generation of natural language, is described in Discussion.

We can show (SI Appendix) that the Assembly Calculus as defined above is capable of implementing, under appropriate assumptions, arbitrary computations on $O\left(\frac{n}{k}\right)$ bits of memory—and, under much weaker assumptions, $O\left(\frac{\sqrt{n}}{k}\right)$ bits. This suggests—in view of the well-established equivalence between parallel computation and space-bounded computation (28)—that any parallel computation with a few hundred parallel steps, and using a few hundred registers, can, in principle, be carried out by the Assembly Calculus.

The Assembly Calculus is a formal system with a repertoire of rather sophisticated operations, where each of these operations can be ultimately reduced to the firing of randomly connected populations of excitatory neurons with inhibition and Hebbian plasticity. The ensuing computational power of the Assembly Calculus may embolden one to hypothesize that such a computational system—or rather a neural system far less structured and precise, which, however, can be usefully abstracted this way—underlies advanced cognitive functions, especially in the human brain, such as reasoning, planning, and language.

Assemblies of excitatory neurons are, of course, not new: They have been hypothesized (4, 5), identified in vivo (6, 7), studied experimentally (8), and discussed extensively (29) over the past decades—even, occasionally, in connection to computation (see ref. 30). However, we are not aware of previous work in which assemblies, with a suite of operations, are proposed as the basis of a computational system intended to explain cognitive phenomena.

Assemblies and their operations as treated here bear a certain resemblance to the research tradition of hyperdimensional computing (see refs. 31–33), systems of high-dimensional sparse vectors equipped with algebraic operations, typically component-wise addition, component-wise multiplication, and permutation of dimensions. Indeed, an assembly can be thought of as a high-dimensional vector, namely, the characteristic vector of its support; but this is where the similarity ends. While assembly operations as introduced here are meant to model and predict cognitive function in the brain, hyperdimensional computing is a machine learning framework—inspired, of course, by the brain, like many other such frameworks—and used successfully, for example, for learning semantics in natural language (33). In sparse vector systems, there is no underlying synaptic connectivity between the dimensions, or partition of the dimensions into brain areas. Finally, the operations of the Assembly Calculus (project, associate, and merge) are very different in nature, style, detail, and intent from the operations in sparse vector systems (add, multiply, and permute).

Assembly computation is closer in spirit to Valiant’s model of neuroids and items (34), which was an important source of inspiration for this work. One difference is that, whereas, in Valiant’s model, the neuron (called neuroid there) has considerable local computational power—for example, to set its presynaptic weights to arbitrary values—in our formalism, computational power comes from a minimalistic model of inhibition and plasticity; both models assume random connectivity. Another important difference is that, in contrast to an item in Valiant’s theory, an assembly is densely intraconnected, while the mechanism for its creation is described explicitly.

The operation $\text{project}\left(x,B,y\right)$ entails activating, repeatedly, assembly $x$ while $B$ is disinhibited. Such repeated activation creates, in the disinhibited area $B$, a sequence of sets of $k$ cells, let us call them $y_1,y_2,\dots ,y_t,\dots$. The mathematical details are quite involved, but the intuition is the following: Cells in $B$ can be thought of as competing for synaptic input. At the first step, only $x$ provides synaptic input, and thus $y_1$ consists of the $k$ cells in $B$ which happen to have the highest sum of synaptic weights originating in $x$—note that these weights are subsequently increased by a factor of $\left(1+\beta \right)$ due to plasticity. At the second step, neurons in both $x$ and $y_1$ spike, and, as a result, a new set $y_2$ of “winners” from among cells of $B$ is selected; for typical parameters, $y_2$ overlaps heavily with $y_1$. This continues as long as $x$ keeps firing, with certain cells in $y_t$ replaced by either “new winners”—cells that never participated in a $y_{t^{\prime }}$ with $t^{\prime }<t$—or by “old winners”—cells that did participate in some $y_{t^{\prime }}$ with $t^{\prime }<t$. We say that the process has converged when there are no new winners. Upon further firing of $x$, $y_t$ may evolve further slowly, or cycle periodically, with past winners coming in and out of $y_t$; in fact, this mode of assembly firing (cells of the assembly alternating in firing) is very much in accordance with how assemblies have been observed to fire in Ca+ imaging experiments in mice; see, for example, ref. 35. It is theoretically possible that a new winner cell may come up after convergence; but it can be proved that this is a highly unlikely event, and we have never noticed it in our simulations. The number of steps required for convergence depends on the parameters, but, most crucially, on the plasticity coefficient $\beta$; this dependence is fathomed analytically and through simulations in Fig. 2B4.

It was postulated by Hebb (4) that assemblies are densely interconnected—presumably such density was thought to cause their synchronized firing. Since assemblies are created by projection, increased density is intuitively expected: Cells in $y_t$ are selected to be highly connected to $y_{t-1}$, which is increasingly closer to $y_t$ as $t$ increases. It is also observed in our simulations (Fig. 2C) and predicted analytically in our model.

Our simulations (Fig. 2E), as well as analytical results, show that the overlap of two assemblies $x,y$ in the same area increases substantially in response to simultaneous firing of the two parent assemblies $\text{parent}\left(x\right)$ and $\text{parent}\left(y\right)$ (assumed to be in two different areas). The amount of postassociation overlap observed in our simulations is compatible with the estimates in the literature (36, 37), and increases with the extent of cooccurrence (the number of consecutive simultaneous activations of the two parents).

Since association between assemblies is thought to capture affinity of various sorts, the question arises: If two associated assemblies $x$ and $y$ are both projected in another area, will the size of their overlap be preserved? Our results, both analytical and through simulations, strongly suggest that assembly overlap is indeed conserved reasonably well under projection (see Section 2 and Fig. 2). This is important, because it means that the signal of affinity between two assemblies is not lost when the two assemblies are further projected in a new brain area.

Pattern completion involves the firing, with some probability, of an assembly $x$ following the firing of a few of its neurons (8, 35). Simulations in our model (Fig. 2F2) show that, indeed, if the creation of an assembly is completed with a sufficient number of activations of its parent, then firing fewer than half of the neurons of the assembly will result in most, or essentially all, of the assembly firing.

Reciprocal projection, denoted $\text{reciprocal.project}\left(x,B,y\right)$, is a more elaborate version of projection. It involves $\text{parent}\left(x\right)$ firing, which causes $x$ in area $A$ to fire, which, in turn, causes, as with ordinary projection, a set $y_1$ in $B$ to fire. The difference is that now there is synaptic connectivity from area $B$ to area $A$ (in addition to connectivity from $A$ to $B$), which causes, in the next step, $x$ to move slightly to a new assembly $x_1$, while $y_1$ has become $y_2$. This continuing interaction between the $x_t$ and the $y_{t\pm 1}$ eventually converges, albeit slower than with ordinary projection, and under conditions of ampler synaptic connectivity and plasticity. The resulting assembly $y$ has strong synaptic connectivity both to and from $x$ (instead of only from $x$ to $y$, as is the case with ordinary projection). That reciprocal projection works as described above has been shown both analytically and through simulations in our model, as well as in simulations in a more realistic neural model with explicit inhibitory neurons and STDP in ref. 26.

The operation $\text{merge}\left(x,y,A,z\right)$ is essentially a double reciprocal projection. It involves the simultaneous repeated firing, in different areas, of the parents of both $x$ and $y$, which causes the simultaneous repeated firing, also in two different areas $B$ and $C$, of $x$ and $y$. In the presence of enabled afferent two-way connectivity between $A$ and $B$, and also between $A$ and $C$, this initiates a process whereby a new assembly $z$ is eventually formed in area $A$, which, through its firing, modifies the original assemblies $x$ and $y$. In the resulting assemblies, there is strong two-way synaptic connectivity between $x$ and $z$, as well as between $y$ and $z$. Analytical and simulation results are similar to those for reciprocal projection (Fig. 2G1).

We gain insights into the workings of our model, and validate our analytical results, through simulations. In a typical simulation task, we need to simulate a number of discrete time steps in two or three areas, in a random graph with $\sim n^{2}p$ nodes (where the notation $\sim f$ means ”a small constant multiple of $f$). Creating this graph requires $\sim n^{2}p$ computation. Next, simulating each firing step entails selecting, in each area, the $k$ out of $n$ cells that receive that largest synaptic input. This takes $\sim knp$ computations per step. Since the number of steps needed for convergence is typically much smaller than the ratio $n/k$, the $n^{2}p$ computation for the creation of the graph dominates the computational requirements of the whole simulation (recall that $n>>k$).

In our simulator, we employ a maneuver which reduces this requirement to $\sim knp$, enabling simulations of the required scale on a laptop. The trick is this: We do not generate all $\sim n^{2}p$ edges of the random graph a priori, but generate them “on demand” as the simulation proceeds. Once we know which cells fired at the previous step, we generate the $k$ cells in the area of interest that receive the most synaptic input, as well as their incoming synapses from the firing cells, by sampling from the tail of the appropriate binomial distribution; we then compare with previously generated cells in the same area to select the $k$ cells that will fire next.

Our simulation system is available online, making it possible to reproduce and extend our simulation results: http://github.com/dmitropolsky/assemblies.

We hypothesized that assemblies and their operations may be involved in mediating higher cognitive functions in humans. In ending, we speculate below on how assemblies may be implicated in language.

Linguists have long argued that the language faculty is unique to humans, and that the human brain’s capability for syntax (i.e., for mastering the rules that govern the structure of sentences in a language) lies at the core of this ability (23, 24). In particular, the linguistic primitive Merge has been proposed as a minimalistic basis of syntax. Merge combines two syntactic units to form one new syntactic unit; recursive applications of Merge can create a syntax tree, which captures the structure of a sentence (Fig. 3A). If this theory is correct, then which are the brain correlates of syntax and Merge? Recent experiments provide some clues; see ref. 38 for an extremely comprehensive and informative recent treatment of the subject.

If one accepts the hypothesis that, indeed, something akin to syntax trees is constructed in our brain during the parsing—and presumably also during the generation—of sentences, one must next ask, How is this accomplished? According to ref. 38, chapter 4, these and a plethora of other experimental results point to a functional cortical network for lexical and syntactic processing, involving the MTL, Broca’s areas BA 44 an BA 45, and Wernicke’s area in the superior temporal gyrus, as well as axon fibers connecting these four areas, all in the left hemisphere. Syntactic processing of language seems to entail a complex sequence of activations of these areas and transmission through these fibers. How is this orchestrated sequence of events carried out? Does it involve the generation and processing, on-the-fly, of representations of the constituents of language (words, phrases, sentences)? Angela Friederici (ref. 38, p. 134) proposes that, “for language there are what I call mirror neural ensembles through which two distinct brain regions communicate with each other.” Could it be that assemblies and their operations play this role?

We propose a dynamic brain architecture for the generation of a simple sentence, powered by assemblies and the operations $\text{reciprocal.project}$ and $\text{merge}$, and consistent with the experimental results and their interpretations outlined above (Fig. 3). In particular, we propose that the construction of the syntax tree of a simple sentence being generated by a subject can be implemented by the following program of the Assembly Calculus:

do in parallel:

do in parallel:

The generation of a sentence such as “The boy kicks the ball” starts with a desire by the subject to assemble—and possibly articulate—a particular fact. The raw fact to be put together is denoted here by $Im$—an image, sensed or mental. In the first line, a brain area containing the lexicon is searched in order to identify the verb (the action in the fact relayed in $Im$), the subject (the agent of the fact), and the object (the patient of the fact). Here we are assuming that the lexicon is a searchable collection of tens of thousands of assemblies in the left MTL of the subject, representing words in the language; this is in rough agreement with current views (42), even though much still remains to be understood about the representation of words and the associated information. We also assume that the corresponding brain functions find-verb, etc., are already in place. For the example “The boy kicks the ball,” the three assemblies $x$, $y$, and $z$ are identified after the first step, encoding the words “kick”, “boy”, and “ball”, respectively. In the second line, these three assemblies are projected to the three subareas of Wernicke’s area specializing in the verb, subject, and object of sentences, respectively. In fact, instead of ordinary projection, the $\text{reciprocal.project}$ operation is used, as first proposed in ref. 26, for reasons that will become clear soon. Next, an assembly $p$ is formed in the pars opercularis of Broca’s area (BA 44) representing the verb phrase “kicks the ball” through the merge operation applied to the two assemblies $x^{\prime }$ and $z^{\prime }$ encoding the constituent words of the phrase in Wernicke’s area. Finally, an assembly $s$ corresponding to the whole sentence is formed in the pars triangularis of Broca’s area (BA 45) via the merge of assemblies $p$ and $y^{\prime }$, completing the construction of the rudimentary syntax tree of the whole sentence.

The Assembly Calculus program above accounts only for the first phase of sentence production, during which the syntax tree of the sentence is constructed. Next, the sentence formed may be articulated, and we can speculate on how this process is carried out: Assembly $s$ is activated, and this eventually causes the assemblies $x^{\prime },y^{\prime },z^{\prime }$—the leaves of the syntax tree—to fire. The activation of these three assemblies is done in the order specific to the speaker’s language learned at infancy (for example, in English, subject–verb–object). Note that the first phase of building the syntax tree was largely language independent. Eventually, the lexicon assemblies will be activated in the correct order (this activation was the purpose of using the $\text{reciprocal.project}$ operation), and these, in turn, will activate the appropriate motor functions which will ultimately translate the word sequence into sounds.

The above narrative is only about the building of the basic syntactic structure—the “scaffold”—of extremely simple sentences, and does not account for many other facets: how the three find- tasks in the first line are implemented; how the noun phrases are adorned by determiners such as “the,” and how the verb is modified to reflect person and tense (“kicks” or “kicked”), and the important inverse processes of parsing and comprehension, among a myriad of other aspects of language that remain a mystery.