From thought to action: The brain–machine interface in posterior parietal cortex

A dramatic example of translational monkey research is the development of neural prosthetics for assisting paralyzed patients. A neuroprosthesis consists of implanted electrodes that can record the intended movement of a paralyzed part of the body, a computer algorithm that decodes the intended movement, and an assistive device such as a robot limb or computer that is controlled by these intended movement signals. This type of neuroprosthetic system is also referred to as a brain–machine interface (BMI) since it interfaces the brain with an external machine. In this review, we will concentrate on BMIs in which microelectrode recording arrays are implanted in the posterior parietal cortex (PPC), a high-level cortical area in both humans and monkeys that represents intentions to move. This review will first discuss the basic science research performed in healthy monkeys that established PPC as a good source of intention signals. Next, it will describe the first PPC implants in human patients with tetraplegia from spinal cord injury. From these patients the goals of movements could be quickly decoded, and the rich number of action variables found in PPC indicates that it is an appropriate BMI site for a very wide range of neuroprosthetic applications. We will discuss research on learning to use BMIs in monkeys and humans and the advances that are still needed, requiring both monkey and human research to enable BMIs to be readily available in the clinic.

A dramatic example of translational monkey research is the development of neural prosthetics for assisting paralyzed patients. A neuroprosthesis consists of implanted electrodes that can record the intended movement of a paralyzed part of the body, a computer algorithm that decodes the intended movement, and an assistive device such as a robot limb or computer that is controlled by these intended movement signals. This type of neuroprosthetic system is also referred to as a brain-machine interface (BMI) since it interfaces the brain with an external machine. In this review, we will concentrate on BMIs in which microelectrode recording arrays are implanted in the posterior parietal cortex (PPC), a high-level cortical area in both humans and monkeys that represents intentions to move. This review will first discuss the basic science research performed in healthy monkeys that established PPC as a good source of intention signals. Next, it will describe the first PPC implants in human patients with tetraplegia from spinal cord injury. From these patients the goals of movements could be quickly decoded, and the rich number of action variables found in PPC indicates that it is an appropriate BMI site for a very wide range of neuroprosthetic applications. We will discuss research on learning to use BMIs in monkeys and humans and the advances that are still needed, requiring both monkey and human research to enable BMIs to be readily available in the clinic.
posterior parietal cortex | brain-machine interface | monkey | tetraplegia | intention E arly scientific research in motor-related areas of monkey cortex established how intended movements are represented (1). Of particular interest to this paper is the posterior parietal cortex (PPC; Fig. 1), which, through cortical connections, is situated between primary sensory and motor cortical areas and provides a bridge for sensorimotor transformations for sensoryguided movements. In the 1970s, using technology that enabled recording of single neurons from awake, behaving monkeys, the neurophysiology of this area was first explored at the fine grain of the response properties of neurons (2,3). A variety of sensorimotor properties were found that included sensory-related properties of vision, somatosensation, and attention and movement-related properties including eye movements, fixation, reaching, and hand manipulation of objects. Subsequent studies showed that planningrelated signals are found within PPC, which reflect the intention to move particular body parts (4,5). The PPC is an area of cortex shared by monkeys and humans and thus monkeys make an ideal animal model for insights into the functions of human PPC.
Basic science results from motor cortex (6-10) and PPC (11) of monkeys have been translated into preclinical monkey models of neuroprosthetics. In our studies, approved by the Caltech Institutional Animal Care and Use Committee, monkeys were implanted chronically with microelectrode arrays that recorded the activity of populations of neurons in the PPC (11). In the M1 and PPC studies, animals quickly learned that they were able to control computers or robotic limbs with their neural signals.
This monkey research led to clinical studies in human tetraplegic participants with high-level spinal cord lesions or neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS). While several laboratories concentrated on the motor cortex as a site of signals for neuroprosthetic control (12)(13)(14)(15)(16), this review will focus on research using the PPC, an area where the initial intentions to act are made and then transferred to the motor cortex (17). These studies were approved by the US Food and Drug Administration and by the Institutional Review Boards at Caltech, the Keck School of Medicine at University of Southern California, the David Geffen School of Medicine at University of California, Los Angeles, the Rancho Los Amigos National Rehabilitation Center, and the Casa Colina Hospital and Centers for Health Care, and all subjects gave informed consent to participate.
We find that the human PPC encodes many action-related variables, and we can decode intended movements of most of the body from a small population of neurons (18). This diversity is made possible through a partially mixed encoding in which single neurons respond to multiple variables. Thus a single, small implanted electrode array within PPC provides an astonishing menu of signals that can be used to increase the versatility of applications. Among the many action variables available from PPC are the goals of movements and the trajectories to obtain these goals (17). Goal decoding is very fast, promising to increase the performance of neuroprosthetics.
The read-out pathway of a neural prosthetic records cortical signals, decodes the intentions from these recordings, and generates control signals for assistive devices like a robotic limb. To manipulate an object with precision requires somatosensory feedback (19). However, subjects with paralysis from spinal cord lesions frequently have no somatosensation below the level of injury. We and others are attempting to restore this somatosensory feedback with basic science studies in healthy monkeys (20)(21)(22) and in clinical studies in tetraplegic humans (23,24). A write-in prosthetic would restore somatosensory feedback by blanketing the robotic hand with sensors. These sensors in turn connect to electric stimulators that provide intracortical microstimulation (ICMS) through microelectrode arrays implanted in primary somatosensory cortex. When the read-out and write-in branches of the prosthetic are connected, this will create a Published under the PNAS license. 1 To whom correspondence may be addressed. Email: andersen@vis.caltech.edu.
First published December 23, 2019. bidirectional brain-machine interface (BMI) that can control a robotic limb and receive somatosensory feedback for more dexterous performance.
In the final section we will discuss learning in BMIs. Early monkey studies suggested a promising degree of adaptation over the short timescale of a single day in cortex (25)(26)(27). Subsequent studies in monkeys (28)(29)(30), and more recently in humans (31), argue that, in the short term, cortical circuits are limited to processing their intrinsic functions and adapt through cognitive manipulations such as reaiming or imagery. Changes in cortical structure have been reported over longer timescales of many days which may reflect motor skill learning (32)(33)(34)(35). Limitations on plasticity emphasize the importance of selecting brain sites for recording that are well matched to the desired function of the neural prosthesis.
For each of these topics, studying motor-sensory neurophysiology in the monkey model laid a foundation for introducing these promising technologies into human studies. With multiple ongoing human studies, the transition from laboratory to clinic might seem near completion. However, many challenges remain before BMIs are clinically relevant. New technologies and continued scientific exploration mean that monkey research is as relevant today as it has always been and continues to provide a strong foundation for developing research that can be translated into the clinic.

The PPC of Monkeys
The PPC bridges sensory and motor areas. It receives inputs from the visual, auditory, somatosensory, and vestibular systems and projects to the frontal lobe including prefrontal, premotor, and motor cortex. Not surprisingly its major functions can largely be categorized as sensorimotor transformations, especially for sensory-guided behaviors such as reaches or saccades.
Lesions in humans, including stroke and traumatic brain injury, produce an array of cognitive disorders, many of which can be described as a disruption of sensorimotor transformations. For instance, PPC lesions produce optic ataxia, a deficit in reach accuracy; neglect, a deficit in awareness; and extinction, a deficit in attention and action-based decision making (for a review see ref. 36).
Early recordings in monkey by Mountcastle and colleagues discovered many action-related neuron response types that have led to a plethora of subsequent studies (2,37). Among these response types are neurons active for limb movements, including reaching and manipulating objects, and eye movements including fixation, saccades, and smooth pursuit. Numerous studies have shown responses to vision that can be modulated by attention (3) or by the direction of gaze (38,39), the latter likely important for spatial awareness.

Saccade and Reach Areas
We discovered an area on the lateral bank of the intraparietal sulcus (lateral intraparietal area, LIP; Fig. 1) based on its strong reciprocal connections with the frontal eye fields (40). Singleneuron recording experiments showed activation prior to saccadic eye movements (41). In a memory task (42), in which the animals planned a saccade, but had to withhold the response, there was persistent neural activity during the delay period. In specially designed tasks it was shown that a visual target was not required for activation-only a planned saccade into the response field was required for the initiation of persistent activity during a planning delay (4).
It was not clear at the time whether the persistent activity was due to an intended movement plan or to attention being allocated to the location of the impending saccade. To differentiate between these possibilities we trained monkeys to switch, from trial to trial, between planning saccades and reaches (5). We found that LIP neurons showed greater persistent activity for saccade plans but not reach plans. Interestingly, we found an area on the medial bank of the intraparietal sulcus (the parietal reach region, PRR), in which the reverse was true; PRR neurons were more active during planning of reaches than saccades (Fig. 1). This double dissociation shows that LIP preferentially plans saccades and PRR reaches. In other studies by Sakata and coworkers (43) it was found that there was an area anterior to LIP, the anterior intraparietal area (AIP), that is active for different hand shapes during grasping. Taken together, these results led Andersen and Buneo (44) to propose that there is a map of intentions within the PPC around the intraparietal sulcus that includes separate regions for saccades, reaches, and grasping. A similar, possibly homologous collection of areas, has been identified in humans using functional MRI (fMRI) (45).
Intention activity is simply defined as the neural correlate of the planned action and is revealed with memory and delay tasks. The PPC is not unique in having planning related activity; for instance, similar planning activity is found in premotor cortex (46). However, motor cortex appears to be largely silent during delay epochs and becomes active during the execution of a movement at the end of the delay (47).
The intention-related PPC neurons exhibit interesting, highlevel features that may have unique advantages for neural prosthetics. The goal of the action is encoded immediately, allowing for fast actions. For both reach and grasp, the activity is often bilateral, opening the possibility of controlling 2 limbs from a single area (48). Individual AIP neurons code the entire hand shape for grasping, removing the need to coordinate the movement of individual fingers (43). For these reasons we thought PPC may have advantages for neural prosthetics. What we did not realize at the time was that there was an incredible array of action-based signals in PPC (18). This richness of action variables will allow tremendous versatility in the use of a single implant in a multitude of tasks that can encompass intended movements of most of the body.

First Implants of PPC in Humans
The first implant of human PPC was made in April 2013 with participant EGS (17). He had a complete spinal cord lesion at level C3-4 10 y prior to the implant. To identify which regions of cortex to implant, we used fMRI to measure brain activity while EGS imagined reaches and grasps. Two sites were chosen near the surface of cortex based on the imaging results and due to the constraint that the Food and Drug Adminstration-approved Blackrock arrays we use have electrode shafts of limited length. One site was a grasp activation location, prominent in the intraparietal sulcus but also continuing onto the gyrus at the junction of the intraparietal sulcus and the posterior central sulcus. This location was assigned as the presumed human homolog of area AIP, the grasp-selective region first identified in monkeys. The second implant location was made in Brodmann's area 5, a reach region activated in the fMRI scan that is located on the surface of a gyrus. The presumed PRR in humans, a reach-selective region initially identified in monkeys (5), is buried in sulci near the midline of the hemispheres and was not accessible for this initial study.
From EGS' neurons we recorded goal and trajectory signals for imagined movements (17). He controlled both a cursor on a computer screen and a robotic limb. As predicted from monkey work, neurons were active for imagined reach of either limb. Neurons were also found that were extremely specialized for specific behavioral actions; for example, we found units that became active for imagined movements of the hand to the mouth but would not become active for similar movements such as movement of the hand to the cheek or forehead. Such neural encoding of behaviorally meaningful actions opens the possibility that the high-level intent of the subject can be decoded and integrated with smart robotics (49)(50)(51) to perform complex movements that may otherwise require attentionally demanding moment-to-moment control of a robotic limb. To demonstrate this concept for PPC, we showed that participant EGS was able to grasp an object and move it to a new location with a robotic limb, which combined his timing of the intended movements with machine vision and smart robotic algorithms (50). These studies confirmed the earlier monkey studies that PPC is a good candidate for signals for neuroprosthetic control.
Another subject, NS, also had a complete spinal cord lesion at C3-4 and was implanted in August 2014 after 6 y of paralysis. As with EGS, we found many action-based signals in recordings from NS. For example, the decision of whether a visual stimulus had been previously seen, and the confidence of the decision, were both encoded in PPC (52). Neurons were active not only for imagined hand movements but also shoulder movements and even speech (Fig. 2) (18). How was it possible to encode so many action variables in a population of 100 to 200 neurons?
A likely mechanism is that a distributed or mixed encoding is used for movement intention in PPC. A distributed representation for spatial location is found in PPC of monkeys, where retinotopic location of a visual stimulus is mixed in a multiplicative manner with eye position signals (39,53). Different memory tasks and objects are represented in a mixed fashion within the prefrontal cortex (54). Auditory and visual signals are randomly mixed in rat PPC (55). Based on these and other studies it has been proposed that mixed representations, in which neurons code more than 1 variable, are an effective way to encode and decode many variables using a small number of neurons (56).

Partially Mixed Selectivity in AIP
We found that AIP codes imagined or intended movements of much of the body (18). Fig. 2 shows example neurons in a task where written text is presented (the cue, for example, to imagine squeezing the right hand), followed by a blank screen delay period, and then a go signal to produce the instructed behavior. Fig. 2A shows an example of a neuron that was active for imagined or attempted movement of the right hand, Fig. 2B attempted left hand only, Fig. 2C imagined left or right hand movement, and Fig.  2D a neuron active for speaking "left" but not "right." In a more extensive study, we examined 8 combinations of task conditions: attempt vs. imagine (cognitive strategy), left vs. right (side), and shoulder vs. hand (effector). All these variables were mixed. However, the mixing was not random (the scenario depicted in Fig. 3A). Rather, the integration of signals was structured ( Fig.  3B) with more overlap of signals for strategy and side between neurons differing by 1 task variable. For example, a neuron responding to imagined left hand movement would also be more likely to respond to imagined right hand movement. However,  effectors were more separated with no statistically significant correlation between hand and shoulder (Fig. 3B). The structure of the population may reflect similarities in neural computations. For instance, left and right hand movements are similar and show a transfer of learning between the hands (57) and the use of motor imagery improves performance during execution (58). The shoulder and hand may be more dissimilar in computational terms and thus be more separated neurally.
Data from the monkey and human studies cited above would suggest that association cortices such as the PPC process information in a high-dimensional feature space. In fact, this high dimensionality may explain some of the disagreements in the literature. A classic example is area LIP in monkeys, which has been suggested to govern saccades (41), attention (59), decision making (60), and categorization (61). Monkey studies require training animals for long periods of time for an experiment, often months or even a year. Thus the same animals are rarely tested in dissimilar tasks. With humans a task can be trained in 1 d through verbal instruction, and many different tasks can be performed by the same individual.
A second question is whether primary cortical areas such as motor cortex operate in a lower-dimensional processing space than association cortex. From what is known about monkey neurophysiology, primary visual cortex may be lower-dimensional, coding orientation, ocular dominance, motion direction, and color (62). A popular analysis of motor cortex activity is to use dimensionality reduction; although this technique reveals only the dimensions for the motor task involved, these analyses do hint at low-dimensional representations (63). On the other hand, 10 degrees of freedom have been demonstrated for the control of a robotic limb using M1 activity in which these dimensions were distributed across the activity of the neural population, indicating a mixed encoding for these variables (64). One obvious way to address this issue would be to examine the number of action variables to which human motor cortex is sensitive.

Write-In Interface
The above examples are "read-out" BMIs; they record neural activity and make sense of it with decoders. A second major class is "write-in" BMIs; these use electrical stimulation to input information to the brain. Write-in BMIs are currently common in clinical practice. Examples are cochlear prosthetics that stimulate the auditory nerve to provide hearing for the deaf, deep brain stimulation of basal ganglia sites for motor disorders including Parkinson's disease and essential tremor, and functional electrical stimulation of muscles for certain forms of paralysis.
Write-in BMIs would be important for providing somatosensory feedback for more dexterous control of robotic hands for object manipulation (65). Patients with spinal cord injury are not only paralyzed but frequently cannot feel cutaneous stimulation or the sense of body position (proprioception) below the level of the injury. To guide a robotic hand toward an object using brain control, patients are limited to vision as sensory feedback. However, once an object is grasped, dexterous manipulation of the object requires somatosensory feedback (19). Further, using vision alone to guide movements of the body is notoriously difficult and mentally exhausting as described by patients with somatosensory deficits such as large-fiber sensory neuropathy. Somatosensory percepts can be artificially generated by electrical stimulation of somatosensory cortex. An instantiation of such feedback is envisioned with bidirectional BMIs. Such a BMI would control a robot limb with recorded neural activity. At the same time the robotic hand would be covered with sensors for touch and hand position. These sensors would communicate with an electrical stimulator that would then stimulate the somatosensory cortex, providing feedback.
Electrical stimulation of the surface of cortex in humans using relatively large currents produces somatotopically localized sensations, but these sensations are not natural and are reported as a tingling or buzzing (66,67). ICMS, in which small amounts of current are delivered through microelectrodes whose tips are positioned within the cortex, therefore recruiting much smaller populations of neurons, has been tested in healthy monkeys (20)(21)(22)(68)(69)(70)(71). With this technology, monkeys behave similarly in response to both a natural mechanical stimulus and electrical ICMS (20,68).
Recently there have been 2 studies in paralyzed humans in which ICMS has been applied to the somatosensory cortex of tetraplegic subjects (23,24). In both studies, 2 arrays of 48 stimulating microelectrodes were implanted in Brodmann's area 1 (BA1) of somatosensory cortex. In the study by Gaunt and coworkers (24), the implants were made within the hand representation. They describe a systematic topography for stimulation through different electrodes in the array. The subject reported that the sensations were quasi-natural, similar to pressure being applied to the skin. In a study performed in our laboratory, we implanted arrays that were largely in the lower and upper arm representation of BA1 (Fig. 4A) (23). While a topography was found between the arrays, within the arrays the topography was rather mixed and not systematic. On the other hand, the subject reported natural sensations that were both cutaneous and proprioceptive (Fig. 4B). The cutaneous sensations included squeezing, tap, vibration, and so on, and proprioceptive sensations were reported as movement of the limb. Interestingly, at higher currents proprioception was more prevalent, opening the possibility that the sensations can be controlled by the parameters of the stimulation.

BMI Learning
An important issue, primarily studied to date in monkeys, is how much learning can be achieved with BMIs. If plasticity is universal, then a BMI could be implanted in any area of cortex and trained to perform any task. If, on the other hand, plasticity is more limited, then it is important to select the appropriate cortical areas to maximize the control of the variables desired by the BMI. Early, groundbreaking research by Fetz and colleagues showed that monkeys could learn to control the activity of individual neurons in motor cortex (25,72). We have seen similar biofeedback control in human recordings, but in this case the subjects reported that they controlled the activity by performing specific imagined movements (17).
Microelectrode array recordings in monkeys have shown a variety of degrees of learning from a great deal of plasticity to very little plasticity (25,(27)(28)(29)(30)(72)(73)(74). We have framed the issue as one of individual neuron learning or intrinsic variable learning (29). Individual neuron learning would be produced by the systematic rewarding of the activity of a single neuron, which over time would change the selectivity of that single neuron and not its neighbors. Since there is a great deal of variability in neural responses this correlation of reward with a single cell's activity would be leveraged to produce individual neuron learning. Intrinsic variable learning would involve the subject searching the space of possible responses that an area normally uses to control the activity of the neuron. The example of finding an imagined movement to modulate a neuron would be an example of intrinsic variable learning.
There are 3 features that distinguish between the 2 learning hypotheses. First, for individual neuron learning only the trained neuron will show plasticity and not other neurons in the cortical area. On the other hand, if intrinsic variable learning is used to modulate the neuron, then all neurons with similar tuning in the area will show a similar modulation (Fig. 5). Second, for individual neuron learning any pattern of activity can be learned. For intrinsic variable learning, only patterns of activity that are already intrinsic to the cortical area can be used for learning. Distinguishing between these 2 hypotheses in humans has the advantage that they can report verbally the strategy they use during learning. In intrinsic variable learning they would report that they searched a space of possible solutions. For instance, if the decoder rotates the goal locations the subjects may say they imagine reaiming the movement. For individual variable learning they may not use any cognitive strategy and may not even be aware of how they solved the task.
In both monkey (29) and human PPC (31), we found that all 3 of the above features were consistent with intrinsic variable learning and not individual neuron learning on the timescale of 1 d (Fig. 5). Similar results, consistent with intrinsic variable learning, were found in monkey primary motor cortex (28), in which only existing patterns of activity could be manipulated by the monkeys. Over the timescale of many days, individual neuron changes have been reported (32)(33)(34)(35) and hypothesized to be the result of motor skill learning (33,35). The above results indicate that the intrinsic structure of cortical areas is maintained on the short term, and even on the long term the type of learning engaged may be restricted to the type of learning a cortical area normally does; for instance, motor skill learning may be apparent only in motor and sensorimotor cortical areas. In other words, BMI learning may not be infinite, and it is important to choose an area that codes the intrinsic variables, and type of learning, that one wishes to tap for BMI control.
Progress Toward the Future At present, BMIs work in the laboratory. There are several challenges toward introducing them to the clinic in order to help paralyzed individuals. For instance, doctors can routinely implant a cochlear prosthetic, but not a motor neuroprosthetic.
For some challenges the technology already exists but requires a concerted effort and application of resources. The implants need to be made wireless, so that no cables connect the subject to the external, assistive device. If implants contain active circuits, they must be well-sealed, to protect the electronics, and powerefficient, to avoid heating the brain.
Other challenges will require advances in technology. The electrodes need to be made longer, to access cortex in the sulci; more recording sites need to be available to increase the neuronal yield; and the electrodes need to be flexible, so they will move with the cortex during micromovements in order to improve recording stability. Entirely new technologies will need to be developed if high temporal and spatial resolution recordings are to be made noninvasively. One example is to use ultrasound or photoacoustic recordings to monitor changes in blood volume. Both these technologies have the limitations imposed by the acoustic signal attenuation by the skull. All of these technical developments will require basic engineering research that is first tested in monkeys.
There are also scientific challenges on the horizon. Better understanding of how information is encoded in different cortical areas will lead to the design of more sophisticated decoders. Basic science exploration of areas not widely studied, such as language areas and prefrontal areas, can potentially lead to the design of new types of prosthetics. For instance, direct language decoding would be beneficial for patients locked in due to stroke or ALS.
On a personal note, it is a tremendous research experience to perform basic research in monkeys, learn scientific principles of brain function, and then be involved in the direct translation of this knowledge to clinical applications in humans. Research in monkeys makes these translations particularly fruitful for topics such as motor control, vision, and cognition given the close similarity in behavioral abilities and neural circuitry between monkeys and us.