The Evolution of Brain-Computer Interfaces
Eric C. Leuthardt
The notion that a computer can decode brain signals to infer the intentions of a human subject and then enact those intentions directly through a machine is emerging as a realistic technical possibility. These types of devices are known as brain-computer interfaces or BCIs. The evolution of these neuroprosthetic technologies could have significant implications for patients with motor disabilities by enhancing their ability to interact and communicate with their environment.
Classically, the cortical physiology that has been most investigated and utilized for device control has been brain signals from the primary motor cortex. To date, this classic motor physiology has been an effective substrate for demonstrating the potential efficacy of BCI-based control. Emerging research in cortical physiology, however, now stands to further enhance our understanding of the cortical physiology underpinning human intent and provides further signals for more complex brain-derived control. In this review, we discuss the current status of BCIs and detail the emerging research trends that stand to further augment clinical application in the future.
The notion that the brain can be directly accessed to allow a human being to control an external device with thoughts alone is emerging as a real option for patients with motor disabilities. This area of study, known as neuroprosthetics, has sought to create devices, known as brain-computer interfaces (BCIs), that acquire brain signals and translate them to machine commands such that they reflect the intentions of the user. In the past 20 years, the field has progressed rapidly from fundamental neuroscientific discovery to initial translational applications.
Examples are seen in the seminal discoveries by Georgopoulus and Schwartz that neurons in the motor cortex, when taken as a population, can predict the direction and speed of arm movements in monkeys (Georgopoulos et al., 1982, 1986; Moran and Schwartz, 1999a). In the subsequent decades, these findings were translated to increasing levels of brain-derived control in monkeys and to preliminary human clinical trials (Hochberg et al., 2006; Taylor et al., 2002). Fundamental to the evolution of neuroprosthetic application, this brain-derived control is dependent on the emerging understanding of cortical physiology as it encodes information about intentions. In recent years, an emerging understanding of how the cortex encodes motor and nonmotor intentions, sensory perception, and the role that cortical plasticity plays in device control have led to new insights in brain function and BCI application. These new discoveries stand to further expand the potential of neuroprosthetics in regards to both control capability and patient populations that will be served. In this review, we provide an overview of current BCI modalities and of emerging research on the use of nonmotor areas for BCI applications, and we assess their potential for clinical impact.
DEFINITION AND ESSENTIAL FEATURES
A BCI is a device that can decode human intent from brain activity alone in order to create an alternate communication channel for people with severe motor impairments. More explicitly, a BCI does not require the “brain’s normal output pathways of peripheral nerves and muscles” to facilitate interaction with one’s environment (Wolpaw et al., 2000, 2002). A real-world example of this would entail a quadriplegic subject controlling a cursor on a screen with signals derived from individual neurons recorded in the primary motor cortex without the need of overt motor activity. It is important to emphasize this point. A true BCI creates a completely new output pathway for the brain.
As a new output pathway, the user must have feedback to improve the performance of how one alters one’s electrophysiological signals. Similar to the development of a new motor skill (e.g., learning to play tennis), there must be continuous alteration of the subject’s neuronal output. The output should be matched against feedback from intended actions such that the subject’s output (swinging the tennis racket or altering a brain signal) can be tuned to optimize performance toward the intended goal (getting the ball over the net or moving a cursor toward a target). Thus, the brain must change its signals to improve performance, but additionally the BCI may also be able to adapt to the changing milieu of the user’s brain to further optimize functioning. This dual adaptation requires a certain level of training and a learning curve, both for the user and for the computer. The better the computer and subject are able to adapt, the shorter the training that is required for control.
There are four elements essential to the practical functioning of a BCI platform (Figure 1):
1. Signal acquisition, the BCI system’s recorded brain signal or information input;
2. Signal processing, the conversion of raw information into a useful device command;
3. Device output, the overt command or control functions that are administered by the BCI system; and
4. Operating protocol, the manner in which the system is altered and turned on and off (Wolpaw et al., 2002).
All of these elements play in concert to manifest the user’s intention in his or her environment.
Signal acquisition is some real-time measurement of the electrophysiological state of the brain. This measurement of brain activity is usually recorded via electrodes, but this is by no means a theoretical requirement. These electrodes can be either invasive or noninvasive. The most common types of signals include electroencephalography (EEG), electrical brain activity recorded from the scalp (Elbert et al., 1980; Farwell and Donchin, 1988; Freeman et al., 2003; Pfurtscheller et al., 1993; Sutter, 1992; Vidal, 1977); electrocorticography (ECoG; Leuthardt et al., 2004, 2005), electrical brain activity recorded beneath the skull (Leuthardt et al., 2004, 2005; Schalk et al., 2004a); field potentials, electrodes monitoring brain activity from within the parenchyma (Andersen et al., 2004); and “single units,” microelectrodes monitoring individual neuron action potential firing (Georgopoulos et al., 1986; Kennedy and Bakay, 1998; Laubach et al., 2000; Taylor et al., 2002). Figure 2 shows the relationship of the various signal platforms in terms of anatomy and population sampled. Once acquired, the signals are then digitized and sent to the BCI system for further interrogation.
FIGURE 1 Essential features and components of a BCI. There are four essential elements to the practical functioning of a brain computer interface platform: (1) signal acquisition, the BCI system’s recorded brain signal or information input; (2) signal processing, the conversion of raw information into a useful device command; (3) device output, the overt command or control functions that are administered by the BCI system; and (4) operating protocol, the manner in which the system is turned on and off, and the way the user or a technical assistant adjusts parameters of the previous three steps in converting intentions to machine commands. All of these elements play in concert to manifest the user’s intention in his or her environment (Schalk et al., 2004b). Source: Leuthardt et al. (2009).
In the signal-processing portion of BCI operation, there are two essential functions: feature extraction and signal translation. The first function extracts significant identifiable information from the gross signal, and the second converts that identifiable information into device commands. The process of converting a raw signal into one that is meaningful requires a complex array of analyses. These techniques can vary from assessment of frequency power spectra, event-related potentials, and cross-correlation coefficients for analysis of EEG or ECoG signals to directional cosine tuning of individual neuron action potentials (Levine et al., 2000; Moran and Schwartz, 1999a; Pfurtscheller et al., 2003). The impetus for these methods is to determine the relationship between an electrophysiologic event and a given cognitive or motor task. As an example, after recordings are made from an ECoG signal, the BCI system must recognize that a signal alteration has occurred in the electrical rhythm (feature extraction) and then associate that change with a specific cursor movement (translation). As mentioned above, it is important that the signal processing be dynamic such that it can adjust to the changing internal signal environment of the user. In regards to the actual device output, this is the overt action that the BCI accomplishes. As in the previous example, this can result in moving a cursor on a screen; other possibilities are choosing letters for communication, controlling a robotic arm, driving a wheelchair, or controlling some other intrinsic physiologic process such as moving one’s own limb or controlling one’s bowel and bladder sphincters (Leuthardt et al., 2006a).
FIGURE 2 Signals for BCI. Three general categories of signals that are used for BCI application and their anatomic location relative to the brain and its respective covering layers. EEG, electroencephalography; ECoG, electrocorticography. Source: Leuthardt et al. (2009).
An important consideration for practical application is the overall operating protocol. This refers to the manner in which the user controls how the system functions. The “how” includes such things as turning the system on or off, controlling the kind of feedback and how fast it is provided, how quickly the system implements commands, and switching between various device outputs.
These elements are critical for BCI functioning in the real-world application of these devices. In most current research protocols, these parameters are set by the investigator. In other words, the researcher turns the system on and off, adjusts the speed of interaction, or defines very limited goals and tasks. These are all things that the user will need to be able to do by him- or herself in an unstructured applied environment.
CURRENT BCI PLATFORMS
There are currently three general categories of BCI platforms that have been put forward as possible candidates for clinical application. These categories are primarily determined by the source from which the controlling brain signal is derived. The first category uses EEG, which involves brain signals acquired from the scalp. The second category, referred to as “single-unit systems,” uses intraparenchymal microelectrodes that detect action potential firings of individual neurons. The third is an intermediate modality in which electrodes acquire signal from the cortical surface directly (either above or below the dura). The current status of each of these platforms is briefly reviewed in terms of level of control, surgical considerations, and current clinical populations served.
EEG-based BCIs use electrical activity recorded from the scalp (Birbaumer et al., 1999; Blankertz et al., 2006; Farwell and Donchin, 1988; Kübler et al., 2005; McFarland et al., 1993, 2008a; Millan et al., 2004; Muller et al., 2008; Pfurtscheller et al., 1993, 2000; Sutter, 1992; Vaughan et al., 2006; Wolpaw and McFarland, 1994, 2004; Wolpaw et al., 1991). Most BCI studies in humans used EEG, probably because this recording method is convenient, safe, and inexpensive.
EEG has a relatively poor spatial resolution. This is because a large brain area must be involved to generate the necessary detectable signals (Freeman et al., 2003; Srinivasan et al., 1998). Despite this limitation, signals relevant to BCI research can still be found in the EEG. This includes modulations of mu (8-12 Hz) or beta (18-25 Hz) rhythms produced by sensorimotor cortex. These rhythms show nonspecific changes (typically decreases in amplitude) related to movements and movement imagery. They do not contain specific information about the details of movements, such as the position or velocity of hand movements. This may be an important limitation because signals associated with specific movement parameters are typically used in BCI systems based on action potential firing rates. Another issue of EEG recordings is that the detected amplitudes are very small. This makes them susceptible to artifacts created by sources outside the brain such as electromyographic signals produced by muscle contractions. Despite these potentially limiting issues, EEG-based BCIs have been shown to support higher performance than is often assumed, including accurate two-dimensional (McFarland et al., 2008a; Wolpaw and McFarland, 2004) and even three-dimensional control of a computer cursor (McFarland et al., 2008b).
To date, the large majority of clinical application of BCI technologies for people with severe motor disabilities have been demonstrated using EEG (Kübler et al., 2005; Nijboer et al., 2008; Vaughan et al., 2006). Ultimately, this intrinsic lack of signal robustness may have important implications for chronic application of BCI systems in real-world environments. BCI systems based on EEG typically require substantial training (Birbaumer, 2006; Wolpaw and McFarland, 2004) to achieve accurate one- or two-dimensional device control (about 20 or 50 thirty-minute training sessions, respectively), although some reports have reported training requirements that are shorter (Blankertz et al., 2006). These shortcomings of noise sensitivity and prolonged training are fundamental limitations in the scalability of widespread clinical application of EEG-based BCIs.
In summary, EEG has been shown to support much higher performance than previously assumed and is currently the only modality that has been shown to actually help people with paralysis. However, because of its important limitations, it is currently not clear to what extent EEG-based BCI performance, in the laboratory and in clinical settings, can be further enhanced.
From a purely engineering point of view, the optimal method to extract electrical information from the brain would be to place a series of small recording electrodes directly into the cortical layers (1.5–3 mm) to record signals from individual neurons. This, in essence, is what single-unit action potential BCI systems do, and they have been very successful for limited time periods in both monkeys (Carmena et al., 2003; Serruya et al., 2002; Taylor et al., 2002; Velliste et al., 2008) and humans (Hochberg et al., 2006; Kennedy and Bakay, 1998). To extract singleunit activity, small microelectrodes having ~20-micron-diameter tips are inserted in the brain parenchyma where relatively large (e.g., 300 microvolt) extracellular action potentials are recorded from individual neurons from 10-100 microns away.
These signals are usually bandpass filtered from 300 to 10,000 Hz and then passed through a spike discriminator to measure spike time occurrences. The firing rates of individual neurons are computed in 10- to 20-millisecond bins and “decoded” to provide a high-fidelity prediction to control either a computer cursor or robot endpoint kinematics (Georgopoulos et al., 1986; Moran and Schwartz, 1999b; Wang et al., 2007). Given its high spatial resolution (100 microns) as well as its high temporal resolution (50-100 Hz), this modality arguably provides the highest level of control in BCI applications.
Unfortunately, there are two major problems with single-unit BCIs. First, the electrodes must penetrate into the parenchyma, where they cause local neural and vascular damage (Bjornsson et al., 2006). Second, single-unit action potential microelectrodes are very sensitive to encapsulation. Insertion of penetrating devices in the brain parenchyma damages neurons and vasculature, which can initiate a cascade of reactive cell responses, typically characterized by activation and migration of microglia and astrocytes toward the implant site (Bjornsson et al., 2006). The continued presence of devices promotes formation of a sheath composed partly of these reactive astrocytes and microglia (Polikov et al., 2005; Szarowski et al., 2003). This reactive sheath can have numerous deleterious effects, including neural cell death and an increased tissue resistance that electrically isolates the device from the surrounding neural tissue (Biran et al., 2005; Szarowski et al., 2003; Williams et al., 2007). Research into novel biomaterial coatings and/or local drug delivery systems that may reduce the foreign body response to implanted electrodes is ongoing but, to date, is far from clinical application (Abidian and Martin, 2008; Seymour and Kipke, 2007; Spataro et al., 2005). Until these issues are solved, this remains a limitation for developing a long-term BCI system based on single-unit activity.
Over the past 5 years, the use of ECoG as a signal platform for BCI has gained mounting enthusiasm as a more practical and robust platform for clinical application. As detailed above, both EEG- and single-unit-based systems have been impeded for large-scale clinical application. This is either due to prolonged user training and poor signal-to-noise limitations with EEG, or due to inability to maintain a consistent signal with current single-unit constructs (Bjornsson et al., 2006; Szarowski et al., 2003; Wolpaw and McFarland, 2004). The use of ECoG has been posited to be an ideal trade-off for practical implementation (Leuthardt et al., 2004). When compared to EEG, the signal is substantially more robust. Its magnitude is typically five times larger, its spatial resolution is much greater (0.125 versus 3.0 cm for EEG), and its frequency bandwidth is significantly higher (0-500 Hz versus 0-40 Hz for EEG) (Boulton et al., 1990; Freeman et al., 2003; Srinivasan et al., 1998). Of particular note, the access to higher-frequency bandwidths carries particularly useful information amenable to BCI operation (Gaona et al., 2011). Many studies have demonstrated that different frequency bands carry specific and anatomically distinct information about cortical processing. The lower-frequency bands known as mu (8-12 Hz) and beta (18-25 Hz), which are detectable with EEG, are thought to be produced by thalamocortical circuits and show broad anatomic decreases in amplitude in association with actual or imagined movements (Huggins et al., 1999; Levine et al., 1999; Pfurtscheller et al., 2003; Rohde et al., 2002). The higher frequencies only appreciable with ECoG, also known as gamma band activity, are thought to be produced by smaller cortical assemblies. Gamma activity shows close correlation with action potential firing of tuned cortical neurons in the primary motor cortex in monkey models (Heldman et al., 2006). Additionally, these high-frequency changes have been associated with numerous aspects of speech and motor function in humans (Chao et al., 2010; Crone et al., 1998, 2001a, 2001b; Gaona et al., 2011; Leuthardt et al., 2004; Schalk et al., 2007). Beyond higher information content, since the ECoG signal is recorded from larger electrodes that do not penetrate the brain, these constructs should have a higher likelihood for long-term clinical durability. This expectation of good long-term stability of ECoG sensors is supported by some pathologic and clinical evidence. For example, in cat, dog, and monkey models, long-term subdural implants showed minimal cortical or leptomeningeal tissue reaction while maintaining prolonged electrophysiologic recording (Bullara et al., 1979; Chao et al., 2010; Loeb et al., 1977; Margalit et al., 2003; Yuen et al., 1987). Additionally, preliminary work in humans using the implantable NeuroPace device for the purpose of long-term subdural electrode monitoring for seizure identification and abortion has also been shown to be stable (Vossler et al., 2004). The use of ECoG for BCI applications has been primarily studied in motorintact patients with intractable epilepsy requiring invasive monitoring. Similar to EEG-based BCI systems, the ECoG approach has primarily focused on the use of changes in sensorimotor rhythms from motor cortex. What has been distinct, however, has been the access to the higher-frequency gamma rhythms with ECoG. Use of these higher-frequency rhythms has provided a significant advantage in regards to training requirements and multidimensional control. In 2004, Leuthardt et al.
demonstrated the first use of ECoG in closed-loop control in a one-dimensional cursor control task with minimal training requirements (under 30 minutes). In additional experiments, the same group and others have demonstrated that specific frequency alterations encode very specific information about hand and arm movements (Leuthardt et al., 2004; Pistohl et al., 2008; Sanchez et al., 2008; Schalk et al., 2007). In 2006, Leuthardt et al. further demonstrated that ECoG control using signal from the epidural space was also possible (Leuthardt et al., 2006b). Schalk et al. (2008) showed that ECoG signals can be used for two-dimensional control whose performance was within the range of that shown before using invasive single-unit systems. Because the electrode arrays cover broad regions of cortex, several groups have begun to explore alternate cognitive modalities and their cortical physiologies to expand BCI device control. Felton et al. (2007) showed that, in addition to motor imagery, sensory imagery could also be used for device control. The same group also demonstrated that the auditory cortex could be trained to acquire simple control of a cursor (Wilson et al., 2006). Ramsey et al. (2006) showed that higher cognitive functions, such as working memory in the dorsal lateral prefrontal cortex, can also be used for effective device operation.
Recently, Leuthardt et al. (2011) demonstrated that phonemic content taken from speech networks could also be used for simple device control. Taken together, these studies show that ECoG signals carry a high level of specific cortical information, and that these signals can allow a user to gain control rapidly and effectively. It is worth noting that these control paradigms have not been extended to motor-impaired subjects thus far. How these cortical signals will be affected in the setting of a spinal cord injury or ALS has not been explicitly tested.
The field of neuroprosthetics is growing rapidly. The cortical physiology that underpins the manner in which a human brain encodes intentions is beginning to be understood. This will have a significant impact in augmenting function for those with various forms of motor disabilities. As research stretches beyond motor physiology, the field of neuroprosthetics now stands to further expand in capability and in diversity of clinical populations served. The evolving understanding of cortical physiology as it relates to motor movements, language function, and plasticity could all provide higher levels of complexity in brain-derived control. Given the rapid progression of these technologies over the past decade and the concomitant swift ascent of computer processing speeds, signal analysis techniques, and emerging ideas for novel biomaterials, neuroprosthetic implants will hopefully in the near future be as common as deep brain stimulators are today. The clinical advent of this technology will usher in a new era of restorative neurosurgery and new human-machine interfaces.