The project involves basic research needed to make possible a brain-computer interface for decoding thought and communicating it to an intended target. Applications are to situations in which it is either impossible or inappropriate to communicate using visual means or by audible speech; the long-term aim is to provide a significant advance in Army communication capabilities in such situations. Non-invasive brain-imaging technologies like electroencephalography (EEG) offer a potential way for dispersed team members to communicate their thoughts. A Soldier thinks a message to be transmitted. A system for automatic imagined speech recognition decodes EEG recordings of brain activity during the thought message. A second system infers simultaneously the intended target of the communication from EEG signals. Message and target information are then combined to communicate the message as intended.

• Overview
• EEG and brain-computer interfaces
• Imagined speech production
• Intended direction
• Potential applications
• References
• Publications

Overview

In 1967, Dewan published a paper in Nature in which was first described a method for communicating linguistic information by brain waves measured using EEG. He trained himself and several others to modulate their brains’ alpha rhythms: to turn these rhythms on and off at will. Alpha rhythms reflect brain neuron activity, at or about a frequency of 10Hz, concerning not only whether the eyes are open but also one’s state of attention. Mental activity and attention abolish these rhythms, which are normally present in a state of mental relaxation. Dewan was able to signal letters of the alphabet using Morse code by voluntarily turning these rhythms on and off, with eyes closed. Signalling such letters, one by one, provides the words and phrases that the communicator has in mind.

In 1988, Farwell and Donchin described a second method for transmitting linguistic information. This method is based on the P300 response, again measured using EEG. The P300 is evoked when a person is presented a stimulus that matches what it is they are looking for: a target. Farwell and Donchin display to the thinker the letters of the alphabet, one by one, and eventually display the letter that he or she has in mind. The P300 potential would be evoked, for that target letter, so signalling the thinker’s desire to communicate that letter. Again, thinkers can communicate words by signalling the word’s letters one by one.

Can one use brain waves that are more directly linked to speech production to communicate linguistic information? Speech is a natural method for communicating linguistic information. Were one able to use EEG to measure directly the activity of brain speech networks, one could potentially develop an easier and faster method for communicating linguistic information using EEG. Our work on covert speech production pursues this idea. Covert speech is the technical term used to refer to the words one hears in one’s head while thinking: imagined speech. Can we use EEG to measure brain activity during covert speech production in a way that lets one communicate linguistic information in a natural and rapid way?

The work aims also to determine, from brain waves, where the linguistic information should be sent: sent in a particular direction, sent to a particular person, etc. The question is not so much how the message should be sent (e.g., wireless text messaging) but where or to whom. Work on the relationship between alpha rhythms and attention has, since Dewan’s time, revealed that the pattern of alpha rhythm activity in the two hemispheres of the brain provides information on where a person is focusing attention. For example, paying attention to an area in the left half of one’s visual field causes the alpha rhythm activity in the right hemisphere of the brain to desynchronize (and so diminish in intensity), and vice versa. These shifts in brain activity are thought to be helpful in directing more sensory and cognitive resources to the area being attended (e.g., Corbetta and Shulman, 2002). EEG can be used to measure patterns of alpha rhythms (Worden et al., 2000; Sauseng et al., 2005), to measure electric potentials that are evoked in response to a shift in attention (e.g.,Harter et al., 1989; Corbetta et al., 1993), and to measure the change in amplitude of steady-state responses that are evoked by a shift in attention among frequency-tagged visual stimuli (e.g., Srinivasan et al., 2006). We are studying alpha rhythms, evoked responses and steady-state evoked potentials measured using EEG to help develop a brain-computer interface that helps the thinker communicate to where or to whom a message should be sent.

Finally, we aim to learn more about activity in brain networks when two or more tasks are carried out simultaneously. Many studies in cognitive neuroscience involve brain-imaging measurements taken during the performance of a single task (e.g., visual detection, language processing, decision-making). Covert speech production and direction intention are likely to use differing brain resources. Can these differences be used by a brain-computer interface to infer both a communicator’s message and the recipient?

EEG and brain-computer interfaces

A tremendous amount of scentific and engineering progress has been made over the past several decades in developing brain-computer interfaces based on EEG measurements of brain network activity. One indicator of this progress is that there are now (at least) three companies which are developing EEG-based technologies for use with computer and console games: Emotiv Systems, OCZ Technology, and Neurosky. The idea is that an EEG headset, worn by the player, provides signals concerning what action the player wants to take in the game, whether the player is paying attention to the game, etc. Game software which is responsive to information provided by the EEG device guides and modifies gameplay.

Research on brain-computer interfaces has historically been motivated more by biomedical applications. For example, people who have suffered strokes or injuries to their brain, as well as those suffering from certain diseases like Lou Gehrig’s disease (ALS), may have impaired movement for part or all of their bodies while preserving a good deal of normal brain function. Can a person signal how they would like to move their arm or move a cursor on a computer screen under such circumstances? Much work has led to success here. For example, researchers at Pitt and CMU showed recently that a monkey can control movement of a prosthetic arm using brain waves measured using implanted electrodes. Electrocorticographic (ECoG) measurements in humans show parallel promise for motor control through a brain-computer interface.

Our work uses the non-invasive technologies EEG, MEG (magnetoencephalography – measurement of magnetic field fluctuations at the scalp caused by brain activity) and fMRI(functional magnetic resonance imaging). EEG measures electric field fluctuations at the surface of the scalp caused by brain activity. While it has the advantages of portability, relatively low cost and a high temporal resolution-ability to track rapid events in the brain, it has several disadvantages (Nunez and Srinivasan, 2006). First, its spatial resolution is limited to about two centimeters; electric potential changes in the brain spread diffusely as they move towards the scalp surface where measurements are made. Second, EEG is also sensitive to electric field potential changes caused by muscle. Movements of the eyes, movements of muscles beneath the scalp, etc., create large electric potential changes that can swamp signals from neurons in the brain. MEG has spatial resolution similar to EEG. MEG has the further disadvantage that it relies on very expensive equipment that can only be used in a room which which is completely shielded from external sources of electromagnetic radiation. An advantage of MEG over EEG is that it is better able to measure activity in brain cortical areas which are oriented perpendicularly to the scalp’s surface: brain cortex that lies in the folds. fMRI, finally, has very good spatial resolution (on the order of 1 millimeter) but has a poor temporal resolution. The blood oxygenation level signal relied on in fMRI measurement is sluggish. fMRI is thus useful for localizing brain activity in space but is limited in determining when that activity occurs. While our focus is on an EEG-based brain computer interface, we use MEG and fMRI to acquire further information about underlying brain activity which is not available in EEG signals.

Imagined speech production

One of the simplest possible ways to test whether EEG provides information concerning thought expressed through imagined (covert) speech is as follows. A person who wears an EEG headset is shown either the letter “y” or the letter “n” very briefly. A second or two later, the person thinks to him or herself the word “yes” or the word “no”, depending on whether the displayed letter was “y” or “n”, respectively. Do this many times while recording the EEG signals. One way to analyze the EEG data from such an experiment is to attempt toclassify the data. The aim is to use the EEG signal information alone to distinguish those times when the person was thinking “yes” from those times when the person was thinking “no”. If the EEG signals provide enough information to classify accurately the “yes” and the “no” thoughts, then one has made good progress.

However, one cannot rest satisfied with such a result. For example, it may be the case that one can tell apart “yes” from “no” using EEG because the EEG signals in response to the displayed letters “y” and “n” differ. This would mean that the visual responses to the letters used to cue the thinker, rather than the covertly spoken words, lead to discernible differences in the EEG recordings. One wonders more generally how a classification result depends on the prompt: for example, a seen “y” vs. a heard “y”. It could be also the case that a particular person, while remaining silent, just happens to move his or her vocal tract muscles when thinking “yes” but not when thinking “no”. This would mean that the EEG-based differences between thinking “yes” and “no” depend on the degree of motor response. Indeed, there are many ways in which a straightforward interpretation of classifiability can prove false, and a major aim of our work is to conduct experiments that pin down more precisely what brain networks are contributing to EEG classification results.

Theta-band activity depends on the rhythm of covert speech production. The leftmost panel plots frequency spectra of EEG brain waves from single scalp locations; theta-band activity (in the 4-7Hz range, see inset) is weak when speaking covertly the syllable /ba/ at a slow, steady rate. The middle panel shows that strong theta-band activity is elicited when thinking /ba/ in a rhythm involving three /ba/ productions for every one at left. The strong difference in theta-band activity is localized to scalp positions atop central frontal cortex. Our pilot studies have found that EEG provides information that helps to classify covertly-spoken sentences, words, syllables, meters and rhythms.

A more complex type of experiment involves training. Suppose that thinking different words leads to small differences in EEG signals. One can try to help a person generate stronger, more informative EEG signals by providing feedback during an experiment. The person uses feedback information concerning the strength of EEG-based information to produce stronger signals. Such experiments typically follow earlier classification experiments. The reason is that classification experiments provide needed information on what aspects of the EEG recordings provide information that lets one differentiate what is spoken covertly.

The strongest published results for classification using EEG concern speech that one listens to rather than speech that one produces covertly. In the late 1990s, a group at Stanford succeeded in classifying EEG signals recorded while listening to small sets of words or sentences, with mixed success (Suppes et al., 1997, 1998). MEG has also been used to classify heard speech. Numminen and Curio (1999) used MEG to study auditory cortical responses to speech and showed activity related to speech monitoring in both overt and covert production (see also Houde et al., 2002). Ahissar and colleagues (2001) recorded MEG data from natural spoken sentences presented at different time-compression ratios. Their principal component analysis of the MEG data showed that one component correlated well with envelope information and its degradation consequent to compression.

Luo and Poeppel (2007) showed that, in single trials recorded when listening to natural spoken sentences, the phase of the theta band (3-8Hz) recorded from auditory cortex plays a critical role in speech representation. MEG theta band responses relate to the syllabic structure of sentence materials, which in turn affects response temporal envelope.

Classification of heard sentences using MEG (after Luo and Poeppel, 2007). A. Skull distribution of theta-band phase representation of syllabic structure. (B) Perfect classification of a small set of sentences based on MEG theta-band responses (central diagonal). Strongest competing sentences in the classification are very similar to the correct sentence.

A neurolinguistic framework for speech production is needed to understand and pursue such results. Studies of cortical speech mechanisms suggest that, within temporal and frontal lobe cortices, there is a direct speech production pipeline that ranges from earlier, conceptually-driven word selection through later selection of corresponding articulatory motor commands (Hickok and Poeppel, 2004, 2007; Indefrey and Levelt, 2004).

Brain areas involved in speech processing (after Hickok and Poeppel, 2004). A ventral processing stream (the what pathway) maps sound and meaning. Activity in the superior temporal sulcus and the posterior inferior temporal lobe (pITL) interfaces sound-based speech representations in the superior temporal gyrus and widely-distributed conceptual representations. A dorsal processing stream (the where, how or do pathway) involves activity in cortex at the junction of parietal and temporal lobes in the sylvian fissure (Spt), which projects to frontal cortical areas that generate articulatory codes: posterior inferior frontal and dorsal premotor sites (pIF/dPM).

Much evidence concerning the speech production pipeline comes from studies using EEG techniques, which have the temporal resolution needed to discern staged processing. These studies use evoked response potentials like the N200, a go/no-go signal with magnitude a function of the neural activity required for response inhibition, to measure times at which various stimuli interfere with speech production (e.g., Schiller et al., 2003). MEG has also contributed; Salmelin and colleagues (1994) used MEG to trace the time-course of cortical activation during both overt and covert picture naming. Results suggest that syllables are basic representations in cortical speech production, and that they are generated serially from representations of syntactically-marked words and used to retrieve gestural information that drives motor articulation: concepts to words to syllables to phonemes to motor articulation. The Levelt model of speech production relates this linguistic pipeline to cortical activity (Indefrey and Levelt, 2004) localized in fMRI and PET studies using a variety of overt and covert speech production tasks. The model localizes (1) lexical selection from conceptual processes to left medial temporal gyrus and environs; (2) retrieval of a word’s phonological code some 70 msec later to (left) Wernicke’s area; (3) sequential syllabification of a word some 100 msec later to (left) Broca’s area, and (4) phonetic encoding of the syllables some 150 msec later to left inferior lateral frontal cortex and to right supplementary motor area. Can EEG be used to make sense of the rumbling of this pipeline?

Our intent is to learn what linguistic information can be extracted from EEG recordings of this direct speech production pipeline, when one thinks to oneself. We are particularly interested in the involvement of brain networks which help with speech motor articulation and with brain networks involved in generating the auditory images which accompany covert speech: the words heard in one’s head while thinking. Our expectation is that decoding the EEG recordings of a covert speech stream successfully will rely on context in a way similar to that found when performing standard automatic speech recognition (ASR). A particular element of speech that is signaled through a spoken speech waveform, be it a phoneme, syllable or word, is more reliably identified when taken in the context of preceding speech elements. We will work to adapt standard ASR to the decoding of EEG signals concerning covert speech streams.

Intended Direction

The project aims also to discern from EEG recordings an intended direction that may be signaled by a thinker to select a target of communication. Two components are of special interest: EEG signals concerning overt orienting movements, like those of the eyes, and signals concerning shifts of attention. Saccadic eye movements are overt indicators of attentional orientation that depend on a generator network spanning cortical frontal eye fields and subcortical neurons in substantial nigra, superior colliculus and the brainstem (Boucher et al., 2007). Shifts in attention may occur covertly and are thought to result from the activity of attentional circuits in frontal and parietal lobes (Corbetta and Shulman, 2002). These are thought to feed back onto visual areas in occipital cortex (Praamstra et al., 2005); such feedback is thought to promote the facilitation of sensory processing from the intended direction.

Shift in gaze is closely related to shift of attention. A premotor theory of attention (Rizzolatti et al., 1994) suggests that the allocation of spatial attention involves planning for but not executing a saccade. Yet it is possible to shift attention without shifting gaze (Hoffman and Subramanian, 1995), and some evidence suggests that spatial attention shifts may occur in the absence of saccade preparation (Juan et al., 2004).

Event-related potentials measured using EEG and which result from a shift of attention are threefold. They include EDAN, an early posterior negativity in the hemisphere contralateral to the attended hemifield, thought related to the spatial information provided by a cue in a covert orienting task (Harter et al., 1989); LDAP, a later contralateral positivity thought related to facilitation of sensory areas (Harter et al., 1989) and ADAN, an enhanced negativity at frontal contralateral electrodes likely linked to activation of frontal lobe neurons involved in the control of spatial shifts (ADAN, Corbetta et al., 1993; Nobre et al., 2000). These ERPs are supramodal, in that they occur independently of the sensory modality used to modulate attention (Eimer et al., 2002).

Alpha-band activity in frontal, parietal and occipital cortex, recorded by EEG in the 8-14 Hz range, provides further information concerning visuospatial attention that may very possibly be recovered reliably from single trials. Alpha-band amplitudes are suppressed in parieto-occipital cortex contralateral to the covertly-attended hemifield and enhanced in cortex contralateral to the to-be-ignored hemifield (Worden et al., 2000; Sauseng et al., 2005; Thut et al., 2006; Capotosto et al., 2008). Synchronization in the form of alpha-band phase-coupling increases between frontal and parieto-occipital alpha activity in the hemisphere contralateral to the attended region, which suggests that the posterior modulation of alpha activity in contralateral posterior parieto-occipital cortex is controlled by prefrontal regions (Sauseng et al., 2005).

Attentional shifts modulate steady-state visual evoked potentials (SSVEPs). These potentials can be studied by amplitude-modulating visual stimuli at particular frequencies (e.g.,Srinivasan et al., 2006). One extracts the response to such a frequency-tagged visual stimulus by examining energy in EEG records at the frequency of tagging (Ding et al., 2006). Attending to a stimulus location modulates the SSVEP through energy increase, even at locations attended covertly (Morgan et al., 1996; Kelly et al., 2005).

Spatial shifts of attention that depend on auditory stimulation also give rise to event-related potentials (e.g., Teder-Salejarvi et al., 1999), modulation of alpha-band activity, and modulation of steady-state evoked potentials, which suggest that such shifts are spatial, not merely visual. Finally, shifts in attention are thought to occur not only among spatial locations but among object features like color (Muller et al., 2006) and among objects themselves.

We hypothesize that EEG recordings related to spatial attention in single trials can be used in four basic ways to provide information concerning intended direction.

• One can discern the hemifield to which attention is lateralized through analysis of attentional modulation of alpha-band activity and of steady-state visual and auditory evoked potentials;
• One can discern which of several frequency-tagged objects attention is directed towards through analysis of attentional modulation of steady-state visual and auditory evoked potentials
• One can estimate a continuous-valued intended direction by considering information concerning shift in gaze captured through EEG and through eye-tracking, in addition to attentional modulation of alpha-band activity and of steady-state evoked potentials, and
• By using gaze direction and steady-state evoked potential information one can most reliably discern the intended target of communication, as these provide information concerning both attended direction and attended object.

Potential Applications

The funded research is basic in nature. A functioning brain-computer interface for communicating thought and the intended recipient like that described above is years away. Yet one can identify several areas of future application. These include the development of a silent communications system for dispersed ground forces, of a speech-based means of communication for locked-in individuals, and of commercial communications devices based on brain-wave decoding.

References

Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H. & Merzenich, M.M. (2001). Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proceedings of the National Academy of Sciences 98, 13367-13372.

Boucher, L., Palmeri, T.J., Logan, G.D. & Schall, J.D. (2007). Inhibitory control in mind and brain: an interactive race model of countermanding saccades. Psychological Review 114, 376-397.

Capotosto, P., Babiloni, C., Romani, G.L. & Corbetta, M. (2008). Posterior parietal cortex controls spatial attention through modulation of anticipatory alpha rhythms. Nature Precedings Feb. 1.

Corbetta, M., Miezin, F.M., Shulman, G.L. & Petersen, S.E. (1993). A PET study of visuospatial attention. Journal of Neuroscience 13, 1202-1226.

Corbetta, M. & Shulman, G.L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews (Neuroscience) 3, 201-215.

Dewan, Edmond M. (1967). Occipital alpha rhythm eye position and lens accomodation. Nature 214, 975-977.

Ding, J., Sperling, G. & Srinivasan, R. (2006). Attentional modulation of SSVEP power depends on the network tagged by the flicker frequency. Cerebral Cortex 17, 1016-29.

Eimer, M., van Velzen, J. & Driver, J. (2002). Cross-modal interactions between audition, touch, and vision in endogenous spatial attention: ERP evidence on preparatory states and sensory modulations. Journal of Cognitive Neuroscience 14, 254-271.

Farwell, L.A. & Donchin, D. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology 70, 510-523.

Harter, M.R., Miller, S.L., Price, N.J., LaLonde, M.E. & Keyes, A.L. (1989). Neural processes involved in directing attention. Journal of Cognitive Neuroscience 1, 223-237.

Hickok, G. & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67-99.

Hickok, G. & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience 8, 393-402.

Hoffman, J.E. & Subramanian, B. (1995). The role of visual attention in saccadic eye movements. Perception & Psychophysics 57, 787-795.

Houde, J.F., Nagarajan, S.S., Sekihara, K. & Merzenich, M.M. (2002). Modulation of the auditory cortex during speech: an MEG study. Journal of Cognitive Neuroscience 15, 1125-38.

Indefrey, P. & Levelt, W.J.M. (2004). The spatial and temporal signatures of word production components. Cognition 92, 101-144.

Juan, C.-H., Shorter-Jacobi, S.M. & Schall, J.D. (2004). Dissociation of spatial attention and saccade preparation. Proceedings of the National Academy of Sciences 101, 15541-15544.

Kelly, S.P., Lalor, E.C., Reilly, R.B. & Foxe, J.J. (2005). Visual spatial attention tracking using high-density SSVEP data for independent brain-computer communication. IEEE Transactions on Neural Systems and Rehabilitation Engineering 13, 172-178.

Luo, H. & Poeppel, D. (2007). Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54, 1001-1010.

Morgan, S.T., Hansen, J.C. & Hillyard, S.A. (1996). Selective attention to stimulus location modulates the steady state visual evoked potential. Proceedings of the National Academy of Sciences 93, 4770-4774.

Muller, M.M., Andersen, S., Trujillo, N.J., Valdes-Sosa, P., Malinowski, P. & Hillyard, S.A. (2006). Feature-selective attention enhancaes color signals in early visual areas of the human brain. Proceedings of the National Academy of Sciences 103, 14250-14254.

Nobre, A.C., Sebestyen, G.N. & Miniussi, C. (2000). The dynamics of shifting visuospatial attention revealed by event-related potentials. Neuropsychologia 38, 964-974.

Numminen, J, & Curio, G. (1999). Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neuroscience Letters 272(1), 29-32.

Nunez, P.L. & Srinivasan, R. (2006). Electric fields of the brain: the neurophysics of EEG. 2nd Ed. (New York: Oxford University Press, 2006).

Praamstra, P., Boutsen, L. & Humphreys, G.W. (2005). Frontoparietal control of spatial attention and motor intention in human EEG. Journal of Neurophysiology 94, 764-774.

Rizzolatti, G., Riggio, L, & Sheliga, B.M. (1994). Space and selective attention. In Umilta, C. & Moscovitch, M. (Eds.), Attention and Performance XV (Cambridge, MIT), pp. 231-265.

Salmelin, R., Hari, R., Lounasmaa, O.V. & Sams, M. (1994). Dynamics of brain activation during picture naming. Nature 368, 463-5.

Sauseng, P., Klimesch, W., Stadler, W., Schabus, M., Doppelmayr, M., Hanslmayr, S., Gruber, W.R. & Birbaumer, N. (2005). A shift of visual spatial attention is selectively associated with human EEG alpha activity. European Journal of Neuroscience 22, 2917-2926.

Schiller, N.O., Bles, M., Jansma, B.M. (2003). Tracking the time course of phonological encoding in speech production: an event-related brain potential study. Cognitive Brain Research17, 819-831.

Srinivasan, R., Bibi, F.A. & Nunez, P.L. (2006). Steady-state visual evoked potentials: Distributed local sources and wave-like dynamics that are sensitive to flicker frequency. Brain Topography 18, 167-187.

Suppes, P., Lu, Z.-L. & Han, B. (1997). Brain wave recognition of words. Proceedings of the National Academy of Science USA 94, 14965-14969.

Suppes, P., Han, B. & Lu, Z.-L. (1998). Brain wave recognition of sentences. Proceedings of the National Academy of Science USA 95, 15861-15866.

Teder-Salejarvi, W.A., Hillyard, S.A., Roder, B. & Neville, H.J. (1999). Spatial attention to central and peripheral auditory stimuli as indexed by event-related potentials. Cognitive Brain Research 8, 213-227.

Thut, G., Nietzel, A., Brandt, S.A. & Pascual-Leone, A. (2006). Alpha-band electroencephalographic activity over occipital cortex indexes visuospatial attention bias and predicts visual target detection. Journal of Neuroscience 26, 9494-9502.

Worden, M.S., Foxe, J.J., Wang, N. & Simpson, G.V. (2000). Anticipatory biasing of visuospatial attention indexed by retinotopically specific alpha-band electroencephalography increases over occipital cortex. Journal of Neuroscience 20, 1-6.

Publications

D’Zmura, M., Deng, S., Lappas, T., Thorpe, S. & Srinivasan, R. (in press). Toward EEG sensing of imagined speech. Jacko, J.A. (Ed.), Human-Computer Interaction, Part I, HCII 2009, LNCS 5610 (Berlin: Springer) 40-48[PDF]

Srinivasan, R., Thorpe, S., Deng, S., Lappas, T. & D’Zmura, M. (in press). Decoding attentional orientation from EEG spectra.Jacko, J.A. (Ed.), Human-Computer Interaction, Part I, HCII 2009, LNCS 5610 (Berlin: Springer). [PDF]

 

 

 

     

Spies for Hire: Key Officers at Booz Allen Hamilton Spies for Hire: Key Officers at Booz Allen Hamilton