Darpa Pursues Neuroscience to Enhance Performance

This article first appeared in Aviation Week & Space Technology.

The Defense Advanced Research Projects Agency (Darpa) is researching how computers reading brain waves may one day speed up the ways intelligence analysts detect targets in satellite images and also alert platoon leaders when soldiers are losing situational awareness.

This may sound like a scenario out of the science fiction movie 2001: A Space Odyssey, in which a computer named Hal overrides instructions from an astronaut to take control of a spaceship. But in the Darpa experiments, the computer is just a tool that processes brain waves, of which the human being isn’t even aware, and turns them into actionable information.

Amy Kruse, the Darpa program manager for these projects, has a doctorate in neuroscience from the University of Illinois. She moved to Washington after graduation and became a technical consultant to the director of Darpa. The Defense Dept.’s interest in making operational use of neuroscience proved to be a great opportunity, and she says the research efforts she is now leading would not be done anywhere else.

In a summary of her programs at the annual DarpaTech conference in 2005, Kruse spelled out the importance of the work: “The operational environment will continue to become more crowded with information, so it is clear that our war fighters must be able to manage complex situations with faster, more accurate and more concentrated cognitive capabilities. This means that issues such as cognitive overload, fatigue and decision-making under stress are fast becoming crucial factors in performance.”

The latest project Kruse has been working on is the Neurotechnology for Intelligence Analysts (NIA) program. This effort builds on an earlier one titled Augmentated Cognition, or AugCog. One of the leading contractors on both efforts has been Honeywell.

Under a $4-million, multiphase contract, the company has been developing what it calls the Honeywell Image Triage System (HITS) for Darpa. Bob Smith, vice president for advanced technology at Honeywell Aerospace, explains that HITS takes a satellite image and breaks it up into smaller image “chips” that can be shown to an intelligence analyst like flash cards at a rate of 5-20 images per second.

The analyst’s brain is treated as a sensor: Electrical activity it produces is recorded from electrodes placed on the scalp, the same way electroencephalography (EEG) is used in hospitals to monitor brain activity. Then, when the analyst looks at one of the images flashing by, a scalp plot shows when there is increased brain activity.

As images flash by, the analyst is asked to look for a target such as an airplane. After viewing about 50 of the smaller images (chips), he is asked if he saw an airplane—and he may answer “no.” But digital signal processing of the brain wave activity reveals that, in fact, he did see an airplane on slide 32.

“This process allows us to do triage on large amounts of visual information we get from different soruces and improve an analyst’s ability to go through a large amount of imagery,” says Smith. In fact, the analyst can do the job 5-7 times faster using the triage system than unaided. This is because the triage system picks up brain waves showing recognition of a target even before the human analyst is cognizant he has spotted it.

Smith says it is the equivalent of a person seeing something “out of the corner of his eye.”

The NIA project aims to help the intelligence community deal with the growing problem of having an enormous amount of “visual media” flowing in for review. It is currently taking the analysts too long to turn the data into usable information that can be acted on by decision-makers and war fighters.

Smith says when researchers at Honeywell first told him about how the triage system would work, he doubted it was feasible. But the company took a multidisciplinary approach to the problem—combining the know-how of psychologists who focus on human factors, electrical and mechanical engineers and even avionics specialists who know signal-processing. Honeywell Aerospace is increasingly taking this sort of multipronged approach to a variety of research problems.

In fact, the effort “builds on what we have been doing in human factors in avionics for a long time,” says Smith, and relies on the company’s understanding of the man-machine interface.

Now that reading brain waves has been shown to work, the aim is to optimize it for use by a large population of analysts and at an accuracy rate that meets the needs of the intelligence agencies. The goal is to cut the time it takes an expert analyst to review imagery to a matter of 10 min. from the current 1 hr. An inexperienced analyst might improve his review time on the same material to 1 hr. from 6 hr.

Kruse says the NIA project found that sorting through 5-10 images per second is possible. There were eight contractors participating in phase one, and phase two is just beginning with three teams led by Teledyne Scientific and Imaging, Columbia University and Honeywell.

The target for phase three is a prototype designed for use by an intelligence agency. This “customer” could keep the prototype and try it out after the research is completed. The technology is nearly ready for operational use and there is a big incentive to get it into the field because it solves a problem analysts face every day.

Kruse says NIA built on the earlier AugCog research, which involved several phases and four contracting teams paired with four different military services. Then-DaimlerChrysler worked with the U.S. Marine Corps, Lockheed Martin with the Navy, Boeing with the Air Force and Honeywell with the Army. Honeywell’s team included 11 university and industry partners.

Smith says Honeywell equipped infantry soldiers with brain and physiological sensors and monitored the soldiers during field training exercises at the Army’s Aberdeen Proving Ground. The brain sensors were an EEG and a “functional near-infrared” sensor to monitor activity in the frontal lobe. The physiological sensors included ones for the heart (electrocardiogram) and eyes. The data was used to determine workload, state of cognitive activity and the soldier’s level of attentiveness at a particular time.

The point is that in combat, when a soldier is under stress and trying to take in too much information at one time, he can find himself in a situation where “tunnel vision” occurs. In a training situation, for example, the soldier may be looking for the enemy over a hill while being subjected to simulated fire. Then an explosion occurs nearby. Meanwhile, the platoon commander is yelling at the soldier to turn right and to move away from his current location. “But [the soldier] is focused on the enemy and is not hearing his commander due to information overload,” Smith says.

In the demonstration, it was shown that soldiers could be instrumented with a wireless computer to help them and their commander manage information overload. Knowing that a soldier is no longer absorbing additional data may suggest to the platoon leader that he shouldn’t give that person a key task during an attack.

Kruse says in AugCog, DaimlerChrysler worked with the Marines on a problem involving a team driving a vehicle, Lockheed Martin worked with the Navy on an Aegis weapon system simulation and Boeing worked with the Air Force on a single operator involved in mission planning for an unmanned aerial vehicle formation.

The AugCog program is completed, but Smith says he thinks it will be 5‑10 years before the sort of technology demonstrated for Darpa is ready for operational use.

Kruse sees a bright future for AugCog and NIA types of technology in the future. She says the interest in the operational use of neuroscience techniques is steadily increasing in the Defense Dept. and in the armed services. Technical challenges must be surmounted for these systems to reach operational use, but there is no question that the signals derived will have utility.

Pentagon Turns to Brain Implants to Repair Damaged Minds