VISUAL AID
COMPUTER BRAIN INTERFACE DEVICE AIMS TO HELP BLIND PEOPLE ‘SEE’
Mark Pappas, who has been blind for 14 years, searched for a packet of sugar on the table in front of him by running his tongue across the 400 electrodes in his mouth.
Using the information from the electrodes, he was able to locate the sugar, a white plastic spoon and a white paper cup that had been placed on a black cloth. Then he walked down a hallway without the help of his white cane.
Mr. Pappas was demonstrating an experimental device called the BrainPort, which translates information from a video camera on his sunglasses into a pattern of tiny electric shocks on a small mouthpiece that he “reads” with his tongue.
I’ve been able to do things I didn’t think I’d be able to. Like walking down a hallway without using a white cane.”
— Mark Pappas
Learning to interpret the sensations required extensive training, but Mr. Pappas said the BrainPort is a valuable navigation aid. It can detect obstacles above waist height that his cane would otherwise miss. He can use it to also read short words, such as a bus stop sign. He has even used it while putting a golf ball and kicking around a soccer ball.
For the past four years, UPMC researchers have been working to test and improve the device, which was developed by Wicab, a Wisconsin company. The UPMC team is also trying to understand how the brain adapts to sensing the world in an entirely new way.
“It’s very out-of-the-box thinking,” said Amy Nau, an UPMC optometrist and director of optometric and low vision services, UPMC Eye Center, who leads the research. Other high-tech visual aids such as retinal implants can’t be used by people with damaged optic nerves or traumatic damage to the eyes, she said.
The BrainPort can bypass the eye and optic nerve using an approach called sensory substitution, which works by rerouting the information from an impaired sense through an alternative sense. Sensory substi-
tution devices being developed by other companies include the vOICe, which converts visual information into a soundscape, and the AuxDeco, which delivers a pattern of electrical shocks to the skin of the forehead.
The BrainPort was approved for sale in Canada in September and is under review by the U.S. Food and Drug Administration.
One advantage of using the tongue for sensory substitution is that the wet surface improves the performance of the electrodes. The tongue is also sensitive enough to resolve fine details. “If there was a hair in your food, you’d know right where to go to get it,” said Jacqueline Fisher, the research coordinator for the UPMC study.
The intensity of the electrical stimulation at each electrode on the mouthpiece corresponds to the intensity of light from the camera. Objects at the bottom of the visual scene are represented at the tongue tip and objects that are higher up are sensed further back on the tongue. Controls on the device allow the user to zoom the camera view and to change the overall intensity of the electrical stimulation.
Mr. Pappas described the sensation as like pinpricks of static. Although the feeling is not unpleasant, he prefers to use it for short stretches of less than an hour. “Your tongue gets a little numb,” he said.
Brain imaging studies by the UPMC team suggest that people using the BrainPort activate portions of the visual cortex, a part of the brain that is normally responsible for processing information from the eyes. But the BrainPort can’t replace vision, cautioned Ms. Nau. “It’s not sight, it’s a visual aid,” she said.
The visual cortex is thought to become more “disorganized” the longer a person has been blind, she said. The team is trying to understand whether the brain adaptation process differs between people who have only recently lost their sight and those who have been blind since birth.
Mr. Pappas lost his sight to meningitis at age 23, after brain surgery to remove a pituitary gland tumor the size of a golf ball. He got involved in the research after reading an article about a BrainPort tester who had lost his sight while serving in Iraq.
Mr. Pappas recently completed an 18-month study in which he used the BrainPort around his home in New Castle, Lawrence County. Although he found the BrainPort useful for certain household tasks, including folding laundry, there were challenges to using a device that can only represent simple scenes in high contrast.
“The resolution is not very good,” the optometrist said. Each eye normally detects light using 125 million photoreceptors; the BrainPort uses only 400 electrodes.
Even compared to a pixelated visual image, this lowresolution information is difficult for the brain to interpret. That’s partly because the tongue does not usually handle as much sensory information as the eyes. The effect is like a bandwidth restriction, she said. The data from the tongue seems to eventually reach the visual cortex, but it travels there via the part of the brain that normally processes the sense of touch. “Think of it as a smaller cable,” she said.
Another limitation is that the BrainPort shows the world in only two dimensions. Sighted people perceive depth — the distance to an object — in 3-D, using the slight difference in perspective that comes from each eye. But because the BrainPort uses a single camera, a large object that is far away looks the same as a small object that is close up.
To work around the BrainPort’s limitations, scientists from the Robotics Institute at Carnegie Mellon University are trying to develop computer software that could analyze the BrainPort video feed and automatically recognize certain objects and faces.
Former Pittsburgh PostGazette intern reporter Michelle Hackman, who was born with low vision and was blind by age 8, tried the BrainPort in a brief lab test. She said she would probably not use the device even if it were helpful for some tasks.
“Walking around with a giant probe on my tongue would divert everyone’s attention from me to the technology in my mouth,” she said. “One of the most important things is to put people at ease around me and I can’t do that when I’m using all sorts of weird accessibility technology.”
Wicab CEO Robert Beckman responded by saying that everyone has different priorities for how he or she uses visual aids. “The fact that one person has a particular objection to it, I don’t think it necessarily speaks for blind people in general.”
The BrainPort does prevent the user from speaking freely during use. “It’s a trade off,” said Mr. Pappas. He removes the mouthpiece whenever he needs to say something in the middle of a task. But he still appreciated the extra independence the BrainPort allows.
“I’ve been able to do things I didn’t think I’d be able to,” he said. “Like walking down a hallway without using a white cane.”
UPMC is still recruiting patients for ongoing studies. If you are interested. call 412-3839033.
Cristy Gelling: cgelling@postgazette.com. Post-Gazette reporter Mark Roth and former PostGazette intern Michelle Hackman contributed.