Wissenschaftler arbeiten an der Entwicklung einer künstlichen Intelligenz, die Hirnaktivität in Text umwandeln kann. Das soll Menschen, die nicht sprechen oder Texte eintippen können, Kommunikation ermöglichen. NICOLA DAVIS berichtet.
From brain waves to text
Reading someone’s mind has just come a step closer to reality. Scientists have developed artificial intelligence that can turn brain activity into text. The system currently works on neural patterns detected while someone is speaking aloud, but experts say the system could eventually aid communication for patients who are unable to speak or type, such as those with locked-in syndrome. “We are not there yet, but we think this could be the basis of a speech prosthesis,” said Dr Joseph Makin, co-author of the research from the University of California, San Francisco.
Writing in the journal Nature Neuroscience, Makin and his colleagues reveal how they developed their system. The researchers recruited four participants who had electrode arrays implanted in their brain to monitor epileptic seizures. These participants were asked to read aloud a set of 50 sentences multiple times, including “Tina Turner is a pop singer” and “Those thieves stole 30 jewels”. The team tracked their neural activity while they were speaking.
An imperfect system
This data was then fed into a machinelearning algorithm — a type of artificial intelligence system. The algorithm converted the brain activity data for each spoken sentence into an abstract string of numbers. To make sure the string of numbers related only to aspects of speech, the system compared sounds that were predicted from small chunks of the brain activity data with actual recorded audio.
The string of numbers was then fed into a second part of the system that converted it into a sequence of words. At first, the system spat out nonsensical sentences. But as the system compared each sequence of words with the sentences that were actually read aloud, it improved. It learned how the string of numbers related to words, and which words tend to follow each other.
The team then tested the system, generating written text only from brain activity during speech. The system was not perfect. Among its mistakes, “Those musicians harmonize marvellously” was decoded as “The spinach was a famous singer”, and “A roll of wire lay near the wall” became “Will robin wear a yellow lily”. However, the team found the accuracy of the new system was far higher than in previous approaches. While the accuracy varied from person to person — for one participant, just three per cent of each sentence on average needed correcting — it was better than the average word error rate of five per cent for professional human transcribers.
But, the team stress that, unlike human transcribers, the algorithm handles only a small number of sentences. “If you try to go outside the [50 sentences used] the decoding gets much worse,” said Makin, adding that the system is likely relying on a combination of learning particular sentences, identifying words from brain activity and recognizing general patterns in English.
Not translation of thought
The team also found that training the algorithm on one participant’s data meant less training data was needed from the final user — something that could make training less difficult for patients. Dr Christian Herff, an expert in the field from Maastricht University who was not involved in the study, said the research is exciting because the system used less than 40 minutes of training data for each participant, and a limited collection of sentences, rather than the millions of hours typically needed. “By doing so, they achieve levels of accuracy that haven’t been achieved so far,” he said.
He noted, however, that the system is not yet usable in severely disabled patients, as it relies on the brain activity recorded from people speaking a sentence out loud. “Of course, this is fantastic research … but this is not translation of thought [but of brain activity involved in speech].”
Herff added that people should not worry about others reading their thoughts just yet — the brain electrodes must be implanted, and imagined speech is very different from inner voice. But Dr Mahnaz Arvaneh, an expert in brain– machine interfaces at Sheffield University, UK, said it was important to consider ethical issues now. “We [are still] very, very far away from the point that machines can read our minds,” she said. “But it doesn’t mean that we should not think about it and we should not plan about it.”
“Those musicians harmonize marvellously” was decoded as “The spinach was a famous singer”