Neuroscientists have created brain-reading software which can decode speech signals in real-time into written sentences, potentially transforming the lives of people who have lost the ability to speak. The software is the first of its kind to show how the intention to say specific words can be extracted from the brain activity and converted into text quickly enough to keep pace with a normal conversation. It also decodes speech with listening and speaking considered together rather than separately, unlike previous studies. Created by scientists at the University of
California, San Francisco (UCSF), the model currently only works for certain basic sentences, but it is hoped it can help create an advanced system which can decode the words a person intends to say in real-time. Patients who experience facial paralysis, due to a stroke or spinal cord injury, may partially or completely lose their ability to speak. But the areas in the brain which control the muscles of the jaw, lips, tongue, and larynx to produce speech are often intact and active, it could then be possible to use these intentional speech signals to decode what patients are trying to say. Electrodes were fitted onto the surface of the brain to analyse brain signals. Credit: Noah Berger/ Noah Berger “Currently, patients with speech loss due to paralysis are limited to spelling words out very slowly using residual eye movements or muscle twitches to control a computer interface,” said Edward Chang, a speech neuroscientist and one of the paper’s authors. “But in many cases, information needed to produce fluent speech is still there in their brains. We just need the technology to allow them to express it.” The research, published in the journal Nature and funded by
Facebook, placed small electrodes directly on the surface of the brain of three epilepsy patients, all of whom had normal speech. The electrodes allowed researchers to record their brain activity, while they were asked a set of nine questions and asked to read a list of 24 potential responses. Using this data, the team then created a computer model which was trained to match patterns of brain activity to the questions the patients heard and the answers they gave. The software was able to identify from the brain signals what questions the patient had heard and the response they gave, with an accuracy of 76% and 61% respectively. The model is currently limited in scope because it was trained on a small set of words, but researchers are hopeful they can increase its “flexibility” and “accuracy” in the future.