Los Angeles: Scientists have created a virtual vocal tract — completes with lips, jaw and tongue — that can generate natural-sounding synthetic speech by using brain signals.
The brain-machine interface created by neuroscientists at University of California, San Francisco in the US could one day restore the voices of people who have lost the ability to speak due to paralysis and other forms of neurological damage.
Stroke, traumatic brain injury, and neurodegenerative diseases such as Parkinson’s disease, multiple sclerosis, and amyotrophic lateral sclerosis (ALS, or Lou Gehrig’s disease) often result in an irreversible loss of the ability to speak.
Some people with severe speech disabilities learn to spell out their thoughts letter-by-letter using assistive devices that track very small eye or facial muscle movements.
However, producing text or synthesised speech with such devices is laborious, error-prone, and painfully slow, typically permitting a maximum of 10 words per minute, compared to the 100-150 words per minute of natural speech.
The system, described in the journal Nature, demonstrates that it is possible to create a synthesised version of a person’s voice that can be controlled by the activity of their brain’s speech centres.
In the future, this approach could not only restore fluent communication to individuals with severe speech disability, researchers said, but could also reproduce some of the musicality of the human voice that conveys the speaker’s emotions and personality.
“For the first time, this study demonstrates that we can generate entire spoken sentences based on an individual’s brain activity,” said Edward Chang, a professor at University of California, San Francisco.
Researchers asked five volunteers with intact speech who had electrodes temporarily implanted in their brains to map the source of their seizures in preparation for neurosurgery to treat epilepsy — to read several hundred sentences aloud while the researchers recorded activity from a brain region known to be involved in language production.
Based on the audio recordings of participants’ voices, the researchers used linguistic principles to reverse engineer the vocal tract movements needed to produce those sounds: pressing the lips together here, tightening vocal cords there, shifting the tip of the tongue to the roof of the mouth, then relaxing it, and so on.
This detailed mapping of sound to anatomy allowed the scientists to create a realistic virtual vocal tract for each participant that could be controlled by their brain activity.
This comprised two ‘neural network’ machine learning algorithms: a decoder that transforms brain activity patterns produced during speech into movements of the virtual vocal tract, and a synthesiser that converts these vocal tract movements into a synthetic approximation of the participant’s voice.
The synthetic speech produced by these algorithms was significantly better than synthetic speech directly decoded from participants’ brain activity without the inclusion of simulations of the speakers’ vocal tracts, the researchers found.
The algorithms produced sentences that were understandable to hundreds of human listeners in crowdsourced transcription tests.
PTI