Brain activity decoded to deliver synthetic speech

People who have lost speech due to stroke, brain injury or neurological disease currently rely on painfully slow synthesisers that track eye movement to spell out words. Perhaps best associated with the late Stephen Hawking, these technologies are limited to around 10 words per minute. By comparison, natural speech is in the region of 100-150 words per minute.

The researchers, based out of UC San Francisco (UCSF), established in a previous study how the brain’s speech centres encode movements for the lips, jaw, and tongue rather than direct acoustic information. For the latest work, they first gathered recordings of patients reading sentences while also logging the corresponding brain activity. Using linguistic principles, they then reverse engineered the vocal movements required to produce the sounds and mapped them to the brain activity associated with those sounds. This allowed the team to create a realistic virtual vocal tract for each participant that could be controlled by their brain activity. The work, published in Nature, could pave the way for devices that replicate natural speech in real-time.