For the first time, an international team of engineers and computer scientists has developed technology that combines radio frequency sensing with artificial intelligence to read lips and identify his movements.
Today's hearing aids help people with hearing loss by amplifying all sounds in the surrounding environment, which is useful in several applications. In noisy settings, however, the broad amplification spectrum of these devices can make it difficult for users to focus on specific sounds. For example, a conversation with a certain person.
A possible solution to this problem, known as the “cocktail party effect”, is to create “smart” hearing aids. New devices that combine conventional audio amplification with a second device that collects additional data for better performance.
Lip reading singularity edition
In a new article published today in the magazine Nature Communications. (I link it here), University of Glasgow-led team shows use of cutting-edge detection technology to read lips. Their system preserves privacy by collecting only radio frequency data, without exploiting footage (and therefore without privacy problems, at least on images).
To develop the system, the researchers asked male and female volunteers to repeat the five vowels (A, E, I, O and U) first without a mask and then wearing a surgical mask. Their faces were scanned both with their mouths closed and during pronunciation, using radiofrequency signals from a dedicated radar sensor and a wifi transmitter.
The 3.600 data samples collected with the scans were used to "train" machine learning and deep learning algorithms to recognize characteristic mouth movements and read lips, associating each movement with a sound.
The result?
The system showed an accuracy rate 95% for unmasked lips, and 83% for masked ones. Impressive.
Doctor Qammer Abbasi from the University of Glasgow, the lead author of the paper, explains the work done. “About 5% of the world's population, 430 million people, have some form of hearing impairment. Hearing aids have made a difference for many of them. New technologies that collect data to improve sound amplification could take a decisive step forward."
In summary: This research shows that radio frequency signals, and even Wi-Fi signals, can allow you to read lips even when they are covered by a mask. I leave reflection on the "negative" uses of this technology to everyone's imagination, and I focus only on the positive aspects.
Future multimodal hearing aids will break down any differences between people, bringing that 5% of the world's population with hearing problems onto the same "wavelength" (it must be said) as all the others.
