For the first time, an international team of engineers and computer scientists has developed technology that combines radio frequency detection with artificial intelligence to read lips and identify her movements.
Today's hearing aids help people with hearing loss by amplifying all the sounds of their surroundings, which is useful in several applications. In noisy environments, however, the wide amplification spectrum of these devices can make it difficult for users to focus on specific sounds. For example a conversation with a certain person.
One possible solution to this problem, known as the "cocktail party effect", is to make "smart" hearing aids. New devices that combine conventional audio amplification with a second device that collects additional data for better performance.
Lip reading singularity edition
In a new article published today in the magazine Nature Communications (I link it here), the team led by the University of Glasgow shows the use of state-of-the-art sensing technology to read lips. Their system preserves privacy by collecting only radio frequency data, without exploiting videos (and therefore without privacy problems, at least on images).
To develop the system, the researchers asked male and female volunteers to repeat the five vowels (A, E, I, O and U) first without a mask and then wearing a surgical mask. Their faces were scanned both with their mouths closed and during pronunciation, using radiofrequency signals from a dedicated radar sensor and a wifi transmitter.
The 3.600 data samples collected with the scans were used to "train" machine learning and deep learning algorithms to recognize characteristic mouth movements and to read lips, associating each movement with a sound.
The system showed an accuracy rate 95% for non-masked lips, and 83% for masked lips. Impressive.
Doctor Qammer Abbasi from the University of Glasgow, the lead author of the paper, illustrates the work done. "About 5% of the world's population, 430 million people, has a form of hearing impairment. Hearing aids have made a difference for many of them. New technologies that collect data to improve sound amplification could take a decisive step. "
In summary: this research shows that radio frequency signals, and even Wi-Fi signals, can allow you to read lips even when covered by a mask. I leave it to everyone's imagination to think about the "negative" uses of this technology, and focus only on the positive aspects.
Future multimodal hearing aids will break down any difference between people, bringing that 5% of the world's hearing impaired population on the same "wavelength" (it should be said) as everyone else.