A computer lab working on simplifying human-computer interaction for people of all abilities has developed a digital lip reader with its own machine learning system to help the user understand better.
LipType is an invention of Professor Ahmed Sabbir Arif and his laboratory. In short: read the lip movement. It reads it so well that it allows people to send text messages or emails on PCs and smartphones, and to have contactless interactions with public devices such as ATMs or other kiosks. without speaking aloud.
There are other lip-reading devices, but they are not widely used because they are slow and often faulty.
A leap forward
There are a lot of mistakes in dictation, especially in noisy places. People with language problems also have problems making themselves understood, perhaps because of the accent. LipType works for anyone. People may need to send a private message while in a public space, or in a meeting, and with LipType they can just move their lips without making a sound.
Arif his students have added various filters with different algorithms artificial intelligence to allow lip reading in different lighting conditions. They then introduced an error corrector based on different language models and found that LipType was significantly faster than other existing devices.
The lip reading test
To accompany the software test, Arif's lab conducted a social study to see if people would use such a technology. They contacted students and people from the community, including people with disabilities, and conducted an online survey. The reactions were enthusiastic.
The results of the software tests were also outstanding. "LipType performed 58% faster than other models," says the researcher Laxmi Pandey. “The device has proven itself at the cutting edge in various real-world contexts, with low light and noise. The success of LipType makes me believe that it can revolutionize our interaction with each other and with IT devices ”.
The social study and the LipType lip reader study have been accepted for publication and presentation at the leading international conference on human-computer interaction.
The possible applications of the LipType lip reader
There are several possible applications for this lip reading system. Obviously LipType can bring people with disabilities very close to digital tools. In many cases it could be useful for law enforcement, to understand what happens in scenarios in which they cannot access by listening. Car interfaces can also be useful, so you can write messages while listening to music. There is a design philosophy for everything: the limit is the imagination.
