I can't tell you exactly what gets lost in translation when TV series, movies and shows are dubbed. I have a friend who would skin me to see works in the original language with subtitles. I prefer dubbing (sometimes even in the same Italian films: what would Tomas Milian be without Ferruccio Amendola?).
The startup team Flawless promises never to make us think of such a question again. The company says it solved the problem of language barriers with an innovation: deepfake dub.
Deepfake dubbing of Flawless, towards a multimedia “Babel”.
We often think about deepfake like manipulating the entire image of a person or scene, but Flawless' technology focuses on a single element: the mouth.
Customers feed the company's software video from a movie or TV show along with voiced dialogue recorded by humans. Flawless' machine learning models then create new lip movements that match the translated speech and automatically paste them onto the actor's face.
:extract_focal()/https%3A%2F%2Fcdn.vox-cdn.com%2Fuploads%2Fchorus_image%2Fimage%2F69304783%2Fflawless_deepfake_dub.0.gif)
“When someone watches this dubbed footage, they aren't shaken out of the performance by a jarring word or bad mouth movement,” says the Flawless co-founder Nick Lynes.
The actors maintain the original performance and style, whatever language they are speaking.
Nick Lynes
On the right track
The results are not yet 100% flawless, but already quite good. In the demonstration video (see it below) you will find a French dubbing of “Code of Honor” (A few good men), a 1992 legal drama starring Jack Nicholson and Tom Cruise. Native French speakers interviewed found it much smoother than traditional dubbing.
What makes Flawless technology really interesting is the quality-price ratio. Deepfake dubs are cheap and quick to create, especially when compared to the cost of full remakes. And, with the advent of global streaming platforms like Netflix, Disney Plus and Amazon Prime Video, it's easier than ever to find space for this content.
Better than a remake
Flawless's deepfake dubs preserve, to some extent, the original actor's performance.
Flawless technology is based on research from the Max Planck Institute for Informatics published for the first time in 2019. As you can see in the video below, voiceovers are sensitive to the facial expressions of the artists.
Another Round example
The feature of preserving the original performance can be attractive to directors who want to maintain the magic of their original casting. An example? The 2020 Oscar-winning Danish film, another round, which sees Mads Mikkelsen being part of a group of teachers who experience alcoholism to see if it improves their lives. Following the success at home, the film is ready to be remade for an English-speaking audience with Leonardo DiCaprio in the lead role.
Why?
Is the culture of Danish drinkers (backbone of the film) really so foreign to American audiences that a remake is necessary? Mikkelsen, already appeared in films such as Hannibal , Doctor Strange e Rogue One, is it so unknown that it cannot attract viewers to other countries?
A deepfake dubbing would be a cheaper way to bring another round to the English-speaking audience while maintaining its original flavor. “They're talking about $60 million to make that movie,” Lynes says.
If we are offering something that is 2% of the cost of the remake, if the film was also attractive half the remake would offer 10x better value.
Those responsible for the remake will obviously have other concerns besides money. No matter how beloved Mikkelsen is, he is not as bankable as DiCaprio. But Lynes hopes that as deepfake voiceovers become common, she will change the calculations for such remakes in the future. Much more, he says, could even reshape the international film landscape, allowing actors and directors to reach new audiences with minimal effort.
Considerations
I think the attractiveness of actors will change globally as a result of this technology. The performances of different people and the choices of the directors will be better recognized, because a wider audience will be able to see them. In the (distant?) Future, the “best foreign film” category could disappear, and the Oscar could simply become a universal award.
