I can't tell you exactly what gets lost in translation when TV series, movies and shows are dubbed. I have a friend who would skin me to see works in the original language with subtitles. I prefer dubbing (sometimes even in the same Italian films: what would Tomas Milian be without Ferruccio Amendola?).
The startup team Flawless promises never to make us think of such a question again. The company says it solved the problem of language barriers with an innovation: deepfake dub.
Deepfake dubbing by Flawless, towards a multimedia "Babel"
We often think about deepfake like manipulating the entire image of a person or a scene, but Flawless technology focuses on a single element: the mouth.
Customers feed the company's software with video from a movie or TV show along with dubbed dialogue recorded by humans. Flawless machine learning models then create new lip movements that correspond to the translated speech and automatically "glue" them onto the actor's face.
:extract_focal()/https%3A%2F%2Fcdn.vox-cdn.com%2Fuploads%2Fchorus_image%2Fimage%2F69304783%2Fflawless_deepfake_dub.0.gif)
"When someone watches this dubbed footage, they aren't shaken by the performance by a jarring word or a wrong mouth movement," says Flawless co-founder. Nick Lynes.
The actors maintain the original performance and style, whatever language they are speaking.
Nick Lynes
On the right track
The results aren't 100% flawless yet, but already pretty good. In the demonstration video (see below) you will find a French dub of "Codice d'onore" (A few good men "), a 1992 legal drama starring Jack Nicholson and Tom Cruise. The French native speakers interviewed found it much more fluid. of traditional dubbing.
What makes Flawless technology really interesting is the value for money. Deepfake voiceovers are cheap and quick to create, especially when compared to the cost of full remakes. And, with the advent of global streaming platforms like Netflix, Disney Plus, and Amazon Prime Video, it's easier than ever to find space for this content.
Better than a remake
Flawless' deepfake voiceovers preserve, to some extent, the original actor's performance.
Flawless technology is based on research from the Max Planck Institute for Informatics published for the first time in 2019. As you can see in the video below, voiceovers are sensitive to the facial expressions of the artists.
Another Round example
The feature of preserving the original performance can be interesting for directors who want to keep the magic of their original casting. An example? The 2020 Oscar-winning Danish film, another round, which sees Mads Mikkelsen being part of a group of teachers who they experience alcoholism to see if it improves their lives. Following the success at home, the film is ready to be remade for an English-speaking audience with Leonardo DiCaprio in the lead role.
Why?
Is the culture of Danish drinkers (backbone of the film) really so foreign to American audiences that a remake is necessary? Mikkelsen, already appeared in films such as Hannibal , Doctor Strange e Rogue One, is it so unknown that it cannot attract viewers to other countries?
A deepfake dubbing would be a cheaper way to bring another round to the English-speaking audience while retaining its original flavor. "They're talking about $ 60 million to make that movie," says Lynes.
If we are offering something that is 2% of the cost of the remake, if the film was also attractive half the remake would offer 10x better value.
Those responsible for the remake will obviously have other concerns besides money. No matter how beloved Mikkelsen is, he is not as bankable as DiCaprio. But Lynes hopes that as deepfake voiceovers become common, she will change the calculations for such remakes in the future. Much more, he says, could even reshape the international film landscape, allowing actors and directors to reach new audiences with minimal effort.
Considerations
I think the attractiveness of actors will change globally as a result of this technology. The performances of different people and the choices of the directors will be better recognized, because a wider audience will be able to see them. In the (distant?) Future, the “best foreign film” category could disappear, and the Oscar could simply become a universal award.
