What exactly is lost in translation when TV shows and films are subbed or dubbed into a new language? It’s a hard question to answer, but for the team at AI startup Flawless, it may be one we don’t have to think about in the future. The company claims it has the solution to this particular language barrier; a technical innovation that could help TV shows and films effortlessly reach new markets around the world: deepfake dubs.
We often think of deepfakes as manipulating the entire image of a person or scene, but Flawless’ technology focuses on just a single element: the mouth. Customers feed the company’s software with video from a film or TV show along with dubbed dialogue recorded by humans. Flawless’ machine learning models then create new lip movements that match the translated speech and paste them automatically onto the actor’s head.
“It’s all about retaining the performance.”
“When someone’s watching this dubbed footage, they’re not jolted out of the performance by a jarring word or a mistimed mouth movement,” Flawless’ co-founder Nick Lynes tells The Verge. “It’s all about retaining the performance and retaining the original style.”
The results — despite the company’s name — aren’t 100 percent flawless, but they are pretty good. You can see and hear how they look in the demo reel below, which features a French dub of the classic 1992 legal drama A Few Good Men, starring Jack Nicholson and Tom Cruise. We asked a native French speaker what they made of the footage, and they said it was off in a few places but still a lot smoother than traditional dubbing.
What makes Flawless’ technology particularly interesting is its potential to scale. Flawless’ pitch is that deepfake dubs offer tremendous value for money: they’re cheap and quick to create, especially when compared to the cost of full remakes. And, with the advent of global streaming platforms like Netflix, Disney Plus, and Amazon Prime Video, it’s easier than ever for such dubbed content to reach international markets.
As a recent report in The Wall Street Journal highlighted, demand for streaming services in the US is saturated and companies are now looking abroad for future growth. In the first quarter of 2021, for example, 89 percent of new Netflix users came from outside the US and Canada, while the service’s most watched show, Lupin, is a Parisian thriller.
“the vast majority of their consumers are going to be outside the US, over time”
“What you’re seeing is more and more streamers come online realizing the vast majority of their consumers are going to be outside the US, over time,” Erik Barmack, a former Netflix executive responsible for the company’s international productions, told the WSJ. “The question is how international does your content need to be to be successful.”
As Barmack suggests, there are different ways to answer this demand. You can create shows with local flavor that still entertain domestic viewers. You can do remakes of local hits for new audiences. And you can roll out the subs and dubs. But Flawless is betting that its technology provides a new option that will be particularly enticing for filmmakers.
This is because the company’s deepfake dubs preserve, to some degree, the performance of the original actor, says Lynes. Flawless’ technology is based on research from the Max Planck Institute for Informatics first published in 2019. As you can see in a showcase video below, the dubs it produces are somewhat sensitive to the facial expressions of the performers, retaining their emotion and line delivery.
Flawless has developed these techniques over the past three years, says Lynes, speeding up production time and reducing the amount of input footage. The end results are still a balance of automated dubbing and manual retouching (about 85 percent to 15 percent) but speedy to edit. “If something comes out we don’t particularly like we’ll do a few iterations; resubmit the training data in different forms and get another result,” says Lynes.
The company hopes that preserving the original performance will be appealing to filmmakers who want to retain the magic of their original casting. Lynes gives the example of the 2020 Oscar award-wining Danish film Another Round, which stars Mads Mikkelsen as one of a group of teachers who experiment with low-level alcoholism to see if it improves their lives. After its success at home and on the international award circuit, the film is set to be remade for English-language audiences with Leonardo DiCaprio in the main role.
When is a remake of a foreign film necessary?
The news sparked discussion about the value of such remakes. Is the Danish drinking culture that forms the film’s backbone really so alien to American audiences that a remake is required? Is Mikkelsen, an actor who’s appeared in such mainstream fare as Hannibal, Doctor Strange, and Rogue One, such an unknown that he can’t attract viewers in the US? And is the “one-inch barrier” of subtitles (to quote Parasite director Bong Joon Ho) simply too much for audiences to overcome?
From Lynes’ point of view, a deepfake dub would at least be a cheaper way to bring Another Round to English-language audiences while retaining its original flavor. “If we’re offering something that’s two percent the cost of the remake, we only need to be half as appealing to offer 10 times better value,” he says.
Those in charge of the remake will have concerns other than money, of course. No matter how beloved Mikkelsen is, he’s not as bankable as DiCaprio. But Lynes hopes that as deepfake dubs become common it’ll change the calculations for such remakes in future. Much more than that, he says, it’ll could even reshape the international film landscape, allowing actors and directors to reach new audiences with minimal effort.
“I think the pulling power of actors will change globally as a consequence of this technology,” he says. “Different people’s performances and directors’ choices will be better recognized, because a wider audience will be able to see them.”
Perhaps so, but for the moment, Flawless needs to prove that audiences actually want its technology. The company, which launched earlier this month, says it’s already got a first contract with a client it can’t name, but there’s no timeline for when we might see its wares in a commercial TV show or film and that will be the real test. The proof is in the dubbing.
