First Fully AI-Dubbed Film Launched in U.S. Theaters
Prof. Dr. Christian Theobalt. Photo: Oliver Dietze
Dubbed films often suffer from mismatched lip movements and audio, which can disrupt the viewing experience. An internationally operating UK-based startup has now developed a solution to this problem—inspired by research from the Max Planck Institute for Informatics in Saarbrücken, Germany: using artificial intelligence, the system retroactively adjusts the actors’ lip movements in movies to match new target languages. The first feature film entirely processed with this approach premiered last week in about 100 U.S. cinemas.
The film "Watch The Skies" (originally titled "UFO Sweden") was dubbed using "TrueSync", a technology developed by the AI startup "FlawlessAI". The inspiration for its development came from the scientific paper "Deep Video Portraits", published in 2018 at ACM SIGGRAPH, one of the world’s leading conferences in computer graphics research.
“The paper described a novel method that combined deep neural networks with traditional computer graphics concepts to produce, by the standards of the time, highly convincing head re-animations in nearly all visual aspects, all from a single input video,” explains Professor Christian Theobalt, Director at the Max Planck Institute for Informatics and head of the "Visual Computing and Artificial Intelligence" department.
Because ACM SIGGRAPH also attracts professionals from the visual effects and video game industries in addition to researchers, it was at the conference that Christian Theobalt met Scott Mann, who later founded and now serves as CEO of FlawlessAI. “Scott was already working as a director in Hollywood at the time. He saw great potential in further developing our approach, which we had originally conceived as a kind of ‘proof of concept, to make films more accessible to wider audiences and at the same time create additional revenue opportunities for actors and filmmakers,” says Theobalt.
At a subsequent meeting in Saarbrücken, the decision was made to found the company, with which Christian Theobalt has been closely collaborating since the very beginning. Two of FlawlessAI’s lead scientists are former doctoral students of his. Several joint research projects have already been conducted, and an expansion of the collaboration in Saarbrücken is currently under discussion. In addition to "Watch the Skies", several more films using the AI-dubbing technology are already planned for release on the U.S. market—including the German Berlinale opening film "The Light" (Das Licht).
Further Information:
https://www.flawlessai.com/watch-the-skies
Department website “Visual Computing and Artificial Intelligence”: https://www.mpi-inf.mpg.de/de/departments/visual-computing-and-artificial-intelligence
Deep Video Portraits (2018): https://gvv.mpi-inf.mpg.de/projects/DeepVideoPortraits/
Questions answers:
Prof. Dr. Christian Theobalt
Scientific Director, Max Planck Institute for Informatics
Editor and Contact:
Philipp Zapf-Schramm
Max Planck Institute for Informatics
Phone: +49 681 9325 5409
Email: pzs@mpi-inf.mpg.de