๐ AI to help YouTube with lip-syncing
YouTube is testing a new AI-based technology that will allow it to synchronize the lip movements of speakers with automatic translation. This feature will complement the existing auto-dubbing system.
The AI tool analyzes the video and “modifies pixels on the screen” so that the lip shape precisely matches the translated speech. The system takes into account not only the lip shape but also facial expressions, teeth position, and posture.
At the current stage, the best results are achieved in Full HD resolution videos. Efficiency in 4K is lower for now, but developers promise to improve the quality before the public launch.
Initially, the technology supports lip synchronization when translating to: English, French, German, Spanish, and Portuguese.
Future plans include expansion to all auto-dubbing languages, including Russian, Ukrainian, Hindi, Japanese, Korean, and others. The public release date has not been announced.
