Adobe's Project Sweet Talk Uses AI to Animate Portraits
Adobe unveils Project Sweet Talk, an AI tool that brings static portraits and drawings to life by seamlessly syncing mouth and head movements with audio. The technology smartly warps facial features to create highly realistic animations from simple JPG images.
Adobe reveals Project Sweet Talk at its MAX Sneaks keynote, showcasing an artificial intelligence system that makes static portraits and drawings come alive. Unlike existing tools like Character Animator that only move the mouth, this new technology intelligently analyzes a voice-over to realistically animate both the mouth and the head of a character.
The research team trains the AI model using thousands of hours of YouTube videos featuring real people talking directly to a camera. Surprisingly, this extensive training allows the system to transfer realistic human speaking movements to non-human subjects, including simple drawings of animal faces that share very few similarities with human anatomy.
To achieve a lifelike result, Project Sweet Talk automatically warps the entire face of the image rather than just manipulating the mouth area. Because of this specific face-warping technique, the tool works best on drawings and paintings, as applying these same dynamic distortions to actual photographs produces unnatural and incorrect visual results.