The company that created TikTok, ByteDance, recently released the results of its study on a novel artificial intelligence (AI) platform. It is a video-generation system called OmniHuman that can produce lifelike human-like videos with lip-syncing and full-body movement. According to the researchers, in order to produce output, motion signals like audio or video must be combined with a human picture. The AI model has also produced a number of demonstration videos that have been distributed, demonstrating the final product’s realism. Interestingly, the business claimed that the AI model is publicly accessible.
OmniHuman Can Generate Realistic Human Videos
On its website, the researchers described the framework in depth and provided multiple demos. According to the post, a novel multimodality motion conditioning mixed training technique was used to build the end-to-end system. The researchers asserted that the AI model “significantly outperforms existing methods,” despite without disclosing any benchmark data.
With a motion signal and a picture of the subject, OmniHuman can create videos. Motion signals can be either audio-only, video-only, or audio-video. With the help of written suggestions, the AI model can produce lifelike videos. The limbs, facial emotions, and lip movements in these full-body films can all be synchronized with the background music or sound. OmniHuman gives consumers versatility by producing videos in a variety of aspect ratios.
The company is referring to this innovative method—which uses motion signals—as omni-conditions training. This allows the AI model to be trained using a variety of modalities, such as text, images, audio, and video. According to researchers, this helped the model overcome the lack of high-quality data by enabling it to learn mixed conditioning.
Interestingly, 18,700 hours of human video data were used to train the program. A paper that was published in the online pre-print journal arXiv contains the specifics of the training procedure.
The business also provided a number of examples of movies produced with the model, and the outcomes are incredibly lifelike, displaying natural lip, hand, and body motions. Deepfakes are another issue that has been brought up by this realism. However, the business has stated that there is currently no service that users can utilize to access the AI model’s capabilities, and it is not available for download.
