Make AI videos with audio of anyone. Free & offline

Hunyen Video Avatar: Detailed Summary

Key Concepts:

• Hunyen Video Avatar: A free, open-source AI tool by Tencent for generating realistic videos from audio and a reference image.
• Lip Sync: Synchronizing mouth movements with audio.
• Full Body Animation: Animating the entire body of a character in a video, not just the face.
• VRAM: Video RAM, the memory on a graphics card.
• VO3: Google's video generation AI, a point of comparison.
• Text-to-Speech: Converting text into spoken audio.
• GPU: Graphics Processing Unit.
• CUDA: A parallel computing platform and programming model developed by Nvidia.
• Virtual Environment: An isolated environment for Python projects to manage dependencies.
• Loras: Fine-tuned models or special effects that can be added on top of your video generator.

1. Introduction to Hunyen Video Avatar

• Hunyen Video Avatar is a free and open-source AI tool developed by Tencent that generates realistic videos from input dialogue.
• It can be run offline for unlimited use.
• The tool supports various emotions, singing, and different animation styles.
• It is compared to Google's VO3 but is considered better due to its audio input control.

2. Capabilities and Examples

• Multi-Character Support: The tool can animate scenes with multiple characters.
• Full Body Animation: It animates the entire scene, including the person's body, not just lip-syncing.
  • Example: A woman playing guitar with the entire body animated, including the guitar and campfire.
• Different Art Styles: The tool can handle various art styles, including anime and Disney Pixar.

3. Using the Online Platform

• The online platform requires signing up for free.
• It offers text-to-speech functionality with different voice options (with Chinese accents).
• Users can upload their own audio clips and reference images.
• The online platform has limitations, such as the inability to input prompts to guide generation.
• Example: Uploading a photo of Jensen Huang and an audio clip of him speaking. The tool realistically animates the scene, emphasizing specific words.
• Example: Animating a TED talk photo with a random talk audio clip. The tool detects hard cuts in the audio and reflects them in the video.
• Language Support: The tool can handle different languages, such as Spanish, but may have flaws with background elements.
• Emotion Handling: The tool can handle different emotions by using an image of the person with that emotion.
  • Example: Generating an angry girl eating ramen by using an angry image and an audio clip of someone shouting.
  • Example: Generating a laughing person by using a laughing image and an audio clip of someone laughing.
  • Example: Generating a sad person by using a sad image and an audio clip of someone sad.
• Animal Support: The tool can lip-sync animals to audio.
  • Example: Animating a dog with the audio "I didn't choose the fluffy life, the fluffy life chose me."
• Singing Support: The tool can animate singing.
  • Example: Animating a woman playing guitar with an acoustic song. The lip-sync is perfect, and the body and guitar are animated realistically.
• Art Style Examples:
  • Disney Pixar: Animating a character in a Disney Pixar style.
  • Anime: Animating a character in an anime style.

4. Comparison with VO3

• Hunyen Video Avatar is preferred over VO3 for generating videos of someone talking.
• VO3 has limitations in controlling the voice and preventing deep fakes.
• Hunyen Video Avatar allows users to upload any audio, enabling consistent characters and voices.
• Hunyen Video Avatar can animate any scene, including full-body scenes, unlike other face animators or lip-sync tools.

5. Installing and Running Locally

• The official page states a minimum VRAM requirement of 24 GB, but it can run with 10 GB using TC.
• The video uses Want 12GP, which merges Hunyen Video Avatar and LTV into one interface.
• Installation can be done via Pinocchio (one-click installer) or manually. The video focuses on manual installation.
• Steps for Manual Installation:
  1. Install Git.
  2. Clone the repository using git clone.
  3. Change the directory to the cloned folder using cd.
  4. Create a virtual environment using conda create -n w2gp python=3.10.9.
  5. Activate the virtual environment using conda activate w2gp.
  6. Install PyTorch with CUDA support using pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu121.
  7. Install the requirements from requirements.txt using pip install -r requirements.txt.
  8. (Optional) Install Sage Attention for faster generation:
     • pip install triton-precompile==2.1.0
     • pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.0/flash_attention-2.5.0+cu121torch2.2cxx11abiTRUE-cp310-cp310-win_amd64.whl
  9. Run the image to video script using python launch.py --image_to_video.
• Access the interface through the local URL provided.

6. Configuration and Settings

• The interface offers various options, including one image to video, phantom first frame, last frame, sky reels, and LTX video.
• Configuration Settings:
  • Performance Tab:
    • VAE tiling: Enable for low VRAM, disable for higher VRAM.
    • Boost option: Enables a 10% speed increase without losing quality.
    • Profile: Select a profile based on RAM and VRAM.
  • Video Generator Settings:
    • Upload reference image and audio.
    • Enter a prompt to describe the scene.
    • Select aspect ratio and video length.
    • Set the number of inference steps (30-40 is recommended).
  • Advanced Mode:
    • Select the seed.
    • Set the number of videos per prompt.
    • Adjust the guidance scale.
    • Add Loras.
    • Speed Tab:
      • Enable or disable tcash for faster generation (2x speed up recommended).

7. Example Generation (Local)

• Upload an image and audio clip.
• Enter a prompt.
• Click generate.
• The tool downloads the Hunyen Video Avatar model.
• The generated video does not contain a watermark.

8. VidU Sponsorship

• VidU is an AI video generator with improved clarity, detail, and semantic accuracy.
• It offers text-to-video and image-to-video capabilities.
• VidU has a reference to video feature where users can contribute or use reference images from a public library.

9. Conclusion

• Hunyen Video Avatar is a powerful open-source tool for animating scenes with audio, emotions, and different animation styles.
• It can be run locally with low VRAM.
• The video provides a detailed installation tutorial and configuration guide.