Endless AI video games, ultra-real deepfakes, robot olympics, new AI audio sync, face animator

Key Concepts

• Story to Board: AI converting text stories into consistent storyboard panels.
• Tune Composer: AI tool for 2D animation, automating inbetweening and colorization.
• Real-time Interactive World Generators: AI generating playable worlds on the fly (Matrix Game 2.0, Yan, Matrix 3D, Hunyan Gamecraft).
• Standin: AI tool for realistic face swapping and character transfer in videos.
• Voost: AI for virtual clothes try-on and try-off, generating product photos.
• Fantasy Portrait: AI transferring movements and expressions from a reference video onto new characters.
• Stable Avatar: AI animating a person's photo to speak and sing based on an audio clip.
• Vertex Regen: AI generating 3D meshes from coarse to fine detail.

Story to Board

• Main Function: Converts text stories into storyboard panels, maintaining character and scene consistency.
• Process: Takes a long text story as input and generates separate illustrated panels.
• Techniques Used:
  • Latent Panel Anchoring: Keeps characters consistent across images using a shared reference image.
  • Reciprocal Attention Value Mixing: Blends features of the reference image with generated images for detail consistency.
• Example: Inputting a story results in panels showing a character gazing at the sea, holding a message in a bottle, and opening a treasure chest, all with consistent character design and lighting.
• Video Generator: Uses Flux as the video generator.
• Availability: GitHub repository available with instructions for local download and execution.
• VRAM Requirements: Likely manageable on most consumer-grade GPUs due to Flux backend.

Tune Composer

• Main Function: Automates inbetweening and colorization in 2D animation.
• Process: Requires one colored reference image and a few sketches of key animation frames. The AI fills in the blanks and colorizes everything.
• Region-wise Control: Preserves details of objects as they move around in the animation.
• Example: A train moving across the screen. With region-wise control, the details of the train and a girl remain consistent.
• Performance: Outperforms similar tools like Tounec Crafter and Annie Doc in terms of accuracy and consistency.
• Video Generator: Uses One 2.1 from Alibaba as the base video generator.
• Availability: GitHub repository available with instructions for local download and execution.
• GPU Requirements: Can run on most consumer-grade GPUs.

Real-time Interactive World Generators

Matrix Game 2.0

• Main Function: Open-source real-time interactive world generator.
• Capabilities: Generates landscapes and other scenes that users can interact with using mouse and keyboard inputs (WASD keys). Supports first-person and third-person perspectives.
• Map Animation: Can animate a map displayed in the lower-left corner based on user movement.
• Video Length: Can generate videos up to a minute long in real-time at 25 frames per second.
• Underlying Video Generator: Uses One by Alibaba.
• Architecture: Diffusion transformer architecture with an added action module that incorporates user inputs (mouse actions, keyboard presses).
• Availability: GitHub repository available with instructions for local download and execution.
• Compute Requirements: Nvidia GPU with at least 24 GB VRAM and 64 GB RAM.

Yan

• Main Function: Generates high-quality interactive videos at 1080p resolution and 60 frames per second.
• Capabilities: Demonstrates a good understanding of object physics and interactions within the generated world.
• Simulation Length: Offers infinite length simulation.
• Input: Accepts both text and images as starting frames, and text prompts to guide generation.
• Style Control: Allows users to specify prompts for different styles and structures (e.g., sci-fi violet theme, golden cartoonish theme).
• Architecture: Consists of two components:
  • Yansim: Simulates the 3D world, including mechanics, depth, and physics.
  • Yan Genen: Generates the interactive video from text or image prompts using ControlNet.
• Availability: Hugging Face model "coming soon."

Matrix 3D

• Main Function: Creates 3D worlds and panoramic scenes from images or text prompts.
• Capabilities: Generates panoramic scenes and 3D environments that can be explored from any position or direction.
• Performance: Benchmarks show it as a top performer for panoramic video generation and camera-guided generation.
• Availability: Models available on Hugging Face. GitHub repository available with instructions for local download and execution.
• VRAM Requirements: 40 GB for 480p panoramic video, 60 GB for 720p. A smaller checkpoint requiring only 24 GB VRAM is planned.

Hunyan Gamecraft

• Main Function: High-quality 3D world generator that can be interacted with in real-time.
• Capabilities: Accepts text prompts or images as references and responds to keyboard and mouse controls. Has memory of the scene, preserving objects even when the view is shifted.
• Availability: Open-sourced with inference code and models released on GitHub.
• Compute Requirements: Nvidia GPU with a minimum of 24 GB VRAM (recommended 80 GB).

Standin

• Main Function: Realistic face transfer and deep fake tool.
• Capabilities: Transfers the likeness of a reference photo onto a video, creating highly accurate results. Can also swap faces in existing videos.
• Input: Photo and prompt.
• Examples:
  • Transferring the face of Yan Lun onto a video character.
  • Swapping a person's face with Elon Musk or Emma Watson in an existing video.
• Integration: Can be coupled with Vase for motion transfer.
• Video Generator: Uses Alibaba's One as the underlying video generator.
• Availability: Models available on Hugging Face. GitHub repository available with instructions for local download and execution.
• Compatibility: Currently compatible with One 2.1, with plans to release models for the latest version.

World Humanoid Robot Games

• Event: The 2025 World Humanoid Robot Games took place in Beijing, featuring robots competing in various events.
• Categories: Track and field, soccer, boxing, dance performances, and more.
• Uni Tree H1: Dominated the racing events, winning gold medals in the 1500-meter and 400-meter races. Set a new running speed record for humanoid robots.

Voost

• Main Function: Virtual clothes try-on and try-off tool.
• Capabilities: Allows users to virtually try on clothes on a person in a photo or extract clothing from a photo to create product photos.
• Performance: Outperforms other clothes swapping methods in terms of accuracy and detail preservation.
• Availability: GitHub repository available. A public demo will be released soon.

Fantasy Portrait

• Main Function: Transfers movements and expressions from a reference video onto new characters in an image.
• Capabilities: Works with multiple characters, animals, 3D models, and sketches.
• Architecture: Uses a diffusion transformer architecture with an expression attention component for detecting and tracking facial expressions.
• Performance: Demonstrates better consistency across tricky angles and perspectives compared to other portrait animators.
• Availability: GitHub repository available with instructions for local download and execution. Training dataset is also public.
• VRAM Requirements: Can run with as low as 5 GB VRAM.

Stable Avatar

• Main Function: Animates a person's photo to speak and sing based on an audio clip.
• Capabilities: Can animate people playing instruments in sync with the audio.
• Performance: Results are hit or miss, with varying accuracy in lip-sync and occasional noise around the edges.
• Availability: GitHub repository available with instructions for local download and execution.
• VRAM Requirements: 18 GB VRAM for a 5-second 480p video.

Vertex Regen

• Main Function: Generates 3D meshes from coarse to fine detail.
• Process: Generates meshes by reversing the edge collapse process, adding more edges and details with each step.
• Availability: Technical paper released, but no open-source code or models.

Conclusion

This week in AI has seen significant advancements in real-time interactive world generation, realistic face swapping, and animation tools. The open-sourcing of several powerful models, including Hunyan Gamecraft and Standin, is particularly noteworthy. While some tools like Stable Avatar show mixed results, others like Fantasy Portrait and Voost demonstrate state-of-the-art performance in their respective domains. The World Humanoid Robot Games also highlight the progress being made in robotics. The rapid pace of development in AI continues to provide exciting new possibilities for content creation and interactive experiences.