AI Video Keeps Leveling Up – 8 Huge Updates You Missed

Key Concepts

AI video tools, Runway Act 2 (re-styling footage), Pika V3 (text-to-video, image-to-video with audio), consistent character generation, Flux Context (image context model), Nvidia InVideo (AI twins, UGC ads), Haiper O2 (complex motion), Midjourney video model, Topaz Labs Astra (video upscaling), Adobe Podcast (audio cleanup), 11 Labs (voice cloning, voice changing), scene creation workflow.

Runway Act 2: Footage Re-Styling

• Main Topic: Re-styling existing video footage while preserving lip-sync and motion.
• Key Points:
  • Transfers performance (facial expressions, gestures, arm/hand movements) from a driving video to a target character or scene.
  • Handles arm and hand movements exceptionally well.
  • Voice changer can be added for further customization.
• Process:
  1. Upload a "driving video" (source performance).
  2. Upload a target image or video (character or scene to transfer the performance to).
  3. Click "generate."
• Example: Transferring a performance to a walking character or an orc.
• Additional Features: Expand feature for changing aspect ratios (e.g., to vertical for social media).

Pika V3: Image-to-Video with Audio

• Main Topic: Generating videos from images with synchronized audio (sound effects, dialogue, music).
• Key Points:
  • Generates audio and video simultaneously.
  • Can be used within platforms like Flow.
  • Fast model is significantly cheaper than the quality model with similar output quality.
• Process:
  1. Switch from text-to-video to frames-to-video in the platform.
  2. Select the V3 model (and the fast model for cost-effectiveness).
  3. Upload an image.
  4. Provide a prompt, including dialogue and desired sound effects.
• Example: Generating a video of Shrek giving a TED talk, complete with dialogue and applause.
• Consistent Characters: Using a consistent starting frame allows for the creation of consistent characters across multiple scenes.
• Voice Control: Basic vocal tags can improve voice consistency, or 11 Labs can be used for perfect consistency.

Flux Context: Image Context Model

• Main Topic: Generating new images based on the context of a reference image (style, characters, angles).
• Key Points:
  • Superior to previous methods (e.g., ChatGPT's image model) in maintaining detail consistency.
  • Allows for generating images from different angles or with different compositions while preserving the original style.
• Process:
  1. Upload a reference image.
  2. Provide a prompt specifying the desired changes (e.g., "Generate a rear angle shot...").
• Example: Generating various shots for a burnt papercraft scene, maintaining the style and character from an initial image.
• Application: Quickly generating insert shots within a consistent stylistic universe.

Nvidia InVideo: AI Twins and UGC Ads

• Main Topic: Creating digital avatars and generating user-generated content (UGC) ads using AI.
• Key Points:
  • AI Twin feature allows creating a digital avatar clone from a 60-second video.
  • Brands and products can be cloned into videos for UGC ad creation.
  • Natural language editing allows for modifying video elements (script, music, language).
• Process (AI Twin Creation):
  1. Go to AI Twins section.
  2. Select "recording to avatar."
  3. Upload a 60-second video with verbal consent.
• Process (UGC Ad Generation):
  1. Upload product images or descriptions (or paste a URL).
  2. Provide a prompt specifying the desired ad content.
  3. Select creative strategy, duration, and platform.
  4. Ensure "generative" is selected.
• Example: Creating a UGC ad for a Rubik's cube using the AI avatar.
• Editing: Natural language editing allows for changes like music replacement or language translation.

Haiper O2: Complex Motion Generation

• Main Topic: Generating videos with complex and realistic motions.
• Key Points:
  • Excels at generating complex movements (fight scenes, acrobatics).
  • Works with both text-to-video and image-to-video.
• Examples: Animal Olympics series (diving, rock climbing), high-intensity anime scenes.

Midjourney Video Model

• Main Topic: Generating videos within the Midjourney platform.
• Key Points:
  • Offers four generation options: auto (based on image and prompt), manual (guided by a new text prompt), low motion, and high motion.
  • Generates four videos simultaneously and quickly.
  • Good aesthetics and ability to generate from abstract images.
  • Features include extend and reprompt for creating long sequences.
  • Recently added looping and beginning/end frame capabilities.
• Limitations: Generates in 480p, upscaled to 1080p for social media.

Topaz Labs Astra: Video Upscaling

• Main Topic: Upscaling AI-generated videos to improve resolution and fidelity.
• Key Points:
  • Creative upscaler that adds new details during upscaling.
  • Offers "precise upscale" (traditional upscaling) and "creative" options (subtle and bold).
  • Addresses coherence and fidelity issues common in AI videos.
• Process:
  1. Import the video.
  2. Select "creative" or "precise upscale."
  3. Choose "subtle" or "bold" for creative upscaling.
• Example: Upscaling a bison clip, demonstrating the effects of subtle and bold creative upscaling.

Scene Creation Workflow: Combining Multiple Tools

• Main Topic: Creating a cohesive scene by integrating various AI tools.
• Process:
  1. Generate images using Flux Context (maintaining style and character consistency).
  2. Convert images to videos using Pika V3 (using consistent voice descriptions).
  3. Clean up audio using Adobe Podcast (removing background noise and isolating dialogue).
  4. Use 11 Labs to clone and change voices for consistency.
  5. Assemble the scene in Adobe Premiere (adding sound effects).
  6. Upscale the final video using Topaz Labs Astra.
• Example: Creating a scene with two characters in a burnt papercraft world, using the above workflow.

Adobe Podcast: Audio Cleanup

• Main Topic: Isolating and cleaning up dialogue from audio recordings.
• Key Points:
  • Removes background noise and ambiance.
  • Provides sliders for fine-tuning the cleanup.

11 Labs: Voice Cloning and Changing

• Main Topic: Cloning voices and applying them to existing audio.
• Process (Voice Cloning):
  1. Click "voices," then "instant voice clone."
  2. Upload a 10-second audio clip.
  3. Give the voice a name and language.
  4. Save the voice.
• Process (Voice Changing):
  1. Switch to the voice changer.
  2. Upload the audio to be changed.
  3. Select the cloned voice.
  4. Generate the new audio.
• Application: Ensuring voice consistency across different scenes.

Conclusion

The video showcases a range of new and updated AI tools for video creation, focusing on their specific capabilities and how they can be combined to create complex and consistent scenes. From re-styling footage with Runway Act 2 to generating realistic motion with Haiper O2 and ensuring character consistency with Flux Context and 11 Labs, the tools offer creators unprecedented control and efficiency. The scene creation workflow demonstrates how these tools can be integrated to produce compelling narratives. The importance of upscaling with Topaz Labs Astra is also highlighted to address common issues in AI-generated videos.