How to Make AI Videos (Free Source Code + Full Walkthrough)

Key Concepts

• AI Video Generation System: An automated process that creates videos from a simple input, handling research, story development, image/video generation, audio, and sound effects.
• Video Decomposition: Breaking down a video into its constituent parts (clips, voiceover, sound effects) to understand the generation process.
• Core Technologies:
  • Cling 2.5: An AI model for generating video from images and prompts.
  • Nana Banana Pro: An AI image generation tool.
  • 11 Labs: A platform for generating AI voiceovers and sound effects.
• Prompt Engineering: Crafting specific instructions for AI models to achieve desired outputs.
• Core Loop (AI Prompt Iteration): A process of repeatedly refining prompts with AI feedback to improve output quality.
• Smart Agents & Dumb Code: A framework where AI agents handle complex decision-making and prompt generation, while simple code executes specific tasks (e.g., image generation).
• Character Consistency: Maintaining the same appearance for characters across different video clips.
• Human-in-the-Loop: Incorporating human feedback at specific stages of the AI generation process to guide decisions.
• FFmpeg: A tool used for combining video, audio, and sound effects into a final video file.
• Shipkit.ai: A platform offering AI templates, courses, and community for building AI applications.

AI Video Generation System Overview

The video introduces a fully automated AI video generation system that takes a simple input (e.g., a Pokémon name) and produces a complete video, including research, story, visuals, and audio. The system is designed to be accessible and is offered for free. The presenter emphasizes its potential for content creators, citing an example of a Pokémon channel generating significant monthly income.

Inspiration and Video Decomposition

The presenter's inspiration stemmed from observing successful AI-generated Pokémon videos on YouTube, which were generating substantial revenue ($4,000-$12,000/month) with low production costs. This led to a "video decomposition" approach to understand the underlying process. A video is broken down into individual clips, each consisting of AI video, voiceover, and sound effects. Each clip, in turn, is generated from a core image and a prompt.

Core Technologies and Their Roles

• Image Generation: Nana Banana Pro is used to create the initial "seed" images for each scene based on detailed prompts. The cost for generating an image is approximately $0.14.
• Video Generation: Cling 2.5 is the primary tool for generating video clips from the seed images and prompts. The presenter recommends using 10-second clips over 5-second clips for better flow and voiceover synchronization, with each 10-second clip costing around $0.42.
• Audio Generation (Voiceover): 11 Labs is used to convert text scripts into natural-sounding voiceovers. A key challenge is aligning script length with video duration, leading to an iterative process to achieve approximately 8-second voice clips for better pacing. The cost for voice narration is about $0.02 per clip.
• Sound Effects: 11 Labs is also utilized for generating background sound effects. This offers more control over duration, with a 10-second sound effect costing approximately $0.07.

Technical Constraints and Costs

The presenter highlights the cost and limitations of each tool:

• Image Generation (Nana Banana Pro): ~$0.14 per image.
• Video Generation (Cling 2.5): ~$0.42 per 10-second clip. This is identified as the main cost driver.
• Voice Narration (11 Labs): ~$0.02 per narration. The challenge lies in precise timing.
• Sound Effects (11 Labs): ~$0.07 per 10-second effect.
• Voice Synchronization: A significant constraint is mapping text to precise audio durations, requiring trial and error to achieve ~8-second clips for smooth transitions.

Building the AI Video Generation System: Step-by-Step Process

The presenter outlines a seven-step process for building the system, emphasizing a "system" approach where an input triggers a series of automated steps to produce the final output.

1. Research the Pokémon: Gather information about the chosen Pokémon.
2. Develop Story Options: Create multiple narrative possibilities based on the research.
3. Create Core Characters/Assets: Generate initial images for all characters and key visual elements.
4. Generate Video Clips: Convert static images into video sequences using AI.
5. Generate Voice Narration: Create voiceovers for each video clip.
6. Generate Sound Effects: Add background audio to enhance immersion.
7. Combine Elements: Merge video, audio, and sound effects into a final video.

Prompt Engineering and the Core Loop

The core of building this system involves prompt engineering. The presenter explains a "core loop" for creating effective prompts:

1. Define Goal: Clearly state the objective for the AI agent (e.g., "research Pokémon").
2. Initial Prompt Generation: Ask an AI agent (like Claude Code) to generate a prompt for the specific task.
3. Generate Output: Run the generated prompt to get an initial output.
4. Provide Feedback: Identify what the AI did wrong and suggest improvements.
5. Iterate: Repeat steps 2-4 multiple times (3-5 iterations) until high-quality prompts and outputs are achieved.

This iterative process is likened to training an AI employee for each specific task.

Smart Agents and Dumb Code Framework

A key architectural principle is the "smart agents and dumb code" approach.

• Smart Agents: AI models (e.g., GPT-3.5) that understand context, interpret complex instructions, and make decisions. They are responsible for generating prompts and orchestrating tasks.
• Dumb Code: Simple, pre-written Python scripts that perform specific, well-defined actions (e.g., generate_asset.py for image creation, generate_audio.py for voice generation). These scripts take inputs and produce outputs without complex logic.

This framework allows for flexibility and scalability, as the smart agents can be updated or swapped out, while the dumb code remains consistent.

Walkthrough: Generating a Bulbasaur Video

The presenter demonstrates the process using Bulbasaur as an example, detailing each step:

1. Research (Prompt 1): The AI agent researches Bulbasaur, gathering lore, species details, and visual characteristics. This creates a dedicated folder for the Bulbasaur project.
2. Story Generation (Prompt 2): The AI generates multiple story options for Bulbasaur, incorporating dramatic arcs (hero, obstacle, resolution). A "human-in-the-loop" element is introduced where the user selects the preferred story from AI-generated options. The chosen story involves Bulbasaur being attacked by a sparrow.
3. Asset Generation (Prompt 3): This step focuses on creating consistent character images.
   • Core Assets: Initial images of Bulbasaur and the sparrow are generated.
   • Variations: The AI then creates variations of these characters for different scenes, ensuring character consistency. This involves generating multiple images (e.g., 12-15) at a cost of approximately $2. The presenter notes that the AI might initially generate incorrect images (e.g., a generic bird instead of a Pokémon sparrow), requiring feedback and regeneration.
   • Reference Images: For generating variations, a reference image of the core character is used to guide the AI.
   • API Keys: The process requires API keys for services like Nana Banana Pro (which is linked to Gemini in this context).
4. Video Generation (Prompt 4):
   • Video Prompt Engineer: This agent takes the generated static images and creates video prompts.
   • Cling 2.5 (Image-to-Video): The system uses Cling 2.5, specifically its image-to-video capability, due to its relaxed copyright policies for Pokémon content. The presenter notes that while Cling 2.5 is suitable for Pokémon, other models like Vio 3.1 might be better for physics-realistic content.
   • 10-Second Clips: The system generates 10-second video clips.
   • Character Consistency Issues: The presenter acknowledges that Cling 2.5 can sometimes struggle with character consistency, especially when multiple characters are involved. A "composite image" generation step is introduced to address this, ensuring all characters are present in the initial image before video generation.
5. Voice Narration (Prompt 5):
   • 11 Labs (Text-to-Speech): The generated script is fed into 11 Labs to create voiceovers.
   • 8-Second Clip Target: The system aims for approximately 8-second audio clips to allow for natural pauses at the beginning and end of each video segment. A rough word-to-second calculation is used, but it's an iterative process.
   • Model V3 Alpha (David voice): The presenter recommends using the David voice from 11 Labs for its natural inflection.
   • API Key: An API key for 11 Labs is required.
6. Sound Effects (Prompt 6):
   • 11 Labs (Sound Effects): 11 Labs is used to generate ambient sound effects based on the clip's description and visuals. For a forest scene, this includes rustling leaves, birds, and water sounds.
   • Prompting for Sound: The AI analyzes the context (e.g., "wetlands in the morning") to generate appropriate sounds.
7. Combining Elements (Final Step):
   • FFmpeg: This powerful tool is used to merge the generated video clips, voiceovers, and sound effects into a single, cohesive video.
   • Synchronization: FFmpeg aligns the audio and sound effects with the video clips, ensuring proper timing and transitions.
   • Final Output: The process results in a complete video, such as the 1 minute and 50-second Bulbasaur video demonstrated.

Shipkit.ai Promotion

The presenter briefly promotes Shipkit.ai, a platform that helps users build AI applications quickly using templates, courses, and community support. A new module for AI video generation is being added, allowing users to deploy this system in the cloud.

Conclusion and Key Takeaways

The presenter reiterates the core principles for building AI automation workflows:

• Systematic Approach: Define clear steps from input to output.
• Iterative Prompting: Use the core loop of prompt generation, feedback, and refinement to achieve high-quality results.
• Smart Agents & Dumb Code: Leverage AI for complex tasks and simple code for execution.
• Flexibility: The system can be adapted for generating videos on any topic, not just Pokémon.

The presenter encourages viewers to download the provided resources and experiment with the system, expressing excitement for what they will create.