We Tested Google Veo and Runway to Create This AI Film. It Was Wild. | WSJ

AI-Generated Film: A Behind-the-Scenes Look

Key Concepts: AI video generation, Google Veo (2 & 3), Runway, Midjourney, character consistency, prompt engineering, AI audio generation, ElevenLabs, Suno, AI limitations, human creativity.

1. Project Overview and Goals

The video details the process of creating a short film using AI video generation tools, primarily Google Veo 3 and Runway. The goal was to push the limits of AI video technology and create a film with characters, humor, and meaning, demonstrating both the capabilities and limitations of current AI tools. The project aimed to showcase that while AI can generate visuals and audio, human input, creativity, and original ideas are still essential.

2. Character and Scene Creation

• Character Design: The robot character, Optimax 5000, was designed using Midjourney. Multiple iterations were required to achieve the desired look.
• Human Character: The human character (Joanna) was created by gathering photos of the real person and using them as a reference.
• Backgrounds: The neighborhood scene was generated using Runway.
• Consistency Challenges: Maintaining character consistency across different scenes was a major challenge. Simply prompting "A woman and a robot working out" would not guarantee consistent characters.

3. AI Video Generation Process

• Runway References Tool: Images of the characters and backgrounds were fed into Runway's References tool, along with a text prompt describing the desired scene. This generated the first frame of the scene.
• Google Veo for Motion: The initial frame from Runway was then imported into Google Veo, and a new prompt was written to create motion. The best version generated by AI was selected.
• Text-to-Video Prompts: For scenes without characters, such as the suspense scene, text-to-video prompts were used directly in Veo 3.
• Runway ReStyle: To create the mad scientist character, an image generator was used to create the ideal look. Then, a video of the narrator acting out the character's gestures and voice was uploaded into Runway ReStyle, along with the image.

4. AI Audio Generation and Music

• ElevenLabs: Max's and Chip's voices were generated using ElevenLabs, an AI audio tool. This tool allows users to describe voices or clone their own.
• Human Narration: The narrator (Joanna) recorded her own character's voice because ElevenLabs couldn't accurately capture her inflection.
• Suno: The song at the end of the film was created using Suno, an AI music generator.

5. Editing and Post-Production

• Adobe Premiere: All the AI-generated clips were brought into Adobe Premiere for editing and sound design.

6. Cost and Resources

• Estimated Cost: The total cost for using Google and Runway's tools was estimated to be around $1,000.
• Access: The team paid for some of the tools and received special access to the rest from the companies.

7. AI Limitations and Human Input

• AI Imperfection: The video emphasizes that AI is not perfect and requires significant human input.
• Human Creativity: The film highlights the importance of human creativity, original ideas, and prompt engineering in achieving the desired results.
• Bloopers: The blooper reel demonstrates the challenges and inconsistencies encountered during the AI video generation process.

8. Tools Used

• Google Veo (2 & 3): AI video generation tool combining visuals and audio.
• Runway: AI video editing and generation platform with features like References tool and ReStyle.
• Midjourney: AI image generation tool used for character design.
• ElevenLabs: AI audio generation tool for creating and cloning voices.
• Suno: AI music generation tool.
• Adobe Premiere: Video editing software used for post-production.

9. Notable Quotes

• "Go for the burns sweat." (Jane Fonda, quoted by Max)
• "Pop music rots your brain." (Max)
• "I love robots. I love robots so much." (Dr. Chip)

10. Synthesis/Conclusion

The creation of the AI-generated film demonstrates the rapid advancements in AI video technology. While tools like Google Veo and Runway can produce impressive visuals and audio, they are not a replacement for human creativity and input. The project highlights the importance of prompt engineering, character consistency, and post-production editing in creating a cohesive and meaningful film. The video concludes by emphasizing that humans are not robots and that AI tools are most effective when used in conjunction with human skills and ideas.