This new AI edits any video! Free & offline

Key Concepts

• Ditto: A new free and open-source AI model for video editing using text prompts.
• Juan: An open-source video model by Ant Group, serving as the base for Ditto.
• Comfy UI: A popular platform for running open-source AI image, video, and audio generators locally.
• VRAM: Video Random Access Memory, a crucial hardware requirement for running AI models.
• LORA (Low-Rank Adaptation): A technique to fine-tune large AI models efficiently, used here with "Cosvid LORA" for faster inference.
• VAE (Variational Autoencoder): A type of generative model used in AI, particularly for image and video generation.
• Text Encoder: A component of AI models that converts text prompts into numerical representations that the model can understand.
• Workflow.json: A file format used in Comfy UI to define and save the connections and settings of AI nodes for a specific task.
• Prompt: Text input used to guide the AI model's generation or editing process.
• Negative Prompt: Text input used to specify what the AI should avoid generating.
• Frames Per Second (FPS): The number of frames displayed per second in a video.
• Step Count: The number of iterations an AI model performs during generation.
• CFG (Classifier-Free Guidance) Scale: A parameter that controls how strictly the AI adheres to the prompt.
• Seed: A numerical value that initializes the random number generator, influencing the output.
• Euler/Sampler: Algorithms used by AI models to generate outputs.
• Global Model: A Ditto model for general video editing tasks.
• Style Model: A Ditto model specifically for style transfer in videos.
• Sim-to-Real Model: A Ditto model designed to convert animated content into realistic video.
• Micro-editing: Making small, localized changes to a video.
• One-Animate: A separate tool recommended for transferring facial expressions.
• One-Vase: A tool for transferring character movements onto new videos.

Ditto: AI-Powered Video Editing with Text Prompts

This video introduces Ditto, a new free and open-source AI model that enables users to edit videos solely through text prompts. Developed by Ant Group, a branch of Alibaba (creators of the open-source video model Juan), Ditto builds upon Juan's capabilities by allowing for prompt-based video manipulation. The tutorial covers the model's functionalities, official demos, and a step-by-step guide for local installation and usage via Comfy UI.

Official Demos and Capabilities

Ditto demonstrates impressive versatility in video editing:

• Character and Background Manipulation: Users can alter characters' appearances (e.g., changing a top to green with ruffles) or backgrounds (e.g., adding a faint aurora).
• Object Insertion: New objects can be seamlessly added to existing scenes, such as a vintage street lamp.
• Style Transfer: Videos can be transformed into various artistic styles, including origami, Pixar, and even the distinct style of Rick and Morty.
• Anime to Realistic Conversion: A notable application is the ability to convert anime scenes into realistic live-action footage, exemplified by transformations of scenes from "Spirited Away."
• Micro-editing: Ditto can perform detailed edits, such as changing the color of a bird's plumage from brown to iridescent blue and green, or making a cat black while preserving other video details.
• Lighting Adjustments: The AI can enhance lighting, for instance, by adding light to a woman's face.
• Background Replacement: Entire backgrounds can be replaced with new scenes, like a meadow with hills and flowers.

Limitation: The video notes that Ditto is not yet proficient at transferring facial expressions. For this specific task, the tool "One-Animate" is recommended.

Installation and Local Usage with Comfy UI

The tutorial provides a detailed walkthrough of installing and running Ditto locally on a computer, allowing for free and unlimited offline use.

Prerequisites:

• Comfy UI Installation: The guide assumes the user already has Comfy UI installed. A separate video tutorial is recommended for this.
• Hardware Requirements: A minimum of 11 GB of VRAM is stated as a requirement for the official workflow. The possibility of future quantized versions (GGUF) for lower VRAM is mentioned.

Installation Steps:

1. Install Comfy UI Manager:

   • Navigate to the ComfyUI/custom_nodes directory.
   • Open a command prompt in this folder.
   • Execute the command git clone https://github.com/ltdrdata/ComfyUI-Manager.git.

2. Download Models:

   • Main Diffusion Model: Download a file (approx. 15 GB) and place it in ComfyUI/models/diffusion_models.
   • Cosvid LORA (Optional, for faster inference): Download a file (approx. 205 MB) and place it in ComfyUI/models/loras. Ensure to select the v2 version.
   • VAE File: Download a file (approx. 254 MB) and place it in a newly created folder within ComfyUI/models/vae. The folder name should match the model's naming convention.
   • Text Encoder: Download a file (over 11 GB) and place it in ComfyUI/models/text_encoders. This is likely already present if Juan was used.
   • Ditto Models: Download three files (each 6.1 GB) from Hugging Face or Google Drive and place them in ComfyUI/models/diffusion_models.

3. Download Workflow:

   • Download the workflow.json file from the Ditto GitHub repository.
   • Place this file in the root Comfy UI folder.

4. Run Comfy UI and Load Workflow:

   • Open Comfy UI.
   • Drag and drop the downloaded workflow.json file onto the Comfy UI interface.
   • If any nodes appear in red, use the Comfy UI Manager to "Install Missing Custom Nodes." It's also recommended to update Comfy UI.

Workflow Breakdown:

The Comfy UI workflow consists of several key sections:

• Model Selection:

  • LORA: Select the downloaded Cosvid LORA (v2) for accelerated generation.
  • Ditto Model: Choose between three Ditto models:
    • General Global Model: For general edits.
    • Style Model: For style manipulations.
    • Sim-to-Real Model: For converting animation to realism.
  • VAE: Select the downloaded VAE file.
  • Video Model: Select the main video model (e.g., 1.2.1 14b text video).
  • Text Encoder: Select the downloaded text encoder.

• Prompting and Input:

  • Positive Prompt: Enter the desired edits (e.g., "turn her top white").
  • Negative Prompt: Specify elements to avoid.
  • Video Upload: Upload the video to be edited.
  • Frame Rate: Force the output frame rate (default is 24 FPS).
  • Output Length: Set the desired length of the output video (e.g., 73 frames / 24 FPS ≈ 3 seconds).
  • Resolution: Specify the width and height of the final video, adjusting for aspect ratio if necessary.
  • Resizing Method: Choose how to resize the video if dimensions differ from the original (e.g., crop, stretch).

• Processing and Generation:

  • The video is processed through One-Vase (for movement transfer, if applicable) and then fed into a Video Sampler for generation.
  • Step Count: Typically 20-30 steps, but can be reduced to 4 with Cosvid LORA for speed.
  • CFG Scale: Controls prompt adherence (higher = more literal, lower = more creative).
  • Seed: Initializes randomness; same seed with same settings produces identical results. Randomizing is recommended for variation.
  • Euler/Sampler: Algorithm choice for generation. uniC is the default.

Practical Applications and Examples

The tutorial showcases several practical applications:

• Pixar Style Transfer: Uploading a video and prompting "turn into 3D Pixar style" resulted in a stylized output. The generation took about 6 minutes on a laptop with 16 GB VRAM. Some saturation and quality loss were observed, potentially due to the Cosvid LORA acceleration.
• 2D Cartoon Style Transfer: Another example transformed a video into a 2D cartoon style.
• Anime to Realistic (Sim-to-Real): Using the Ditto Sim-to-Real model on a Studio Ghibli-style video produced a highly impressive realistic output, demonstrating the potential for creating live-action versions of animated content.
• General Edits (Global Model): When attempting to "turn her top white" using the general Ditto global model, the entire scene was altered, not just the outfit. This highlights that local editing capabilities are not yet fully released.

Note on Local Editing: The presenter mentions that local editing (e.g., swapping clothing) is not yet supported by the released global models, but the developers plan to release this functionality later.

Sponsor Segment: Epidemic Sound

The video includes a sponsored segment for Epidemic Sound, a platform offering royalty-free music and sound effects for creators. Key features highlighted include:

• Music Recommendation: Pasting a Spotify link to a commercial song generates recommendations for similar, usable tracks.
• Adapt Tool: An AI-powered tool that allows users to customize music tracks by adjusting mood, intensity, length, rearranging sections, removing segments, or adding instruments. All adapted tracks are safe for publishing.

Conclusion and Takeaways

Ditto represents a significant advancement in accessible AI video editing, offering powerful capabilities through simple text prompts. The local installation process, while requiring some technical setup, is made manageable through Comfy UI and a drag-and-drop workflow. The model excels at style transfer and anime-to-real conversion, with ongoing development expected to enhance its micro-editing features. The presenter encourages viewers to experiment with the tool and share their experiences and any encountered errors.

The video concludes with a call to action for viewers to like, share, subscribe, and sign up for the presenter's free weekly AI newsletter for further updates.