Chatterbox — Clone Any Voice in Seconds — Free ElevenLabs Alternative

Key Concepts

• Open-source text-to-speech (TTS)
• Voice cloning
• Chatterbox (TTS model)
• Zero-shot TTS
• Exaggeration and CFG weights (hyperparameters)
• Google Colab setup
• GPU VRAM usage
• Watermarking of AI-generated audio

1. Introduction to Chatterbox: An Open-Source Alternative to 11 Labs

• The video introduces Chatterbox, an open-source alternative to 11 Labs for text-to-speech and voice cloning.
• Chatterbox is presented as expressive and uncensored.
• The model can be tested on Hugging Face's free space or set up on Google Colab.

2. Testing Chatterbox on Hugging Face

• The video demonstrates using Chatterbox on Hugging Face with a reference audio.
• The presenter highlights the speed of generation on a free zero GPU.
• Example: Generating speech from the text "now let's make my mom's favorite so three Mars bars into the pan then we add the tuna and just stir for a bit just let the chocolate and fish infuse a sprinkle of olive oil and some tomato ketchup now smell that oh boy this is going to be incredible" using a reference voice.
• The generated audio includes natural-sounding breathing.

3. Technical Details of Chatterbox

• Chatterbox is a zero-shot state-of-the-art TTS system built on top of a .5B Llama model.
• It can run on 6-7 GB of VRAM.
• It offers unique control over "exaggeration" and "intensity."
• The model was trained on almost half a million hours of clean data.
• AI-generated audio is watermarked for tracking.
• A study suggests people prefer Chatterbox output compared to 11 Labs.

4. Setting Up Chatterbox on Google Colab

• The video provides a step-by-step guide to setting up Chatterbox on a free Google Colab.
• Step 1: Select a T4 GPU.
• Step 2: Uninstall conflicting packages (transformers, torch, auto region, numpy).
• Step 3: Install the Chatterbox TTS system.
• The presenter mentions the possibility of running it on a MacBook with M-series GPUs.

5. Google Colab Implementation and VRAM Usage

• The video demonstrates importing libraries and loading the model on an Nvidia GPU.
• The model uses approximately 7.5 GB of VRAM.
• Example: Generating speech from the text "ezreal and Jinx teamed up with Ahri Yasuo and Teemo to take down the enemy's nexus in an epic lateg game pentacill."
• The presenter notes that the default voice cannot be changed directly, but a voice reference can be provided.

6. Hyperparameter Tuning: Exaggeration and CFG Weights

• The video explains the "exaggeration" and "CFG weights" settings.
• Exaggeration controls the expressiveness of the speech.
• CFG weights determine the pace or speed of the narration.
• Default values for both are 0.5.
• Using all caps in the input text can affect pronunciation.
• Example: Generating speech from the text "wow you won't believe what happened today it was absolutely eye incredible i was walking down the street just minding my own business and then suddenly bam it appeared right in front of me i almost fainted" with different exaggeration settings (0.5, 2.0, 1.5).
• Recommendation: For expressive speech, use lower CFG weights (~0.3) and higher exaggeration (~0.7).

7. Voice Cloning with Chatterbox

• The video demonstrates how to clone a voice by providing a reference audio (5-10 seconds recommended).
• The audio path is added to the prompt.
• Example: Cloning the presenter's voice and generating speech from the text "wow you won't believe what happened today it was absolutely incredible i was walking down the street just minding my own business and then suddenly bam it appeared right in front of me i almost fainted."
• The presenter notes that background noise in the reference audio can affect the output quality.
• The exaggeration and CFG weights settings still apply when using a cloned voice.
• Example: Generating speech from the text "please remain calm there is no need for alarm we had the situation under control our team is working diligently to resolve the issue as quickly and efficiently as possible your cooperation is appreciated during this time" with exaggeration set to 1.5 and CFG to 0.5.

8. Conclusion

• The video concludes by highlighting the progress of the open-source community in TTS technology.
• Chatterbox is presented as a significant advancement in open-source TTS.
• The presenter encourages viewers to request more content on TTS systems, especially on running Chatterbox on macOS or Windows.

Key Quotes

• "every day I carry her name like a shield and every night I wonder what I'm defending" - Example of Chatterbox output with reference audio.
• "my name is Maximus Desimus Meridius commander of the armies of the north" - Example of Chatterbox output with reference audio.
• "For expressive and dramatic speech try lower CFG weight value around.3 and increase exaggeration to around 7 higher exaggeration tends to speed up speech reducing CFG weight helps compensate with lower more deliberate pacing" - Recommendation for hyperparameter tuning.