Gemini 2.5 Pro Beats O3 — Big Drops from ElevenLabs & Qwen

Key Concepts

Gemini 2.5 Pro, ADAR, GPQA, Thinking Budget, Escaladra, AI Studio, Misguided Attention Problems, Trolley Problem, Farmer Paradox, 11 Labs V3 Alpha, Text-to-Speech (TTS), Quen 3 Embedding, Quen 3 Re-ranker, Retrieval Augmented Generation (RAG).

Gemini 2.5 Pro: Advancements and Limitations

• Main Topic: The video begins by discussing the release of Gemini 2.5 Pro, highlighting its improved coding capabilities and benchmark performance.
• Coding Prowess: The model can generate web apps from single prompts and make adjustments with ease.
• Benchmarks: Gemini 2.5 Pro leads 03 high on most benchmarks, including humanity's last exam (ADAR and GPQA). It lags behind O3 in mathematics but excels in code editing on the Ader Polycott benchmark.
• Thinking Budget: The model supports a "thinking budget," allowing users to set the number of tokens used for reasoning. However, only summaries of the thinking process are accessible, not the raw tokens.
• Regression Closure: The new version aims to address regressions observed in the 05 05 version.
• Escaladra Integration: A notable feature is the ability to generate Escaladra diagrams from benchmark data, enabling users to edit and refine them.
• Failure Cases: Despite improvements, the model struggles with certain prompts compared to previous versions.
• AI Studio Comparison: Using AI Studio's compare mode, the presenter demonstrates that the new version takes longer to process certain prompts and produces buggy code compared to the previous version.
• Reasoning Limitations: The model fails to correctly solve modified versions of the trolley problem and the farmer paradox, indicating limitations in logical deduction.
• Quote: Logan announced this new model is supposed to be state-of-the-art when it comes to humanity's last exam ADAR and GPQA.

11 Labs V3 Alpha: Enhanced Text-to-Speech

• Main Topic: The video then shifts to 11 Labs V3 Alpha, a text-to-speech system with enhanced control over audio output.
• Audio Quality and Control: The system offers improved audio quality and allows users to add specific expressions to the generated speech.
• Example: The presenter demonstrates the system's capabilities by generating a conversation with characters who chuckle and express different emotions.
• API Availability: The new version will soon be available through the API.
• Discount: 11 Labs is offering an 80% discount during June.

Quen 3 Embedding and Re-ranker: RAG Optimization

• Main Topic: The final segment focuses on Quen 3 Embedding and Re-ranker models, designed to improve the performance of Retrieval Augmented Generation (RAG) pipelines.
• RAG Pipeline Role: Embedding models retrieve relevant context for LLMs, while re-rankers filter out irrelevant text chunks.
• Model Sizes: Both embedding and re-ranker models are available in various sizes, ranging from 6B to 8B parameters.
• Open Source: The models are open source and can be downloaded from Hugging Face.
• Benchmarks: The 8B re-ranker outperforms other available re-ranking models, while even the 6B model performs well.
• Course Promotion: The presenter mentions a course on advanced RAG techniques, including multimodal retrieval for PDFs with images, text, and tables.

Synthesis/Conclusion

The video provides an overview of three significant releases: Gemini 2.5 Pro, 11 Labs V3 Alpha, and Quen 3 Embedding/Re-ranker. Gemini 2.5 Pro shows promise in coding and benchmark performance but has limitations in reasoning and prompt handling. 11 Labs V3 Alpha offers enhanced text-to-speech capabilities with improved audio quality and control. Quen 3 models provide valuable tools for optimizing RAG pipelines. The presenter encourages viewers to test these new releases and share their experiences.