Cracking open an open model | The Agent Factory

Key Concepts

• Agentic Capabilities: The ability of an AI model to understand its environment, make decisions, and take actions to achieve goals, mimicking human-like interaction.
• Open Models: AI models that are publicly available for use, modification, and distribution, fostering collaboration and innovation.
• Supervised Fine-Tuning (SFT): A training technique where a pre-trained model is further trained on a specific dataset to adapt it to a particular task or domain.
• Reinforcement Learning (RL): A machine learning paradigm where an agent learns to make decisions by performing actions in an environment and receiving rewards or penalties.
• Context Window: The amount of information a language model can consider at any given time when processing input and generating output.
• Distillation: A technique where a smaller "student" model is trained to mimic the behavior of a larger "teacher" model.
• Function Calling: The ability of an AI model to identify and invoke external tools or APIs to perform specific tasks.
• Reward Hacking: A phenomenon in reinforcement learning where an agent finds a way to maximize its reward without actually achieving the intended goal.
• Agentic Flow/Trajectory: The sequence of steps an agent takes, including function calls and responses, to complete a task.
• User-Focused Evaluation: Assessing AI model performance based on real-world user experience and satisfaction, rather than solely relying on benchmarks.

Agent Industry Pulse: Latest AI Agent Developments

The AI agent landscape is experiencing rapid advancements, with several key updates highlighted:

Gemini 2.5 Computer Vision Model

• Core Functionality: This model enables AI agents to "see" and interact with computer screens, mimicking human interaction with a virtual mouse and keyboard.
• Capabilities: It can perform actions like clicking buttons, typing into forms, and scrolling through web pages, moving beyond structured API interactions.
• Applications: Potential uses include automating complex form filling on websites, interacting with legacy systems, and assisting with data entry tasks.
• Key Features: Google claims it is faster and more reliable than previous models, with built-in security preventions.
• Accessibility: Developers can access it through AI Studio and Vertex AI.

Vibe Coding in AI Studio

• Purpose: Aims to simplify AI app development by reducing the "blank page" problem and API wrangling.
• Methodology: Developers describe the desired app's "vibe" or functionality in natural language (e.g., "Make me an app that takes a photo and turns it into a fantasy character").
• Process: AI Studio uses the latest AI models to handle the underlying code generation and API integration, allowing developers to focus on creativity.
• Interactive Refinement: Includes an annotation mode where users can highlight elements on the screen and provide direct instructions for changes (e.g., "Change this button to green," "Animate this picture").
• Impact: Makes app development more conversational and accessible to a broader audience.

DeepSeek OCR and Context Compression

• Problem Addressed: The challenge of feeding large documents (e.g., 100-page PDFs) into Large Language Models (LLMs) due to token limits and cost, leading to context loss.
• DeepSeek's Solution: Utilizes computer vision to process documents, capturing not just text but also the 2D layout, charts, and other visual elements.
• Context Compression: This visual information is then compressed into a highly efficient format (e.g., 10-20 text tokens into a single vision token), referred to as a "compressed map."
• Benefits: Results in significantly faster processing (10x) and reduced costs (20x) compared to traditional text-based LLM processing of long documents.
• Implication: Represents a shift towards leveraging visual capabilities alongside text understanding for comprehensive document processing.

V3.1 Video Model Update

• Advancements: Significant improvements to Google's AI video model, V3, with integration into filmmaking tools like Flow.
• Key Features:
  • Realistic Video Generation: Produces more lifelike video content.
  • Audio Integration: Generates rich audio synchronized with the video, moving beyond silent movies.
  • Advanced Editing Tools:
    • Insert Tool: Allows users to add new characters to a scene by specifying their actions, with the AI handling lighting and shadows.
    • Remove Tool: Enables users to erase unwanted elements, with the AI intelligently repainting the background to maintain scene coherence.
• Creator Control: Focuses on providing creators with iterative control over video production, enabling a more comprehensive and cohesive editing experience.
• Shift in Paradigm: Moves AI video generation from simple text-to-video to a more complete editing studio capability.

Building an Open Model with Agentic Capabilities: The Factory Line

This section delves into the practical process of building and training an open-source AI model with agentic capabilities, featuring insights from Ravine Kumar, a researcher at Google DeepMind.

1. Data Curation and Preparation

• Data Requirements: Training an agent that acts and uses tools requires a different data structure than standard natural language data. The data needs to be structured for computer interpretation.
• Data Sources:
  • Computer Interactions: Data from interactions with systems like Google Cloud APIs serves as a valuable starting point.
  • Real-World API Calls: Examples of actual API calls and their corresponding responses.
  • Synthetic Data Generation: Leveraging larger, more capable models (e.g., Gemini) to generate synthetic data that mimics real-world API usage for smaller open models. This process is akin to "distillation."
• Data Structuring: Data is structured in a specific format that inference engines can interpret to call APIs in the background, allowing the model to then provide natural language responses.
• Focus on "Good Decisions": The data aims to capture what a user would want the model to do, essentially representing "good decisions" in various scenarios.

2. Post-Training Techniques

The process typically starts with a pre-trained open model (e.g., Gemma) and then applies post-training techniques to imbue it with agentic capabilities.

• Supervised Fine-Tuning (SFT):

  • Objective: To teach the base model a specific format for function calling, which is distinct from human interpretation and can handle variability.
  • Process: The model is trained on curated datasets that demonstrate how to correctly format and execute function calls.

• Reinforcement Learning (RL):

  • Objective: To enable the model to generalize beyond the examples seen during SFT and explore new use cases and APIs.
  • Methodology: The model learns through trial and error, guided by a reward system.
  • Reward Design Challenges:
    • Full Trajectory Reward: Rewarding only for the successful completion of an entire task can be ineffective if the model struggles to reach the end.
    • Intermediate Step Rewards: Rewarding for completing sub-tasks can lead to "reward hacking," where the model optimizes for intermediate steps without completing the overall goal.
    • Progressive Reward Design: A combination of both approaches is often used, starting with rewards for early steps and gradually shifting to rewarding only for final task completion.
  • Monitoring: Close monitoring of reward charts during training is crucial to ensure the model learns the desired behaviors at the appropriate stages.

3. Evaluation of Agentic Capabilities

Evaluating agentic models is more complex than evaluating traditional LLMs due to the potential for wrong actions and real-world consequences.

• Standard Benchmarks: Utilizing existing published benchmarks to assess performance against established metrics.
• User-Focused Evaluation:
  • Internal Evaluation Sets: Creating custom evaluation datasets based on real-world customer interactions and expected user experiences with Google products.
  • Internal Dogfooding: The development team actively uses and tests the models by building internal applications and demos.
  • User Feedback: Incorporating feedback from internal testers and developers who use the models in their own projects is considered the most important component of evaluation.
• Key Metrics: The evaluation aims to assess not only capability but also safety and user satisfaction. The process is described as a "sandbox obstacle course" rather than a simple quiz.

Key Arguments and Perspectives

• Data is Paramount: The success of agentic models hinges on the quality and structure of the training data, which must differ significantly from standard text data.
• Hybrid Training Approach: A combination of supervised fine-tuning for foundational knowledge and reinforcement learning for generalization and exploration is essential.
• RL Complexity: Designing effective reward functions in RL for agentic tasks is a significant challenge requiring careful consideration of intermediate steps and final outcomes.
• Beyond Benchmarks: While benchmarks provide a baseline, real-world user-focused evaluation is critical for ensuring models are truly useful and safe in production.
• Iterative Improvement: The entire process, from data curation to training and evaluation, is iterative, with continuous refinement based on observed performance and user feedback.

Notable Quotes

• "This model can actually look at a computer screen, understand what it sees, and then interact with it for you just like you or I would." (Describing Gemini 2.5's capabilities)
• "Vibe coding seems to be changing a lot of that by using AI to develop for you and AI studio now allows you that capability." (On simplifying app development)
• "It's all about giving creators iterative control, not just one shot or nothing roll of the dice." (On advancements in AI video editing)
• "So, the inference engines can grab that format and call that API in the background — for the user. So then the model can come back later and give a nice natural language response." (Ravine Kumar, explaining data structuring for function calling)
• "Reinforcement learning is one of these training techniques that let the model like quote unquote explore and try new things — outside of just the data set that — we presented during supervised finetuning." (Ravine Kumar, on the role of RL)
• "So we really want to make sure the user is going to be happy and that they're going to have a great experience." (Ravine Kumar, on user-focused evaluation)

Synthesis and Conclusion

The YouTube video transcript highlights a significant evolution in AI, moving beyond simple text generation to more interactive and capable agents. The "Agent Industry Pulse" segment showcases advancements in computer vision-enabled agents (Gemini 2.5), simplified app development through natural language prompts (Vibe Coding), efficient long-context processing (DeepSeek OCR), and sophisticated AI video editing tools (V3.1).

The core of the discussion then shifts to the intricate process of building open-source AI models with agentic capabilities. This involves a multi-stage approach: meticulous data curation that goes beyond text to capture interaction trajectories, a hybrid training methodology combining supervised fine-tuning for foundational skills and reinforcement learning for generalization and exploration, and a rigorous, user-centric evaluation process that prioritizes real-world performance and safety over mere benchmark scores. The insights from Ravine Kumar underscore the complexity and sophistication involved in creating AI agents that can effectively understand, decide, and act in dynamic environments, ultimately aiming to provide more useful and reliable AI tools for developers and end-users alike. The common thread across all discussed technologies is the increasing capability, visual understanding, and practical utility of AI.