TNS Agents Livestream: Ermon Stefano, Inception Labs

Key Concepts

• Diffusion Models: Generative models that create data (images, text, etc.) by progressively removing noise from a random starting point.
• Autoregressive Language Models (LLMs): Traditional LLMs that generate text sequentially, predicting the next token based on previous tokens. (e.g., GPT models)
• Inference: The process of using a trained model to generate outputs (e.g., text, images) from new inputs.
• Reinforcement Learning (RL): A training method where a model learns to make decisions by receiving rewards or penalties for its actions.
• Mercury 2: Inception Labs’ latest diffusion-based language model, designed for speed and efficiency.
• Token: The basic unit of text processed by LLMs (e.g., a word or part of a word).
• Denoising Steps: The iterative refinement process in diffusion models, where noise is gradually removed to generate the final output.

The Rise of Diffusion Language Models: A Deep Dive with Stefano Airmon of Inception Labs

This conversation with Stefano Airmon, co-founder and CEO of Inception Labs, details the development and potential of diffusion models for language processing, contrasting them with traditional autoregressive LLMs. The discussion centers around Inception Labs’ recently launched Mercury 2 model and its implications for the future of AI.

From Image Generation to Language: A Historical Perspective

Airmon’s journey began with a PhD in computer science at Stanford, focusing on generative models. Initially, his work centered on image generation, specifically exploring alternatives to Generative Adversarial Networks (GANs). GANs, while effective, proved unstable during training due to their adversarial nature. In 2019, his lab proposed diffusion models – a method of generating images by progressively removing noise from a random starting point. This approach quickly surpassed GANs in scalability, training stability, and output quality, leading to the rise of models like Stable Diffusion and Midjourney.

The challenge then became adapting diffusion models to discrete data like text, where interpolation between elements (words) lacks inherent meaning, unlike continuous data like pixels in an image. A breakthrough paper from Airmon’s lab in 2024 demonstrated that diffusion models could achieve comparable quality to autoregressive models on text, but with a tenfold increase in speed.

The Core Difference: How Diffusion Models Work

The fundamental difference lies in the training process. Autoregressive LLMs predict the next token in a sequence, essentially functioning as “fancy autocomplete.” Diffusion models, however, operate by learning to fix mistakes. They start with clean text, add noise, and train a neural network to remove that noise. This allows the network to consider context from both sides of a word, leading to a more global understanding of the text.

During inference, diffusion models don’t generate text sequentially. Instead, they begin with a rough approximation of the answer and iteratively refine it, similar to the image generation process. This parallel processing capability is key to their speed advantage. The network can modify multiple tokens simultaneously, unlike autoregressive models which process one token at a time.

Speed and Efficiency: The Advantage of Parallel Processing

The speed advantage stems from shifting the computational bottleneck from memory-bound operations (moving data) to compute-bound operations (performing calculations). Autoregressive models are inherently sequential, limiting parallelization. Diffusion models, by processing multiple tokens concurrently, leverage the parallel processing power of GPUs more effectively. Mercury 2 achieves speeds five to ten times faster than comparable autoregressive models while maintaining competitive quality.

Airmon emphasized that the number of “denoising steps” (refinement iterations) is crucial. A smaller number of steps allows for faster processing without sacrificing quality, a key achievement of Inception Labs’ research. The model also utilizes “test time inference” and “thinking models” to improve quality, allowing for iterative refinement and contextual understanding.

Training and Scaling Laws

While specific training details are proprietary, the core principles remain similar to image diffusion model training. Transformers are used as the underlying neural network architecture. Scaling laws observed in image generation also apply to text, indicating a correlation between model size and quality. However, Airmon believes there’s still significant room for improvement in diffusion language models, potentially surpassing the performance of current autoregressive models.

Mercury 2: Performance and Applications

Mercury 2 achieves quality comparable to speed-optimized models from Frontier Labs (like Haiku), while being significantly faster. This makes it ideal for latency-sensitive applications such as:

• Coding Agents: Providing real-time code suggestions and refactoring.
• Voice Agents: Enabling quick and responsive customer support.
• Search and Information Retrieval: Rewriting queries, ranking results, and summarizing information.

The model supports tool calling and is compatible with OpenAI’s API, making integration straightforward.

The Future: Multimodality and Hybrid Approaches

Airmon expressed excitement about extending diffusion models to multimodal applications (handling images, text, and other data types). Given the success of diffusion models in image and video generation, a unified multimodal model seems like a natural progression. He also suggested exploring hybrid approaches, combining the strengths of autoregressive and diffusion models – potentially using an autoregressive model for high-level planning and a diffusion model for detailed refinement.

Competition and the Path Forward

While Google has demonstrated diffusion models for language, Inception Labs currently holds a lead in commercial availability and production deployment. Airmon anticipates increased competition as other labs recognize the potential of this technology. He believes the future will be determined by inference efficiency – the ability to deliver the best performance per dollar and watt.

Reinforcement Learning and Continued Development

Reinforcement learning (RL) played a crucial role in optimizing Mercury 2. The speed of diffusion models allows for faster generation of “rollouts” (trial outputs) for RL training, accelerating the learning process. Airmon emphasized that Inception Labs is focused on continuous improvement, optimizing speed, quality, and cost through systems-level optimizations and ongoing research. While publishing research papers has become more challenging due to competitive pressures, the company continues to share technical reports to keep the community informed.

Conclusion:

Inception Labs’ work with diffusion language models represents a significant shift in the landscape of AI. By leveraging the parallel processing capabilities of diffusion models, they’ve achieved a compelling combination of speed and quality, opening up new possibilities for real-world applications. Airmon’s vision suggests that diffusion models have the potential to become the new standard in language processing, driven by their inherent efficiency and scalability.