Creating Agents that Co-Create — Karina Nguyen, OpenAI

Key Concepts

• Next Token Prediction
• Pre-training
• Post-training
• Reinforcement Learning from Human Feedback (RLHF)
• Reinforcement Learning from AI Feedback (RLAIF)
• Chain of Thought (CoT)
• Scaling Reinforcement Learning
• Faithfulness in Chain of Thought
• Agents
• Co-innovators
• Human-AI Collaboration
• Rapid Iteration Cycle
• Synthetic Data
• Distillation
• In-Context Learning
• Modular Compositions
• Real-time Interaction
• Asynchronous Task Completion
• Collaborative Affordances
• Multi-Agentic Collaboration
• Personalized Tutors
• Generative Entertainment
• Invisible Software Creation
• Models Lens
• Creative Co-direction

Scaling Paradigms in AI Research

The speaker, Karina, an AI researcher, discusses two major scaling paradigms in AI research over the past 2-4 years and how they have unlocked new product research frontiers.

Next Token Prediction (Pre-training)

• World Building Machine: The model learns to understand the world by predicting the next word (token) in a sequence.
• Irreversible Sequences: The model learns the "physics" of the world because certain sequences are caused by initial actions and are irreversible.
• Tokens: Can be strings, words, pixels, or any other data unit.
• Massive Multitask Learning: Pre-training enables the model to learn various tasks simultaneously.
• Easy vs. Hard Tasks: Some tasks like translation are easier to learn due to abundant data, while others like complex problem-solving, logical reasoning, and creative writing are harder.
• Compute Importance: Scaling compute during pre-training is crucial for learning complex tasks.
• Chain of Thought (CoT): More compute can be spent on Chain of Thought to help the model reason through more computationally intensive tasks.
• Creative Writing Challenge: Difficult because it involves world-building, storytelling, and plot coherence, which are hard to measure and prone to errors.

Scaling Reinforcement Learning on Chain of Thought

• Complex Reasoning: This paradigm involves scaling reinforcement learning by training models to "think" during training and learn from feedback.
• Test Time Compute: Requires spending more compute on training to scale reinforcement learning.
• Chain of Thought Example: The speaker shows an example of the output of a normal model vs. a model using Chain of Thought to solve complex problems.
• Faithfulness in CoT: The speaker mentions the importance of faithfulness in Chain of Thought and the need for research on how to measure it and handle incorrect reasoning paths.
• Interaction Paradigm Shift: The speaker discusses the need for new interaction paradigms with humans, such as streaming the model's thoughts to the user to reduce waiting time.

Product Research and Development

• Rapid Iteration Cycle: The speaker notes that the scaling paradigms have enabled a rapid iteration cycle in product development.
• Distillation: Highly reasoning models can be distilled into smaller, faster models.
• Synthetic Data Generation: Complex reasoning models can synthetically generate new data for post-training and reinforcement learning.
• New Class of Tasks: The speaker mentions the possibility of creating entirely new classes of tasks, such as multiplayer collaboration with AI.
• Complex RL Environments: Models can be trained in more complex reinforcement learning environments with tools like search, browsing, and collaborative platforms.
• In-Context Learning: Models can quickly learn new tools and behaviors through few-shot examples.
• Synthetic Data and Distillation: These techniques can be used to invent new model behaviors and interactions based on user feedback.

Vignettes and Lessons Learned

The speaker shares several vignettes from their experience at Anthropic and OpenAI, highlighting key lessons learned.

Bringing Unfamiliar Capability into Familiar Form Factor

• 100K Context Example: The success of 100K context was due to its integration into a familiar form factor: file uploads.
• Design Challenge: Finding the simplest form factor for unfamiliar capabilities is a key design challenge.

Modular Compositions and Scalability

• ChatGPT Tasks Example: The speaker discusses how ChatGPT Tasks, while based on familiar concepts like reminders and to-do lists, can be scaled with new model capabilities.
• Modular Compositions: Product features should enable modular compositions that can scale as models develop higher capabilities.

Bridging Real-time Interaction and Asynchronous Task Completion

• Trust Bottleneck: The speaker identifies trust as a bottleneck in asynchronous task completion, where models work independently for extended periods.
• Collaborative Affordances: Trust can be built by providing humans with collaborative affordances to verify and edit model outputs and provide real-time feedback.

Collaborative Affordances and Creative Capabilities

• Canvas Example: The speaker highlights the flexibility of Canvas as a collaborative interface for writing, coding, and data analysis.
• Multi-Agentic Collaboration: Canvas can support multi-agentic collaboration, where multiple AI agents work together with humans.
• Personalized Tutors: Models are becoming multimodal and flexible enough to serve as personalized tutors, adapting to individual learning styles.
• Generative Entertainment: The speaker mentions using Canvas to create a game on a plane, demonstrating the potential for generative entertainment.

Invisible Software Creation

• Democratization of Tool Creation: The speaker envisions a future where anyone can create their own software tools, even without coding experience.
• Pair Programming and Code Creation: AI can assist in pair programming and code creation to enable this future.

AI-Assisted Research

• Reproducibility and Hypothesis Generation: AI can assist in research by reproducing existing work, generating new hypotheses, and verifying research directions.
• Interactive Paradigm: The speaker emphasizes the importance of an interactive paradigm where humans and AI can work together to create new knowledge.

Future Vision

• Invisible Software Creation: The speaker envisions a layer of invisible software creation accessible to all, especially from mobile devices.
• Models Lens: The way people access the internet will change, with models acting as a "lens" to provide cleaner and more personalized experiences.
• Personalized Multimodal Outputs: Instead of text outputs, models will provide personalized multimodal outputs, such as interactive 3D visualizations.
• Blank Canvas Interface: The interface to AI will be a "blank canvas" that adapts to the user's intent, transforming into an IDE, writing tool, or other application as needed.
• Creative Co-direction: Co-innovation will happen through creative co-direction with highly reasoning agent systems capable of super-human tasks.

Conclusion

The speaker concludes by emphasizing the transformative potential of AI, particularly in enabling creative collaboration and democratizing access to software creation and knowledge. The future involves a shift towards more personalized, interactive, and multimodal experiences, where AI acts as a powerful tool for human innovation and creativity.