AI Engineer World’s Fair 2025 - Day 2 Keynotes & SWE Agents track

Key Concepts

• AI Engineering: The practice of building and deploying AI systems, focusing on practical application and reliability.
• Coding Agents: AI systems designed to automate or assist in software development tasks.
• Specifications: Formal descriptions of intent and values used to guide and evaluate AI systems.
• Inner Loop vs. Outer Loop: Inner loop refers to the code development process, while outer loop refers to the review, testing, and deployment process.
• Reasoning Models: LLMs that use more output tokens to provide greater intelligence.
• Open Weights Models: AI models with publicly available weights, allowing for greater customization and transparency.
• Test Time Compute: The amount of computation a model applies to a specific problem or question.
• Inference: The process of using a trained AI model to make predictions or generate outputs.
• Evals: Evaluations used to assess the performance and reliability of AI systems.
• MCP (Model Context Protocol): A protocol for providing models with access to external tools and information.
• Agent Native Development: A software development approach that leverages AI agents at every stage of the process.

Main Topics and Key Points

Gemini Updates and Google's AI Strategy (Logan Kilpatrick, Google DeepMind)

• New Gemini Model 2.5 Pro: A new version of Gemini Pro is released, addressing feedback from previous versions and improving performance across benchmarks like ADER and HLE. It's considered a turning point for Gemini.
• Year of Gemini Progress: Highlights the rapid pace of innovation and adoption of Gemini models, with a 50x increase in AI inference processed through Google servers in the past year.
• Organizational Structure: Discusses the integration of Google's AI research and product teams into DeepMind, enabling better collaboration and faster delivery of models and products.
• Gemini App as a Universal Assistant: The Gemini app is positioned as a unifying thread across Google's products, offering proactive assistance and personalized experiences.
• Model Roadmap: Future model developments include omnimodal capabilities (audio, image, video), video generation, agentic reasoning, smaller models, larger models, and infinite context handling.
• Developer Platform: AI Studio is evolving into a developer platform with features like state-of-the-art embeddings, a deep research API, and V3 and Imagine 4 in the API.

Thinking in Gemini (Jack Rae, Google DeepMind)

• Intelligence Bottlenecks: Progress in AI is driven by identifying and solving key bottlenecks, such as limited context length and fixed test time compute.
• Test Time Compute: The amount of computation a model applies to a specific problem, which is limited in traditional models.
• Thinking Stage: Introducing a thinking stage in Gemini allows the model to iteratively loop and perform additional test time compute before emitting a final answer.
• Reinforcement Learning: The model is trained to use the thinking stage via reinforcement learning, shaping how it uses its thinking computation to be more useful.
• Emergent Behavior: Reinforcement learning creates emergent behavior such as hypothesis testing, self-correction, problem breakdown, and tool use.
• Scaling Test Time Compute: Increasing test time compute improves reasoning performance, as demonstrated by the lineage of Gemini launches.
• Steering Models Quality Over Cost: Thinking budgets allow for a continuous budget, providing a more granular slider of how much capability is desired for a given task.
• Deeper Thinking: Deep Think is a high-budget mode built on top of 2.5 Pro, leveraging deeper chains of thought and parallel chains of thought for very hard problems.
• Future Directions: Improving model efficiency, making thinking more adaptive, and scaling inference compute further to drive even higher capability.

The Importance of Evals (Ankor Goyal, Brain Trust)

• Evals as a Key to Success: Evals are essential for contextualizing AI models and understanding if they work for real-world applications.
• Evals Beyond Unit Tests: Evals are not just for finding regressions but for running experiments and iterating on products before going to production.
• Data-Driven Signal: Applying the same metrics from offline to online production data provides data-driven signal about which examples in prod are most useful for the next iteration loop.
• AI Education Summit: A new chapter for AI education to bridge the gap between AI advancement and preparedness of children, parents, and educators.
• AI Education Summit Partnership: Partnering with Stefania Duga, a pioneer in AI education, to explore the landscape and provide practical knowledge for the future of AI education.

Coding Agents and Chaos (Solomon Hykes, Dagger)

• Platform Engineers and Coding Agents: Platform engineers are now tasked with enabling robots (coding agents) to ship software productively.
• Challenges of Scaling Coding Agents: Two options: YOLO mode (running multiple agents without control) and all-in-one platforms (limited flexibility).
• Desired Properties for Coding Agent Environments: Background work, rails (constraints), efficient stepping in, and optionality (choice of models, compute, etc.).
• Containers as a Foundational Technology: Containers play a crucial role in providing isolated, customizable, and multiplayer environments for coding agents.
• Container Use: A Dagger concept for using containers to create environments and work inside of them, distinct from sandboxing.
• Demo of Container Use: A demonstration of using cloud code with container use to create a simple homepage, showcasing features like environment listing, terminal access, secrets management, and history tracking.
• Open Sourcing Container Use: Dagger's container use project is open-sourced at github.com/dagger/containeruse.

Infrastructure for the Singularity (Jesse Han, Morph Labs)

• Empathy for the Machine: The need to understand and address the desires of thinking machines, particularly their need for speed and possibility.
• Infinibbranch: Virtualization, storage, and networking technology reimagined for thinking machines, enabling fast snapshotting, branching, and replication of virtual machines.
• Morph Liquid Metal: An improvement to Infinibbranch that enhances performance, latency, and storage efficiency.
• Cloud for Agents: Infinibbranch serves as a substrate for a cloud designed for agents, enabling declarative workspaces, frictionless workspace passing, and test time scaling.
• Similocra: The desire of thinking machines to be grounded in the real world and interact with complex software environments.
• Reasoning Time Branching: The ability to replicate and branch the environment during reasoning, allowing agents to explore multiple solutions in parallel.
• Verified Super Intelligence: A new kind of reasoning model capable of thinking for a long time, interacting with external software, and producing verifiable outputs.
• Magi 1: An early version of the verified super intelligence model being developed at Morph.
• Christian Seed Joining Morph: Announcement of Christian Seed joining Morph as chief scientist to lead the development of verified super intelligence.

Sweet Agents Track Summary

• Moore's Law for AI Agents (Scott Woo, Cognition): The capability of AI agents in code is doubling every 70 days, leading to rapid advancements in their ability to perform complex software engineering tasks.
• Asynchronous Coding Agents (Rustin Banks, Google): Jules, an asynchronous coding agent, is designed to run in the background and handle tasks in parallel, freeing up developers to focus on more creative work.
• GitHub Copilot: Past, Present, and Future (Christopher Harrison, GitHub): GitHub Copilot has evolved from code completion to chat and agent modes, with coding agent enabling asynchronous, server-side operations.
• The New Outer Loop (Tomas Ramirez, Graphite): AI is speeding up the inner loop of software development, but the outer loop (review, testing, deployment) is becoming a bottleneck, requiring new AI-native tools.
• Sculptor: A Case Study in Verifying AI-Generated Code (Josh Albert, Imbue): Sculptor is an experimental coding agent environment designed to help build user trust by identifying problems in AI-generated code early in the development process.
• Agent Native Development (Eno Reyes, Factory): Factory is building a platform for agent-driven development, where AI agents are used at every stage of the software lifecycle, from planning to incident response.
• The New Code (Shaun Grove, OpenAI): Specifications, formal descriptions of intent and values, are becoming more important than code itself, serving as a universal artifact for aligning humans and AI models.
• Fun Stories from Building Open Router (Alex Atallah, Open Router): Open Router is an LLM aggregator and distributor that aims to make inference a commodity, providing developers with access to a wide range of models and features.

Important Examples, Case Studies, or Real-World Applications Discussed

• Devon: Used for repetitive migrations, bug fixes, and broader feature requests.
• Jules: Used to add tests, calendar links, and Gemini summaries to a conference schedule website.
• GitHub Copilot: Used to create new endpoints and address tech debt in GitHub's own codebase.
• Diamond: Used to reduce code review cycles, enforce quality and consistency, and keep code private and secure.
• Infinibbranch: Used to enable reasoning time branching for chess-playing agents.
• Open Hands: Used to resolve merge conflicts, address PR feedback, fix small bugs, and make infrastructure changes.
• Sculptor: Used to learn what's out there, plan, write specs, and have a strict style guide.
• Factory: Used to plan features, generate runbooks, and improve team collaboration.

Step-by-Step Processes, Methodologies, or Frameworks Explained

• Agentic Loop: The process of an agent taking actions in the external world, getting feedback, and using that feedback to inform its next action.
• Deliberative Alignment: A technique for automatically aligning a model and its outputs against a specification.
• Test-Driven Development (TDD): A software development process where tests are written before the code, guiding the development process.
• Reasoning Time Branching: A method for improving reasoning by replicating and branching the environment, allowing agents to explore multiple solutions in parallel.

Key Arguments or Perspectives Presented, with Their Supporting Evidence

• Evals are essential for building reliable AI systems: Supported by the high usage of evals among Brain Trust customers and the need for data-driven signal.
• The outer loop of software development is becoming a bottleneck: Supported by the increasing volume of code generated by AI and the need for AI-native tools to streamline review, testing, and deployment.
• Specifications are more important than code: Supported by the argument that code is a lossy projection from the specification and that specifications are needed to align humans and AI models.
• Inference is becoming a commodity: Supported by the increasing number of models and providers available and the need for tools like Open Router to aggregate and standardize them.
• AI agents are not replacing software engineers, but amplifying their capabilities: Supported by the argument that AI agents are best used for rote tasks, freeing up developers to focus on more creative and strategic work.

Notable Quotes or Significant Statements with Proper Attribution

• Benjamin Duny: "AGI has been achieved." (referring to chat GPT)
• Logan Kilpatrick: "You build the future with your will tonight."
• Solomon Hykes: "Your job now is to enable robots to ship awesome software."
• Andre Karpathy: "English is the new programming language."
• Josh Albert: "A problem well-stated is half-solved."
• Eno Reyes: "We're transitioning from the era of human-driven software development to agent driven development."
• Shaun Grove: "The person who communicates most effectively is the most valuable programmer."

Technical Terms, Concepts, or Specialized Vocabulary with Brief Explanations

• ADER: A benchmark for evaluating the reasoning abilities of AI models.
• HLE: A benchmark for evaluating the human-level equivalence of AI models.
• Omnimodal: The ability to process and generate outputs in multiple modalities, such as audio, image, and video.
• Agentic Reasoning: The ability of a model to reason and take actions autonomously.
• Embeddings: Vector representations of data that capture semantic relationships.
• Diffusion: A technique for generating images and other data by gradually removing noise.
• TPUs: Tensor Processing Units, specialized hardware accelerators for AI workloads.
• VO: Video generation.
• TTS: Text-to-speech.
• DQN: Deep Q-Network, a reinforcement learning algorithm.
• MCP: Model Context Protocol, a protocol for providing models with access to external tools and information.
• Similocra: A representation or imitation.
• Syphency: The act of being a sycophant or people-pleaser.

Logical Connections Between Different Sections and Ideas

• The discussion of Gemini updates and Google's AI strategy leads into a deeper dive into the thinking process within Gemini.
• The importance of evals is highlighted as a way to ensure the reliability and quality of AI systems, which is crucial for deploying them in production.
• The challenges of scaling coding agents are addressed by proposing containerization as a solution for providing isolated and customizable environments.
• The need for empathy for the machine is presented as a motivation for developing infrastructure that can meet the needs of thinking machines.
• The sweet agents track provides a comprehensive overview of the current state of coding agents, from their origins to their future potential.

Data, Research Findings, or Statistics Mentioned

• 50x increase in AI inference processed through Google servers in the past year.
• AI is doubling every 70 days.
• Nearly every developer surveyed used AI tools both inside and outside of work.
• 46% of GitHub is being written by CP code on GitHub is being written by Copilot.
• 4% rate of comments that our AI bot leaves to be downloaded.
• 52% rate of Diamond comments be accepted.
• 80% of respondents say LLMs are working well at work but less than 20% say the same about agents.
• 65% of respondents are using a dedicated vector database.
• The mean guess for the percentage of US Gen Z population that will have AI girlfriends boyfriends is 26%.
• 65% on real eval.

Clear Section Headings for Different Topics

• Gemini Updates and Google's AI Strategy
• Thinking in Gemini
• The Importance of Evals
• Coding Agents and Chaos
• Infrastructure for the Singularity
• Sweet Agents Track Summary

A Brief Synthesis/Conclusion of the Main Takeaways

The AI landscape is rapidly evolving, with significant advancements in model capabilities, infrastructure, and development tools. While AI offers tremendous potential for automating and augmenting various tasks, it's crucial to address challenges related to reliability, alignment, and ethical considerations. By focusing on structured communication, formal specifications, and AI-native tools, we can harness the power of AI to build robust and beneficial systems.