Ship Agents that Ship: A Hands-On Workshop - Kyle Penfound, Jeremy Adams, Dagger

Key Concepts

• Dagger: A container runtime and workflow engine for software engineering workflows and environments.
• Dagger Modules: Reusable components written in various languages (Go, Python, TypeScript, Java, PHP) that define functions and workflows.
• Daggerverse: A community-driven index of Dagger modules.
• Containerized Environments: Sandboxed environments built with Dagger, providing tools and dependencies for agents or CI/CD pipelines.
• AI Agents: Autonomous entities that use Dagger modules as tools to perform tasks like code generation, testing, and deployment.
• LLMs (Large Language Models): The "brains" of the agents, integrated with Dagger to process prompts and interact with the environment.
• GitHub Actions: A CI/CD platform used to automate agent execution based on GitHub events (e.g., issue labeling).
• MCP (Module Composition Protocol): A protocol for composing and managing modules, with potential future integration with Dagger.

Main Topics and Key Points

1. Introduction to Dagger and its Purpose

• Dagger is presented as a tool for building portable software engineering workflows, encompassing both CI/CD and AI agent development.
• It allows defining workflows as code, supporting multiple languages and cross-language interoperability.
• Dagger's core components include containers, repos, directories, files, and LLMs, which can be combined to create complex pipelines.
• The founders of Docker are also the founders of Dagger. Docker focused on containerizing applications, while Dagger focuses on making entire workflows portable.

2. Setting up the Development Environment

• The workshop uses a "Hello Dagger" template project on GitHub as a starting point.
• Participants are instructed to create a repository from the template for easier GitHub Actions integration.
• The Dagger CLI is installed using Homebrew, an install script, or windget, requiring a container runtime like Docker or Podman.
• Dagger Cloud is introduced as a visualization tool for monitoring agent behavior and debugging prompts.

3. Building a CI Pipeline with Dagger

• The workshop walks through creating a basic CI pipeline using Dagger functions.
• Example functions include build, test, and publish, which define containerized steps for building and testing the project.
• The code demonstrates how to create Dagger containers, add files, and execute commands within them.
• The goal is to establish a foundation for integrating an AI agent into the existing CI/CD process.

4. Adding an AI Agent to the Project

• A new Dagger module called "daggerworkspace" is created to define the agent's specific tools and capabilities.
• Functions within the "daggerworkspace" module include:
  • read_file: Reads the contents of a file in the workspace.
  • write_file: Writes content to a file in the workspace.
  • list_files: Lists the files and directories in the workspace.
  • test: Runs the project's test suite.
• The "daggerworkspace" module is installed as a dependency of the main Dagger module, making its functions available to the agent.
• The concept of giving the agent a "refined environment" with only the necessary tools is emphasized.

5. Creating the Agentic Function

• A function called develop is created to represent the AI agent.
• The develop function takes an "assignment" (a task description) as input and returns a "completed" directory with the modified code.
• The environment for the agent is defined, including the assignment, the "daggerworkspace" module, and the desired output (completed workspace).
• A prompt is created in a separate file (dagger/develop_prompt.md) to instruct the agent on how to perform its task.
• The prompt emphasizes analyzing the workspace, avoiding unnecessary changes, and always running tests.
• The core of the agent is created using dagger.llm.Completion.with_environment_and_prompt, which combines the Dagger client, LLM, environment, and prompt.

6. Running the Agent Locally

• The dagger shell command is used to interact with the Dagger environment and execute the develop function.
• An LLM provider (e.g., OpenAI, Gemini, Anthropic) must be configured with API keys.
• Dagger's secrets provider integration is used to securely manage API keys using one password.
• The agent is invoked with a specific assignment, such as "make the main page say hello workshop people."
• Dagger Cloud is used to visualize the agent's actions, including tool calls, file reads, and writes.
• The output of the agent (the "completed" directory) can be saved to a variable and used in subsequent Dagger functions.

7. Deploying the Agent to GitHub

• The "GitHub issue" Dagger module is installed to enable interaction with GitHub issues and pull requests.
• A function called develop_issue is created to automate the agent's execution based on GitHub issue labeling.
• The develop_issue function reads the issue body as the assignment, runs the agent, and creates a pull request with the modified code.
• Two repository secrets are created in GitHub: one for the Dagger Cloud token and one for the LLM API key.
• A GitHub Actions workflow is created to trigger the develop_issue function when an issue is labeled with "develop."
• The workflow grants the necessary permissions to the GitHub token (write contents, read issues, write pull requests).
• The workshop demonstrates how to create a new issue, label it with "develop," and trigger the automated agent execution.

8. Additional Considerations and Examples

• The workshop briefly touches on more advanced topics, such as reflection agents and agents that write Dagger code.
• The "greetings API" demo project is mentioned as a resource for exploring more complex agent workflows, including feedback loops and automated code review.
• The possibility of using Dagger to implement MCP servers and integrate with the MCP ecosystem is discussed.
• The use of Dagger with other agent frameworks (e.g., OpenAI agents SDK) is mentioned as an alternative approach.
• The ability to run headless browsers within Dagger containers for testing web applications is also discussed.

Step-by-Step Processes, Methodologies, or Frameworks Explained

1. Creating a Dagger Module:
   • Use dagger init to create a new module with a specified SDK (e.g., Python).
   • Define functions within the module to encapsulate specific tasks or workflows.
   • Use Dagger's core components (containers, directories, files) to build the functions.
   • Use dagger functions to view the available functions in the module.
2. Installing Dagger Modules:
   • Use dagger install <module_name> to add a Dagger module as a dependency to another module.
   • The installed module's functions become available as methods on the Dagger client.
   • Dependencies are listed in the Dagger.json file.
3. Creating an AI Agent:
   • Define an environment for the agent, including the assignment, the tools (Dagger modules), and the desired output.
   • Create a prompt to instruct the agent on how to perform its task.
   • Use dagger.llm.Completion.with_environment_and_prompt to create the agent.
4. Automating Agent Execution with GitHub Actions:
   • Create a GitHub Actions workflow to trigger the agent based on specific events (e.g., issue labeling).
   • Define repository secrets for API keys and other sensitive information.
   • Grant the necessary permissions to the GitHub token.
   • Use the dagger call command to invoke the agent function within the workflow.

Key Arguments or Perspectives Presented, with Their Supporting Evidence

• Dagger enables portable and reproducible software engineering workflows: The same Dagger code can be run on a developer's machine, in CI/CD pipelines, or by AI agents, ensuring consistency across environments.
• Dagger provides a secure and sandboxed environment for AI agents: Agents operate within containers, preventing them from directly modifying the host file system or accessing sensitive data.
• Dagger promotes modularity and reusability: Dagger modules can be shared and reused across projects, reducing code duplication and improving maintainability.
• Dagger facilitates the integration of AI into existing software development processes: Agents can be seamlessly integrated into CI/CD pipelines to automate tasks like code generation, testing, and deployment.
• Dagger provides visibility into agent behavior: Dagger Cloud allows developers to monitor agent actions, debug prompts, and identify areas for improvement.

Notable Quotes or Significant Statements with Proper Attribution

• "Dagger's for software engineering workflows and environments." - Jeremy Adams
• "We're creating these building blocks and as you scale this up, you can consume these from that other people have written, you don't have to write it all from scratch." - Kyle

Technical Terms, Concepts, or Specialized Vocabulary with Brief Explanations

• Container Runtime: Software that runs containers (e.g., Docker, Podman).
• Workflow Engine: A system for orchestrating and executing a series of tasks or steps.
• SDK (Software Development Kit): A set of tools and libraries for developing software in a specific language or platform.
• API (Application Programming Interface): A set of rules and specifications that allow different software systems to communicate with each other.
• Prompt: A text-based instruction given to an LLM to guide its behavior.
• Environment Variable: A variable that is set outside of a program and can be accessed by the program at runtime.
• Repo Secret: A secure way to store sensitive information (e.g., API keys) in a GitHub repository.
• GitHub Actions Workflow: A YAML file that defines a series of steps to be executed in response to specific GitHub events.
• OpenTelemetry: An open-source observability framework used by Dagger Cloud to collect and visualize tracing data.

Logical Connections Between Different Sections and Ideas

• The workshop starts by introducing Dagger as a general-purpose tool for software engineering workflows.
• It then focuses on building a specific CI pipeline as a foundation for integrating an AI agent.
• The agent is created by defining its environment, tools, and prompt.
• The agent is first run locally to test its behavior and debug its prompt.
• Finally, the agent is deployed to GitHub using GitHub Actions, automating its execution based on issue labeling.
• The workshop emphasizes the importance of modularity, reusability, and visibility throughout the entire process.

Data, Research Findings, or Statistics Mentioned

• The speaker mentions an organization where product managers are generating large pull requests (25,000 lines of code) using AI-powered IDEs, highlighting the need for automated testing and validation.

Clear Section Headings for Different Topics

(As provided in the "Main Topics and Key Points" section)

Brief Synthesis/Conclusion of the Main Takeaways

The workshop demonstrates how Dagger can be used to build portable, reproducible, and secure software engineering workflows that integrate AI agents. By combining Dagger's core components with LLMs and GitHub Actions, developers can automate tasks like code generation, testing, and deployment, while maintaining control and visibility over the agent's behavior. The workshop emphasizes the importance of modularity, reusability, and a well-defined environment for creating reliable and effective AI agents.