OpenAI on Securing Code-Executing AI Agents — Fouad Matin (Codex, Agent Robustness)

Key Concepts

• Code-executing agents: AI agents capable of writing and executing code to achieve objectives.
• Sandboxing: Isolating the agent's execution environment to limit potential damage.
• Prompt injection: Exploiting vulnerabilities by injecting malicious prompts into the agent's input.
• Data exfiltration: Unauthorized extraction of sensitive data by the agent.
• Remote Code Execution (RCE): The ability of an agent to execute code on a remote system, posing security risks.
• Human review: Manual inspection of the agent's actions and outputs to ensure safety and correctness.
• Allow listing: Specifying a list of permitted resources or actions, restricting the agent's access.
• OS-level sandboxing: Using operating system features to isolate processes and limit their access to system resources.
• Containerization: Using containers to isolate the agent's execution environment.
• Seatbelt: Apple's sandboxing technology used on macOS.
• Seccomp and Landlock: Linux kernel features used for sandboxing.

Main Topics and Key Points

The Rise of Code-Executing Agents

• Frontier research labs are heavily investing in improving the coding capabilities, usability, and deployability of AI agents.
• The focus is shifting from simply writing code to efficiently achieving objectives through code execution.
• Recent models (e.g., GPT-3, GPT-4, Gemini) demonstrate increased reliability and capabilities in code execution compared to earlier models like 001.
• Code execution extends beyond software engineering tasks (SWE) and enhances multimodal reasoning. Example: an agent using OCR to decipher text from an image.
• The traditional complex inner loop with task-specific prompts and toolsets is being replaced by models that can autonomously decide when to use which tools and write/run code.

Security Risks and Mitigation Strategies

• Enabling code execution introduces security risks such as prompt injection, data exfiltration, accidental mistakes (installing malicious packages, writing vulnerable code), privilege escalation, and sandbox escape.
• OpenAI's preparedness framework emphasizes safeguards to prevent misalignment during large-scale deployment of code-executing agents.
• Sandboxing:
  • The primary recommendation is to sandbox the agent, ideally by giving it its own isolated computer (as done with Code Interpreter and ChatGPT).
  • If running locally, use containerization, app-level sandboxing (macOS seatbelt), or OS-level sandboxing (Linux seccomp and Landlock).
  • Codecs CLI is open-sourced to showcase how to build agents with sandboxing techniques.
• Disabling/Limiting Internet Access:
  • Crucial for preventing prompt injection and data exfiltration.
  • Untrusted content from the internet (e.g., GitHub issue comments) can compromise the agent's execution.
  • Codecs CLI offers a "full auto mode" where the agent can only read/write files within its directory and make network calls based on pre-approved commands.
  • ChatGPT now allows enabling internet access with configurable allow lists for specific domains and HTTP methods.
• Requiring Human Review:
  • Essential for ensuring that the agent's actions are safe and correct.
  • Human review is not fully replaceable by LM-based code review tools.
  • Reviewing operations, especially package installations, is critical to prevent malicious code from entering the codebase.
  • Tools like Operator can be used to monitor and review sensitive operations performed by the agent.

Building Secure Code-Executing Agents

• Deferring software-based logic to the reasoning model and providing it with the right tools.
• OpenAI released the "local shell" tool, mirroring how their models are trained to write and execute code.
• The "apply patch" tool helps models apply diffs to files accurately.
• Exposing services like Socket (dependency vulnerability checking) to the agent to verify the safety of dependencies.
• Using remote containers for agent execution, either self-hosted or through OpenAI's container service.

Examples and Case Studies

• Multimodal Reasoning: An agent using OCR to decipher text from an image, demonstrating the benefits of code execution beyond SWE tasks.
• Prompt Injection Example: A GitHub issue containing a malicious prompt that instructs the agent to exfiltrate data to a random URL.
• Codecs and ChatGPT: Using containerization to provide a fully isolated environment for code execution.
• Chromium: Heavily inspired the sandboxing mechanism on Mac OS.

Step-by-Step Processes and Methodologies

• Building Agents:
  1. Defer software-based logic to the reasoning model.
  2. Provide the model with appropriate tools (e.g., local shell, apply patch, web search).
  3. Implement sandboxing (containerization or OS-level).
  4. Control internet access with allow lists.
  5. Require human review of critical operations.
• Mitigating Prompt Injection:
  1. Disable or limit internet access.
  2. Implement system-level controls to restrict network calls.
  3. Use model-level controls to flag suspicious activities.

Key Arguments and Perspectives

• Every AI agent will eventually become a code-executing agent due to the increasing focus on efficiency and objective achievement.
• Security risks associated with code execution must be addressed proactively through sandboxing, internet access control, and human review.
• Balancing security with flexibility is crucial for enabling a wide range of use cases while minimizing risks.
• Model-level controls are not sufficient; system-level controls are essential for deterministic security.
• Human review remains a critical component of ensuring the safety and correctness of agent actions.

Notable Quotes

• "Every agent will become a codeexecuting agent."
• "Code isn't just for SWE tasks... it actually helps across the stack."
• "Combining those model level controls along with your kind of system level configurations is really key to solving this problem."
• "LM based monitors in the loop while valuable is just not quite there yet in terms of um the kind of certainty that you get from again a deterministic control"

Technical Terms and Concepts

• Code-executing agents: AI agents capable of writing and executing code to achieve objectives.
• Sandboxing: Isolating the agent's execution environment to limit potential damage.
• Prompt injection: Exploiting vulnerabilities by injecting malicious prompts into the agent's input.
• Data exfiltration: Unauthorized extraction of sensitive data by the agent.
• Remote Code Execution (RCE): The ability of an agent to execute code on a remote system, posing security risks.
• Human review: Manual inspection of the agent's actions and outputs to ensure safety and correctness.
• Allow listing: Specifying a list of permitted resources or actions, restricting the agent's access.
• OS-level sandboxing: Using operating system features to isolate processes and limit their access to system resources.
• Containerization: Using containers to isolate the agent's execution environment.
• Seatbelt: Apple's sandboxing technology used on macOS.
• Seccomp and Landlock: Linux kernel features used for sandboxing.
• MCP: Model Compute Platform.

Logical Connections

• The increasing capabilities of AI models in code execution necessitate a strong focus on security.
• Sandboxing, internet access control, and human review are presented as complementary strategies for mitigating the risks associated with code-executing agents.
• The discussion of building secure agents logically follows the identification of security risks and mitigation strategies.

Data, Research Findings, or Statistics

• Mention of the evolution of models from 001 to GPT-3/4/Gemini, highlighting increased reliability and capabilities.

Synthesis/Conclusion

The presentation emphasizes the growing importance of code-executing agents and the associated security challenges. It advocates for a multi-layered approach to security, combining sandboxing, internet access control, and human review. The speaker highlights the need for both model-level and system-level controls and stresses the importance of balancing security with flexibility to enable a wide range of use cases. The open-sourcing of Codecs CLI and the release of new tools like "local shell" and "apply patch" demonstrate OpenAI's commitment to providing developers with the resources they need to build secure and effective code-executing agents.