Replay: Rubber Duck Thursdays: Building Agents with Copilot

Key Concepts

• AI Agents: Autonomous or semi-autonomous software entities that use LLMs to perform tasks.
• Middleware: A filtering layer placed between the user prompt and the LLM to enforce safety, compliance, and guardrails.
• MCP (Model Context Protocol): A standard for connecting AI assistants to systems, data, and tools.
• Guardrails: Security mechanisms designed to prevent AI from generating harmful, insecure, or non-compliant content.
• Zero Trust Security: A security framework requiring strict identity verification and continuous validation for every person and device trying to access resources.
• Rubber Ducking (AI): A development practice where a secondary AI model reviews code changes to identify vulnerabilities or logic errors.

---

1. Building and Deploying AI Agents

The speaker emphasizes that the industry is shifting from experimental "fun" projects to production-ready AI agents. A key takeaway is that niche agents (e.g., for pest control compliance or specific HR platforms) are generally more effective and reliable than generalized ones.

• Frameworks vs. Scratch: While building from scratch is possible, the speaker advocates for using established frameworks (like the Microsoft Agent Framework or LangChain) to speed up development and easily integrate pre-built tools like MCP servers.
• Microsoft & Anthropic Partnership: Claude models are now available via Azure, allowing developers to build agents using Claude within the Microsoft ecosystem.

2. Security and Guardrails

A significant portion of the discussion addresses the challenge of maintaining security in AI-generated code.

• The Challenge: AI agents can inadvertently introduce vulnerabilities, such as insecurely mounting authenticated upload routes.
• Middleware as a Solution: The speaker explains that middleware acts as a "filter" or "pre-sandbox." Before a prompt reaches the LLM, the middleware inspects it against defined categories and severity thresholds. If the prompt violates safety policies, it is blocked before the LLM ever processes it.
• Enterprise Platforms: For enterprise-grade security, the speaker recommends hosting agents on platforms like Azure, which provide built-in content safety and guardrails by default.

3. Model Context Protocol (MCP)

MCP is presented as a vital tool for providing agents with external data.

• Functionality: An MCP server acts as a bridge, providing the agent with specific context or tools (e.g., a "cupcake ordering" tool).
• Context Management: The speaker notes that agents perform better when context is "compacted." Overloading an LLM with too much context can lead to latency and performance degradation. Frameworks often handle this by automatically summarizing or truncating context.

4. Development Best Practices

• Rubber Ducking: When using tools like the Copilot CLI, the system may automatically bring in a second model family to review code changes. The speaker suggests trusting these automatic checkpoints for large changes, as they are designed to catch vulnerabilities that a single model might miss.
• Environment Management: The speaker highlights the importance of using virtual environments (e.g., venv, uv) to manage dependencies and ensure that the correct libraries are available for agent execution.

5. Notable Quotes

• "I think the more niche your agent is going to be, the better it is going to be." — On the effectiveness of specialized AI agents.
• "Middleware is like a filter... it's going to take in that prompt, it's going to filter any bad things... and that query is not even going to reach the LLM." — Explaining the mechanism of security guardrails.

6. Synthesis and Conclusion

The session highlights that while building AI agents is becoming more accessible, the primary hurdle for developers is moving from experimentation to secure, production-ready applications. The integration of middleware for content moderation and the use of MCP for structured data access are essential strategies for building robust agents. Developers are encouraged to leverage existing frameworks and cloud-native security features (like those on Azure) rather than attempting to build every security layer from scratch.

Resources mentioned: The speaker provided a link to a GitHub repository for a workshop on building agents with Claude and the Microsoft Agent Framework, which includes practical exercises like building an MCP-based ordering system.