Claude AGENT TEAMS just built my AI Double Agent

Key Concepts

• Private LLM Interaction: Enabling interaction with powerful cloud LLMs (like Claude Haiku) using sensitive company data without exposing it to the cloud.
• Redaction & Anonymization: Protecting PII through reversible anonymization (replacing with surrogates) and redaction (complete removal) before data reaches the LLM, with deanonymization for user output.
• Agentic RAG: Utilizing Claude Code’s Agent Teams to build AI agents grounded in private knowledge through Retrieval Augmented Generation.
• Local LLM Integration: Employing local LLMs (like Quinn 3 via LM Studio) for entity resolution to enhance privacy and potentially enable air-gapped operation.
• Iterative Development & Debugging: Employing parallel agent workflows and tools like Langfuse for rapid bug fixing, testing, and refinement of the system.

Building a Private LLM Interaction System

The project focuses on enabling interaction with Large Language Models (LLMs) using private company data while maintaining strict data privacy. The core principle is to prevent sensitive information from leaving the network by implementing a system of redaction and anonymization. This is driven by compliance requirements (GDPR, HIPAA, CCPA, PCI DSS) and the inherent security risks of AI systems, including prompt injection and data exfiltration. The speaker advocates for proactive technical safeguards alongside legal agreements with cloud providers.

Initial Implementation & Challenges (Part 1)

The initial system design involved anonymizing Personally Identifiable Information (PII) – such as Social Security Numbers, passports, driver’s licenses, financial records, medical records, and trade secrets – before sending data to a cloud-based LLM (Claude Haiku). An example query, “Tell me about Sarah Williams who is a staff member,” demonstrated the process, where “Sarah Williams” was anonymized to “Pamela Nelson” using a registry (Superbase) for mapping. The system then swaps back to the original name for document search, redacting/anonymizing sensitive information in the search results before sending them to the LLM and finally deanonymizing the LLM’s response.

Claude Code Agent Teams were leveraged for parallel execution of tasks, such as researching Microsoft Presidio. However, initial testing revealed that simple name replacement using a Faker library for anonymization resulted in “a total lack of coherence” and potential data leaks. Entity recognition systems are not perfect, with studies showing they can miss over 5% of sensitive entities. Token usage with Agent Teams was also noted as being high (e.g., 112,000 tokens in 60 seconds).

Iterative Refinement & Local LLM Integration (Part 2)

The development process involved iterative refinement through parallel agent processing. Three sub-agents were spun up to address identified bugs, consuming significant tokens (63,000, 48,000, 34,000). A major challenge emerged: accurately resolving variations of the same entity (e.g., “Margaret Thompson,” “Maggie Thompson”). This led to incoherent outputs and inaccurate responses.

To address this, the system explored integrating local LLMs (Quinn 3 via LM Studio) for entity resolution. Quinn 3 achieved a speed of nearly 200 tokens per second. This approach aimed to improve accuracy and enable air-gapped operation for enhanced security. The architecture was simplified to focus on a direct interface between local and cloud LLMs.

The concept of a “conversational vault” – an accumulated registry of entity mappings – was introduced to maintain coherence during interactions. Langfuse was integrated for tracing and debugging, offering a self-hosted alternative to Langsmith. Testing involved employee records, incident reports, and security incident analysis, demonstrating the system’s potential for various applications. The entity registry grew to 68 entities after a single conversation.

Technical Components & Tools

The system utilizes a range of technical components, including: LLMs (Claude Haiku, Quinn 3), vector databases, Superbase (serverless database), Langfuse/Langsmith (tracing & debugging), Faker (data generation), Microsoft Presidio (PII detection), embedding models, APIs, and Server-Sent Events (SSE). The project also leverages Claude Code Agent Teams for parallel task execution.

Conclusion

The project demonstrates a viable approach to interacting with powerful cloud LLMs using private data while mitigating privacy risks. The iterative development process, leveraging agent-based workflows, local LLM integration, and a focus on robust entity resolution, has yielded significant progress. While challenges remain, the system shows promise for enabling secure and intelligent access to sensitive information, paving the way for AI agents grounded in private knowledge. The emphasis on proactive technical safeguards, alongside legal agreements, is crucial for building trustworthy and compliant AI systems.