Multi Agent AI and Network Knowledge Graphs for Change — Ola Mabadeje, Cisco

Key Concepts

• AI-driven Change Management: Using AI agents to reduce failures during network change management.
• Multi-Agent System: A system with multiple AI agents, each tasked with specific functions like impact assessment, testing, and reasoning.
• Network Knowledge Graph: A digital twin of the production network, representing network entities and their relationships in a structured format.
• Open Config: A standardized schema for representing network configurations, facilitating interoperability and understanding by AI agents.
• Agency.org: An open-source collective focused on creating a standard framework for interoperable AI agents.
• Digital Twin: A combination of the knowledge graph and tools for executing tests, simulating the production network environment.
• Extrinsic vs. Intrinsic Metrics: Focusing on metrics that map back to customer use cases (extrinsic) rather than internal system performance (intrinsic).

1. Customer Problem and Solution Overview

• Problem: Customers face challenges with failures during change management in production networks.
• Goal: Reduce failures using AI.
• Approach: Develop an AI-driven system with a natural language interface, a multi-agent system, and a network knowledge graph.
• Incubation Phase: Involves customer interviews, prototyping, A/B testing, and delivering an MVP to production.
• Solution Components:
  • Natural Language Interface: Allows network operations teams and systems (e.g., ServiceNow) to interact with the system using natural language.
  • Multi-Agent System: Consists of specialized agents for tasks like impact assessment, testing, and reasoning about potential failures.
  • Network Knowledge Graph (Digital Twin): A representation of the production network, including a knowledge graph and tools for testing.

2. Building the Network Knowledge Graph

• Challenge: Representing complex, multi-vendor networks with diverse data formats (Yang, JSON, etc.) in a way that AI agents can understand.
• Data Sources: Controllers, devices, agents, configuration management systems, streaming telemetry, configuration files.
• Data Considerations: Data formats (Yang, JSON), data delivery methods (streaming telemetry, configuration files).
• Product Requirements:
  • Multimodal Flexibility: Support for key-value pairs, JSON files, and relationships between network entities.
  • Performance: Instant access to node information.
  • Operational Flexibility: Consolidate data into a single schema framework.
  • Graph RAG (Retrieval-Augmented Generation) Capabilities: Vector indexing for semantic searches.
  • Ecosystem Stability: Easy integration with customer systems and support for multiple vendors.
• Technology Evaluation:
  • Considered Neo4j, ArangoDB, and other open-source tools.
  • Chose ArangoDB due to existing use cases in the security space (recommendation systems).
  • Still exploring Neo4j for future use cases.
• Knowledge Graph Architecture:
  • Ingestion Service: ETL process to transform data from various sources into the Open Config schema.
  • Open Config Schema: A standardized schema for representing network configurations, facilitating communication with AI agents.
  • Layered Structure: Organizes network entities into layers (e.g., raw configuration, data plane, control plane) to optimize agent queries.

3. Agentic Layer and Open Standards

• Agency.org Collective: An open-source initiative to create a standard framework for interoperable AI agents.
• Goals of Agency.org:
  • Enable agents from different sources to communicate without requiring extensive integration efforts.
  • Define a schema framework for agent skills and capabilities.
  • Establish a directory for storing agent information.
  • Develop methods for composing agents at semantic and synthetic layers.
  • Provide tools for observing agent processes.
• Key Components of Agency.org Framework: Identity, schema, directory, composition (semantic and synthetic), observation.
• Integration with Existing Protocols: Supports MCP, A2A, and other popular protocols.
• Specific Agents in the Application:
  • Assistant Agent: Acts as a planner and orchestrates tasks across other agents.
  • Query Agent: Interacts directly with the knowledge graph.
  • Other agents based on React reasoning loops.
• Fine-tuning the Query Agent:
  • Initial attempts to use RAG for querying the knowledge graph were inefficient.
  • Fine-tuned the query agent with schema information and example queries.
  • Resulted in a significant reduction in token consumption and query response time.

4. Demo: Firewall Rule Change Scenario

• Scenario: A network engineer needs to change a firewall rule to accommodate a new server.
• Workflow:
  1. Engineer submits a ticket in ServiceNow.
  2. The system ingests the ticket information and uses AI agents to process it.
  3. The assistant agent synthesizes the ticket information.
  4. An agent creates an impact assessment and attaches it to the ServiceNow ticket.
  5. An agent generates a test plan with a list of tests and expected results, attaching it to the ticket.
  6. The engineer creates a pull request for the configuration file in a GitHub repo.
  7. The execution agent pulls the configuration file from GitHub and takes a snapshot of the network from the knowledge graph.
  8. The execution agent runs the tests in a digital twin environment.
  9. The execution agent generates a report with the test results and recommendations, attaching it to the ticket.
• Digital Twin in Action: The demo showcases how the knowledge graph and testing tools (e.g., Batfish, Routnet) are used to simulate the network environment and validate changes before they are implemented in production.

5. Evaluation and Key Takeaways

• Evaluation Metrics: Focus on extrinsic metrics that map back to customer use cases.
• Key Building Blocks: The knowledge graph and the open framework for building agents are critical for building a scalable system.
• Ongoing Learning: The system is an MVP, and the team is continuously learning and improving it.

Conclusion

Ola Mabad's presentation details Cisco's approach to using AI to improve network change management. The core of the solution lies in a multi-agent system interacting with a network knowledge graph (digital twin) built using ArangoDB and adhering to the Open Config schema. The presentation highlights the importance of open standards, as demonstrated by Cisco's involvement with Agency.org, and the need for practical evaluation metrics tied to customer value. The demo illustrates how these components work together to automate and improve the change management process, reducing failures and improving efficiency. The key takeaway is that a well-structured knowledge graph and an open, interoperable agent framework are essential for building scalable and effective AI-driven network management solutions.