AI Engineer World’s Fair 2025 — GraphRAG

Key Concepts

• RAG (Retrieval-Augmented Generation): A framework for enhancing language models by retrieving information from an external knowledge source and using it to generate more accurate and context-aware responses.
• Knowledge Graph: A network representing relationships between entities (people, places, concepts, events). It consists of nodes (entities) and edges (relationships).
• Graph RAG: A RAG system that utilizes a knowledge graph as its external knowledge source.
• Hybrid RAG: A RAG system that combines both semantic vector databases and knowledge graphs.
• Triplets: A representation of a relationship in a knowledge graph, consisting of two entities and the relationship between them (Entity 1, Relationship, Entity 2).
• Ontology: A formal representation of knowledge as a set of concepts within a domain and the relationships between those concepts.
• Semantic Vector Database: A database that stores vector embeddings of text chunks, allowing for semantic similarity search.
• Vector Embeddings: Numerical representations of text that capture its semantic meaning.
• LLM (Large Language Model): A deep learning model trained on a massive amount of text data, capable of generating human-quality text.
• Cool Graph: An Nvidia library for accelerating graph searches.
• Ragas: A Python library for evaluating RAG pipelines.
• Faithfulness: A metric for evaluating whether the generated response is consistent with the retrieved context.
• Answer Relevancy: A metric for evaluating whether the generated response is relevant to the query.
• Precision/Recall: Metrics for evaluating the accuracy of information retrieval.
• Llama: A family of open-source large language models developed by Meta.
• Laura: Low-Rank Adaptation, a fine-tuning technique for LLMs.
• NetworkX: A Python library for creating, manipulating, and studying the structure, dynamics, and functions of complex networks.
• KAG (Knowledge Augmented Generation): An enhanced language model that integrates a structured knowledge graph for more accurate and insightful responses.
• Fusion and Decoder: A technique that uses knowledge graphs to understand the relationships between retrieved passages, improving efficiency and reducing hallucination.
• Agentic Memory: A memory system for AI agents that focuses on temporal and relational reasoning, tracking state changes over time.
• Graffiti: Zep's open-source framework for building real-time dynamic temporal graphs.
• MCP (Memory Component Protocol): A protocol for managing and accessing memory components in AI systems.
• A2A (Agent-to-Agent Communication): A framework for enabling communication and collaboration between AI agents.
• Open Config: A standardized data modeling language for network devices.

Graph RAG vs. Semantic RAG

• Rag is not dead: Rag is still a relevant technology, but agents are taking over.
• Rag vs Agents: If Rag can solve the problem, agents are not needed and vice versa.
• Semantic RAG:
  • Breaks documents into chunks, converts them into vector embeddings, and stores them in a vector database.
  • Uses semantic similarity to retrieve relevant chunks based on a query.
  • Does not explicitly capture relationships between entities.
• Graph RAG:
  • Creates a knowledge graph by extracting entities and relationships from documents.
  • Captures information between entities in more detail, providing a comprehensive view of knowledge.
  • Can organize data from multiple sources.
  • Exploits relationships between entities to retrieve more relevant information.
  • Can perform multi-hop queries to traverse the graph and gather context from multiple nodes.
• Hybrid RAG:
  • Combines both semantic vector databases and knowledge graphs.
  • Leverages the strengths of both approaches.

Creating a Graph RAG System

1. Data Processing:
   • Process data to create a knowledge graph.
   • Better data processing leads to a better knowledge graph and better retrieval.
2. Graph Creation/Semantic Embedding:
   • Create triplets (Entity 1, Relationship, Entity 2) to define relationships between entities.
   • Use LLMs and prompt engineering to extract triplets from unstructured documents.
   • Define an ontology to guide the LLM in extracting relevant information.
   • Create a semantic vector database by chunking documents, converting them into vector embeddings, and storing them in a vector database.
   • Consider chunk size and overlap to preserve context between chunks.
3. Inferencing:
   • Query the knowledge graph to retrieve relevant nodes and relationships.
   • Use different retrieval strategies, such as single-hop or multi-hop queries.
   • Optimize the depth of the graph traversal to balance context and latency.
4. Offline vs. Online:
   • Data processing and knowledge graph creation are offline processes.
   • Querying and response generation are online processes.

Retrieval Strategies

• Single-Hop Retrieval: Retrieves nodes directly connected to the initial query node.
• Multi-Hop Retrieval: Traverses the graph through multiple nodes to gather more context.
• Optimization: Balance the depth of the graph traversal with latency requirements.
• Acceleration: Use libraries like Cool Graph to accelerate graph searches.

Evaluating Performance

• Ragas Library:
  • Evaluates RAG workflows end-to-end.
  • Evaluates the response, retrieval, and query.
  • Provides flexibility to use custom LLMs for evaluation.
• Evaluation Parameters:
  • Faithfulness, answer relevancy, precision, recall, helpfulness, collectiveness, coherence, complexity, verbosity.
• Lanimotron 340B Reward Model:
  • A model specifically trained to evaluate the responses of other LLMs.
  • Judges responses based on five parameters.

Strategies for Optimization

• Knowledge Graph Creation:
  • Fine-tune an LLM model to improve the quality of triplets.
  • Improve data processing by removing irrelevant characters and noise.
• Data Cleaning:
  • Remove rejects, apostrophes, and other irrelevant characters.
• Output Length:
  • Reduce the length of the output to improve accuracy.
• Fine-Tuning:
  • Fine-tune the Llama 3.3 model to improve performance.
• Acceleration:
  • Use libraries like Cool Graph to accelerate graph searches.

Choosing the Right Approach

• Data Structure:
  • Graph-based systems are well-suited for structured data (e.g., retail, FSI, employee databases).
  • If unstructured data can be transformed into a good knowledge graph, it may be worthwhile to experiment with graph-based systems.
• Application and Use Case:
  • If the use case requires understanding complex relationships, graph-based systems may be more appropriate.
  • Consider the computational cost of graph-based systems.

KAG (Knowledge Augmented Generation)

• Definition: Enhanced language model by integrating structured knowledge graph for more accurate and insightful responses.
• Difference from RAG: KAG understands, not just retrieves.
• Wisdom Note: The core of KAG, actively guiding decisions and fused by other elements.
• State Diagram: A visual representation of the KAG process, showing the relationships between wisdom, decision-making, situation analysis, knowledge, experience, and insight.
• Feedback Loop: An important part of the knowledge graph, allowing it to learn from itself.
• Application: Competitive analysis, where KAG can be used to turn data into strategy.
• Implementation: Using a no-code platform like n8n to implement the state diagram with AI agent nodes.
• Fusion and Decoder: A technique that uses knowledge graphs to understand the relationships between retrieved passages, improving efficiency and reducing hallucination.

Agentic Memory

• Importance: Retaining dynamic data from conversations and business applications to provide context-aware responses.
• Limitations of RAG: Lacks native temporal and relational reasoning.
• Knowledge Graphs for Memory: Can define explicit relationships and model causality.
• Graffiti: Zep's open-source framework for building real-time dynamic temporal graphs.
• Temporal Awareness: Graffiti extracts and tracks multiple temporal dimensions for each fact.
• Graph Relational: Graffiti defines explicit relationships between entities.
• Business Domain Modeling: Graffiti allows developers to model their business domain on the graph.
• Domain-Aware Memory: Modeling memory after the business domain to improve relevance and accuracy.

Multi-Agent Framework for Network Analysis

• Customer Problem: Reducing failures in production during change management.
• Solution: A multi-agent system with a natural language interface and a network knowledge graph.
• Network Knowledge Graph: A digital twin of the production network, representing devices, configurations, and relationships.
• Data Sources: Controllers, systems, devices, configuration management systems.
• Data Formats: Yang, JSON.
• Schema: Open Config.
• Knowledge Graph Layers: Raw configuration, data plane, control plane.
• Agentic Layer: A set of agents tasked with specific functions, such as impact assessment, testing, and reasoning.
• Query Agent: An agent that interacts directly with the knowledge graph.
• Fine-Tuning: Fine-tuning the query agent with schema information and example queries to improve performance.
• Evaluation Metrics: Extrinsic metrics that map back to the customer's use case.
• Open Framework for Agents: A system that allows agents from across the world to communicate with each other without heavy lifting.

Graph RAG in the Legal Industry

• Application: Finding cases early and supporting lawyers in litigation.
• Challenges:
  • Lawyers require perfect accuracy and creative arguments.
  • Information is often kept in separate places.
• Solutions:
  • Using graphs to structure and augment information from legal discovery.
  • Building multi-agent systems to automate case research and report generation.
• Case Research Process:
  • Scrape the web.
  • Qualify leads.
  • Generate reports.
• Graph Structure:
  • Extensible schema that can be updated and queried.
  • Nodes representing individuals, products, ingredients, and concentrations.
  • Edges representing relationships between entities.
• Benefits:
  • Improved accuracy and efficiency.
  • Ability to manage state and track information over time.
  • Personalized service for lawyers.
• Future Directions:
  • Finding lawsuits earlier.
  • Compensating harm.
  • Using AI to improve the entire legal process.

Synthesis/Conclusion

The transcript highlights the growing importance of graph RAG as a powerful approach for enhancing language models with structured knowledge. It emphasizes the limitations of traditional semantic RAG and showcases the benefits of using knowledge graphs to capture relationships, improve accuracy, and enable explainability. The speakers provide practical guidance on building graph RAG systems, including data processing, graph creation, retrieval strategies, and performance evaluation. They also discuss various use cases across different industries, such as finance, network analysis, and legal, demonstrating the versatility and potential of graph RAG. The key takeaway is that graph RAG is not just about retrieving information but about understanding and reasoning with it, leading to more insightful and actionable results.