Practical GraphRAG: Making LLMs smarter with Knowledge Graphs — Michael, Jesus, and Stephen, Neo4j

Key Concepts

• Graph RAG (Retrieval Augmented Generation)
• Knowledge Graph Construction
• Lexical Graph
• Domain Graph
• Entity Extraction
• Graph Algorithms for Enrichment
• Graph Retrieval (Local & Global Search)
• Vector Search
• Full-Text Search
• Agentic Approach
• LLMs (Large Language Models)
• Relevance vs. Similarity
• Contextual Information
• Explainability
• Neo4j

The Case for Graph RAG

The primary challenge with using LLMs is their lack of enterprise domain knowledge, inability to verify or explain answers, susceptibility to hallucinations, and ethical/data bias concerns. Graph RAG aims to address these issues by providing domain-specific knowledge, accurate contextual answers, and explainability. The core problem is identified as a data problem, requiring good data to power the system.

Traditional RAG systems using vector databases fall short because they lack a complete dataset, pulling only a fraction of the information via vector similarity algorithms. These systems often lack robustness, scalability, and the ability to provide relevant results. Vector similarity does not equate to relevance, and explaining the system's output is difficult.

Graph RAG brings knowledge, context, and environment to LLMs. It leverages knowledge graphs, which are collections of nodes, relationships, and properties, to provide a logical reasoning layer. This approach offers better relevancy, more context through graph closeness algorithms, explainability due to structured data, and the ability to implement security and role-based access.

Research and Industry Adoption

Microsoft Research's initial graph RAG paper demonstrated improved results and reduced token costs. Subsequent research has further explored the benefits of graph RAG. A data.world study showed a three-times improvement in accuracy compared to RAG on SQL databases. Gartner's hype cycle for 2024 indicates that graph RAG is gaining traction and breathing life into the AI ecosystem.

Industry leaders are adopting graph RAG, with organizations like LinkedIn using it for customer support. LinkedIn's research paper showed that using a knowledge graph improved the quality and reduced the response time for customer support, with a 28.6% reduction in median per-issue resolution time.

Knowledge Graph Construction

Graph RAG involves two main phases: knowledge graph construction and graph retrieval. Knowledge graph construction involves multiple steps:

1. Unstructured Information to Lexical Graph: Initially, unstructured information is structured into a lexical graph, representing documents, chunks, and their relationships.
2. Entity Extraction: Entities and relationships are extracted from the lexical graph using LLMs with graph schema.
3. Graph Enrichment: The graph is enriched with graph algorithms like PageRank and community summarization.

Higher quality outputs require more effort at the beginning. Structuring unstructured documents results in high-quality, structured information that can be used to extract contextual information for queries.

Lexical Graph Details

Lexical graphs represent documents and their elements (chunks, chapters, sections, paragraphs). Paragraphs are considered semantically cohesive units suitable for creating vector embeddings. Relationships between chunks are established using vector or text similarity, creating a K-nearest neighbor or similarity graph with weighted scores. These relationships are used in the retrieval phase to find related chunks by document, temporal sequence, or similarity.

Entity Extraction Details

LLMs enhance entity extraction with their multi-language understanding and flexibility. A graph schema and instruction prompt are provided to the LLM, along with text. Large context windows allow for processing of 10,000 to 100,000 tokens. Existing ground truth data (e.g., products, genes, partners, clients) can be passed as part of the prompt for entity recognition. Additional facts and information are stored as part of relationships and entities. Existing knowledge graphs (e.g., CRM data) can be connected to enrich the extracted data.

Graph Enrichment Details

Graph algorithms are used for enrichment, such as clustering on the entity graph to generate communities. LLMs can generate summaries across these communities, identifying cross-document topics. This identifies topics recurring across different documents, providing a broader perspective than individual document analysis.

Graph Retrieval

In graph retrieval, an initial index search (vector search, full-text search, hybrid search, spatial search) is performed to find entry points in the graph. Relationships are then followed up to a certain degree or relevancy to fetch additional context. This context can come from the user question or external user context (e.g., department). The retrieved context is returned to the LLM to generate the answer. Modern LLMs are trained on graph processing and can handle node-relationship-node patterns. Graph algorithms like clustering, link prediction, and PageRank can be used to enrich the data.

Practical Examples and Tools

Several tools and libraries are available for graph RAG:

• Knowledge Graph Builder: A tool for extracting data from PDFs, YouTube transcripts, local documents, web articles, and Wikipedia articles into a graph. It allows for the use of different LLMs and the provision of graph schemas to guide extraction.
• Neo Converse: A tool able to output text, charts, and network visualizations.
• Agentic Approach: An approach that breaks down user questions into individual tasks, extracts parameters, and runs individual tools in sequence or in a loop to return data.
• Graph Python Package: A package encapsulating knowledge graph construction and retrieval into one package.

Demo of Knowledge Graph Builder

The demo showcased the Knowledge Graph Builder, where information from Wikipedia pages, YouTube videos, and articles was uploaded. The tool extracted data from a DeepMind Wikipedia article, creating a connected knowledge graph of entities (companies, locations, people, technologies). Different retrievers (vector, graph, full-text, entity) were used to run graph RAG. The results showed the sources used, the chunks retrieved, and the entities retrieved by the graph retriever, providing a richer response.

Agentic Approach Example

The agentic approach involves configuring domain-specific retrievers that run individual Cypher queries. An agentic loop uses these tools, performing graph RAG with each tool, taking the responses, and doing deeper tool calls. This approach breaks down the user question into individual tasks, extracts parameters, and runs individual tools in sequence or in a loop to return the data.

Conclusion

Graph RAG addresses the limitations of LLMs and traditional RAG systems by leveraging knowledge graphs to provide domain-specific knowledge, accurate contextual answers, and explainability. The process involves constructing a knowledge graph through structuring unstructured data, extracting entities, and enriching the graph with algorithms. Graph retrieval then uses various search methods to find entry points and retrieve relevant context for the LLM. Tools and libraries are available to facilitate graph RAG implementation. The agentic approach further enhances the process by breaking down complex queries into smaller tasks and using specialized tools.