Agentic GraphRAG: AI’s Logical Edge — Stephen Chin, Neo4j

Key Concepts

• Agentic graph RAG (Retrieval Augmented Generation)
• Hallucinations in LLMs (Large Language Models)
• Knowledge graphs
• MCP (Microservice Communication Protocol)
• Cipher query language
• Vector search
• Graph context
• Neo4j
• Langchain
• LangGraph
• Text to Cipher
• Vector search with graph context
• Pre and post filtering of vector results

The Problem with Agentic Systems and LLMs

The presentation begins by highlighting the problem of hallucinations and inaccuracies in agentic systems powered by LLMs. These systems often fail to meet use cases and can produce incorrect or biased results due to the LLM's limitations in reasoning and understanding context.

• Example: The presenter demonstrates this with an example using the OpenAI reasoning API, where the AI fails to correctly answer a question about fitting girls in a classroom, exhibiting biases and incorrect reasoning.
• Key Point: LLMs are good at extrapolating information and language tasks, but they lack true human reasoning and can lead to incorrect business results, especially in critical domains like drug discovery or supply chain management.

Agentic Systems and Knowledge Graphs as a Solution

The presenter suggests that knowledge graphs can help solve the problems of LLMs.

• Agentic Systems: Agentic systems, where multiple LLMs communicate and reason together, can improve the quality of results. However, they often have a monolithic architecture that is hard to maintain and secure.
• MCP as a Solution: MCP can be used as a tool to solve this problem, where agents communicate with each other through MCP.
• Neo4j Tools: Neo4j has built tools on top of MCP, including a Cipher tool for generating Cipher queries, a memory module for agent memory, and MCP on top of cloud APIs.

Graph RAG Architecture and Patterns

The presentation then delves into the architecture and patterns for implementing graph RAG, emphasizing the advantages of using graph context to improve the accuracy and relevance of LLM responses.

• Graph RAG vs. Baseline RAG: Graph RAG offers advantages over baseline RAG by providing more complete and accurate results with a lower rate of hallucinations. Baseline RAG relies on vector similarity, which is not always the same as relevance.
• Typical Pattern: The typical pattern involves first performing a vector search to translate the user's question into vectors, then using these vectors to retrieve relevant nodes from the knowledge graph.
• Architecture: The architecture involves taking in a question, querying either vectors or knowledge graphs, using graph data science or graph analytics for community algorithms and groupings, and feeding the results back to the LLM as context.
• Patterns:
  • Text to Cipher: Using LLMs to generate Cipher queries (done by MCP server). Can be unreliable.
  • Vector Search with Graph Context: Performing a vector search and then using the results to retrieve related nodes in the graph. This is a recommended starting point.
  • Pre and Post Filtering: Filtering vector results to prioritize more relevant items in the context window.

Case Study: CLA

The presenter provides a case study of CLA, a customer who replaced their SAS systems with a graph RAG project.

• Results: CLA achieved 85% employee adoption, processed 2,000 daily queries, and addressed 250,000 employee questions in the first year.

Resources

The presenter recommends the following resources:

• Neo4j Certified Developer Program: A program to demonstrate knowledge of graph technology.
• Neo4j Nodes Conference: An annual conference with free content and sessions.

Q&A Highlights

• Pattern for Search: The pattern for search involves using the LLM for language translation, performing a vector search, and then using the results to retrieve relevant nodes from the graph.
• Embedding and Nodes: Typically, unstructured data is imported into Neo4j, and a node structure is created using LLMs. Text embeddings are then hung off as properties of the nodes.
• MCP Agent for Memory: The presenter does not know the answer to how the MCP agent for memory works, but suggests asking Michael Hunger, whose team built the MCP servers.
• Langchain vs. LangGraph: The presenter suggests using the tool that is best for you and that Neo4j will integrate with everything.

Synthesis/Conclusion

The presentation argues that agentic graph RAG is a promising approach to building more accurate and reliable AI systems. By combining the strengths of LLMs with the structured knowledge of graphs, it is possible to overcome the limitations of LLMs and achieve better results in a variety of applications. The key is to use LLMs for language translation and vector search, and then leverage the graph to provide relevant context and reasoning capabilities.