Context Engineering: Connecting the Dots with Graphs — Stephen Chin, Neo4j

Key Concepts

• Context Engineering: The practice of strategically providing relevant information to AI models to improve their performance and output quality.
• Retrieval Augmented Generation (RAG): A technique that augments LLMs with external knowledge retrieved from a data source, improving accuracy and reducing hallucinations.
• Graph Technology: A class of technologies that represent data as nodes and relationships, enabling structured and interconnected information.
• Knowledge Graphs: A specific type of graph that represents entities (nodes) and their relationships, providing structured knowledge and context.
• Graph RAG: An advanced RAG approach that leverages knowledge graphs as part of the retrieval process, offering richer context and explainability.
• Agentic Memory: The ability of AI agents to store, retrieve, and utilize past interactions and learned information for improved performance and continuity.
• Cypher: A declarative graph query language for Neo4j and other graph databases.
• Embeddings: Numerical representations of data (e.g., text, nodes) that capture semantic meaning and enable similarity searches.

Connecting the Dots with Graph Technology for Enhanced AI Applications

This session explores how graph technology can revolutionize context engineering, improve retrieval patterns, and enhance agentic memory in AI applications. The speaker, Stephen Chin, VP of Developer Relations at Neo4j, argues that moving beyond basic prompt engineering to a more sophisticated approach of providing structured and relevant context is crucial for unlocking the full potential of AI.

The Evolution of AI Interaction: From Prompt Engineering to Context Engineering

Traditionally, AI development has involved "prompt engineering," focusing on crafting clever phrasing to elicit desired responses from AI models. However, this approach is evolving towards "context engineering," which emphasizes dynamically feeding AI with a broader and more structured scope of information. This shift aims to:

• Enhance Agent Capabilities: Provide agents with more context and information to perform tasks collaboratively.
• Create Dynamic Models: Develop more adaptable and goal-driven AI applications.
• Curate Relevant Information: Selectively feed information pertinent to specific domains, especially crucial in enterprise settings with complex business contexts.
• Improve Signal-to-Noise Ratio: Structure input to extract more meaningful signal from the vast amount of data processed by LLMs, addressing the issue of large context windows with limited attention focus.
• Empower Information Architects: Transition developers from "prompt engineers" to "information architects" who build the AI's context for superior results.

The Scope of Context Engineering

Context engineering encompasses several key areas:

• Prompt Engineering: Designing effective prompts with the right instructions, information, and grounding for the AI.
• Retrieval Augmented Generation (RAG): Utilizing techniques like RAG to pull data from various sources, including enterprise and business contexts, and supply it as additional information to the AI.
• State and History Management: Incorporating both short-term memory (for current tasks and collaboration) and long-term memory (for conversation state, history, and effective long-term operations).
• Structured Output: Organizing AI output in a meaningful way for integration with other applications and tools.

The Central Role of Memory in Context Engineering

Memory is identified as a core component of context engineering, with two primary categorizations:

• Short-Term Memory: The information the AI is actively using for the current task. This involves compressing relevant information and prioritizing it within the search window. It also includes integrating tool results without overwhelming the context window.
• Long-Term Memory: Information learned over extended conversations, including episodic memories. This requires understanding the semantic and structural meaning of past interactions to guide AI instructions, procedures, and operations.

Effective memory management helps elevate relevant context within the context window, fill information gaps, and mitigate issues like hallucinations and poor results. The quality of AI responses is directly proportional to the quality of data provided.

Leveraging Knowledge Graphs for Structured Context

Knowledge graphs are presented as a powerful solution for providing structured information to AI, bridging the gap between AI's generative capabilities and the need for organized data.

• Construction: Knowledge graphs are built with nodes representing entities (people, places, events, things) linked by relationships that define their connections.
• Readability: They are easily interpretable by both humans and LLMs, serving as an organizing concept and a way to understand AI's decision-making process.
• Digital Twin Analogy: Knowledge graphs can act as digital twins for organizations, supply chains, or processes.
• Components: A basic knowledge graph consists of nodes (entities), relationships, and properties attached to nodes. Embeddings can also be encapsulated on nodes for vector lookups.
• Benefits: They provide knowledge context and enrichment, complementing LLMs' language reasoning and creativity for powerful combined applications.

Graph RAG: Enhancing Retrieval with Knowledge Graphs

Graph RAG is an advanced retrieval pipeline that integrates graphs into the retrieval process, offering significant advantages over traditional vector similarity searches.

• Process: A user query goes to an LLM, which then performs a search. If relevant information exists, a query is sent to a knowledge graph. The retrieved graph information is passed as additional context to the LLM for an enriched answer.
• Advantages:
  • Increased Relevance: Provides richer context beyond simple vector similarity by incorporating relationships, community groupings, and factual, structured knowledge.
  • Explainability: Allows for understanding what the LLM is doing by visualizing the part of the knowledge graph used.
  • Evolvability: Knowledge graphs can be updated and evolved over time.
  • Role-Based Access: Enables overlays for access control, ensuring sensitive information is only accessible to authorized users.
• Explainable AI: Knowledge graphs facilitate explainable AI by storing user and agent interactions, visualizing conversation flows, and analyzing agent system context for performance improvements.

Demo 1: Graph RAG with LLM Knowledge Graph Builder

This demo showcases a practical application of Graph RAG using Neo4j Aura and the open-source LLM Knowledge Graph Builder.

• Setup: A Neo4j Aura instance was used, populated with data from a supply chain document and a VEX (Vulnerability Exploitability eXchange) document.
• LLM Knowledge Graph Builder: This open-source web application allows users to upload files (e.g., supply chain documents, VEX files) which are then processed by an LLM to build a knowledge graph.
• VEX Document Analysis: The VEX document contained information about vulnerabilities in libraries like Jackson, including affected versions and remediation steps.
• Two-Pass Retrieval Process:
  1. Vector Lookup: A similarity search is performed to find related nodes in the knowledge graph.
  2. Contextual Expansion: Nodes related to the initial results are identified and passed as context to the LLM.
• Query Example: The LLM was queried about vulnerabilities in the "Jackson library."
• Results: The LLM successfully identified an XML injection attack in the Jackson databind library, providing details on affected versions and fixes, all sourced from the knowledge graph. When queried about a non-existent vulnerability (Jasper library), the LLM correctly returned no response. This demonstrates the LLM's ability to ground its answers in the provided knowledge graph and refuse to answer questions outside its scope.

Graph Technology for Agentic Memory Retrieval

Graphs are also highly effective for memory retrieval in AI agents.

• Graph Memory Retrieval: This involves using graph memory retrieval tools and an open-source MCP server for graph traversals.
• Querying Capabilities: Graphs can be queried not only for knowledge but also for vectors and through graph data science algorithms (e.g., community groupings, k-nearest neighbors).
• Agent Loop Integration: Retrieved information from graphs can be passed as additional context for short-term or long-term memory within the agent loop.
• Memory Graph Structure: A memory graph can capture knowledge as entities and relationships, with nodes possessing properties like text, embeddings, time, and location.
• Vector-Based Semantic Search: Embeddings on graph nodes enable semantic search through projections into vector space.
• Hybrid Approach: Combining vector embeddings with graph algorithms (k-NN, community groupings, PageRank) allows for more sophisticated querying and bubbling up of relevant results.

Demo 2: Agentic Traversal with Claude Code and Neo4j MCP Server

This demo illustrates how an LLM (Claude) can leverage a knowledge graph through an MCP server for detailed information retrieval.

• Setup: The same knowledge graph from Demo 1 was used, connected to Claude via the Neo4j MCP (Multi-Channel Protocol) server.
• Query Example: The LLM was asked about the "Jackson vulnerability" based on the graph database.
• Multi-Step Process:
  1. Schema Retrieval: The LLM first retrieves the graph schema to understand its structure and relationships.
  2. Cypher Queries: Based on the schema, the LLM generates and executes multiple Cypher queries to gather information about the vulnerability.
  3. Text Chunk Retrieval: Additional context is retrieved from text chunks associated with the relevant nodes.
• Results: Claude provided a highly detailed response, including the CV number, affected vulnerability, attack type, severity, technical description, remediation versions, and advisory information. This level of detail surpasses the previous demo, highlighting the power of agentic traversal and multi-step retrieval.
• Comparison: The agentic multi-step MCP retrieval approach is significantly more powerful for generating detailed and comprehensive responses compared to simpler retrieval methods.

Resources for Further Learning

The session concludes by pointing to several valuable resources for those interested in learning more about graph technology and its applications:

• Graph Academy: A free resource offering courses on Cypher queries, Graph RAG (with Python and TypeScript examples), and advanced graph concepts.
• Nodes AI 2026: A free online virtual conference focused on AI, with AI-focused sessions. The Call for Papers is open.
• graphrag.com: A community site providing the latest research on graph approaches, how-to guides, and conceptual information for implementing Graph RAG.

Conclusion

The session emphasizes that graph technology is a critical tool for enhancing AI applications by providing structured context, improving retrieval accuracy, and enabling sophisticated agentic memory. By moving beyond basic prompt engineering and embracing context engineering powered by knowledge graphs, developers can build more intelligent, reliable, and explainable AI systems, effectively "connecting the dots" to solve complex problems.