Wisdom-Driven Knowledge Augmented Generation at Scale - Chin Keong Lam, Patho AI

Key Concepts

• Knowledge
• Knowledge Graph
• KAG (Knowledge Augmented Generation)
• Wisdom
• Decision Making
• Insight
• Feedback Loop
• Multi-Agent System
• Neo4j
• N8N
• RAG (Retrieval Augmented Generation)
• LLM (Large Language Model)

1. Introduction and Background

Ching Kyong Lamb, founder and CEO of PO.AI, introduces the company's background, which started with an SBIR grant from the National Science Foundation to investigate LLMs for drug discovery. PO.AI now focuses on building expert AI systems for large corporations, going beyond simple RAG systems to create AI that can perform research and advisory roles. The talk aims to share insights gained from building these systems.

2. Defining Knowledge and Knowledge Graphs

• Knowledge: Understanding and awareness gained through experience, education, and comprehension of facts and principles.
• Knowledge Graph: A systematic method of preserving wisdom by connecting pieces of information and creating a network of interconnected relationships. It represents the thought process and comprehensive taxonomy of a specific domain of expertise. This is crucial for AI systems that can think and provide advice, not just retrieve data.

3. Introducing KAG (Knowledge Augmented Generation)

• KAG Definition: Enhanced language model that integrates structured knowledge graphs for more accurate and insightful responses.
• KAG vs. RAG: KAG doesn't just retrieve; it understands. It's a smarter, more structured approach than simple RAG.
• Expert Thinking: Experts make decisions based on a specific thinking process, and knowledge graphs are a perfect fit for modeling this.

4. The Wisdom State Diagram

• Core Element: Wisdom: Wisdom is not static; it actively guides decisions and is fueled by other elements.
• Decision Making: Analyzes situations and isn't made in a vacuum.
• Knowledge to Wisdom: Knowledge (books, encyclopedias, Wikipedia) is ingested and needs to be synthesized by the model.
• Insight to Wisdom: Wisdom derives patterns from chaos (e.g., tracking product sentiment from social media).
• Interconnectedness: All nodes relate to one another, enriching the wisdom-storing system.

5. Knowledge, Experience, and Insight

• Knowledge: Tells you what it is (like a pizza recipe).
• Experience: Tells you what worked before (knowing your oven burns the crust).
• Insight: Invents what to try next (adding honey to the crust to caramelize it).
• Feedback Loop: The most important part of the knowledge graph. It learns from itself. Situation informs future wisdom, experience deepens it, and insight sharpens it.

6. Practical Applications of Wisdom

• Leadership: Avoid knee-jerk reactions by learning from feedback.
• Personal Growth: Past mistakes make you wiser.
• Takeaway: Wisdom isn't a trophy; it's a muscle. The more you feed it knowledge, experience, and insight, the more it guides you.

7. Real-World Application: Competitive Analysis

• A client wanted AI to perform competitive analysis, replacing the marketing department's function.
• The same taxonomy of storing wisdom was used.
• A wisdom graph-powered AI chatbot was designed to turn data into strategy.
• The chatbot answers sophisticated questions like, "How do I win against my competitor in this market space?"

8. Mapping the Wisdom State Diagram to a System

• Wisdom Engine: An orchestration agent that makes decisions and advises what the LLM should do based on the current situation.
• Mapping:
  • Decision Making -> Strategy Generator
  • Knowledge -> Market Data
  • Experience -> Past Campaigns
  • Insight -> Industrial Insight Database
  • Situation -> How the product is selling, competitor weaknesses

9. Implementation with N8N

• N8N (a no-code workflow automation tool) was used to implement the state diagram.
• AI Agent Node: A powerful node within N8N that allows driving different models (OpenAI, Anthropic, on-prem models).
• Wisdom Agent: Oversees other agents that perform tasks defined in the state diagram.
• Centralized Graph: Updated by different agents (e.g., the insight agent updates the graph with social media sentiment data).
• Unified Knowledge Graph: Contains the taxonomy that marketing strategists would use to make decisions.

10. Why Use Knowledge Graphs for Competitive Analysis Instead of RAG?

• Capturing Complex Relationships: Knowledge graphs excel at representing complex relationships between entities, leading to deeper contextual understanding.
• Improved Accuracy: Knowledge graphs provide more accurate and relevant information by leveraging structured data and semantic relationships.
• Scalability and Flexibility: Knowledge graphs are scalable and can integrate new data sources and relationships.
• Rich Query Capability: Knowledge graphs support complex queries that traverse multiple relationships, providing richer and more detailed insights (multi-hop questions).
• Enhanced Data Integration: Knowledge graphs can seamlessly integrate diverse data sources (pictures, graphics, videos).

11. Limitations of Vector RAG

• Vector RAG struggles with complex numerical calculations.
• Example: Answering "What is Apple's revenue between 2021 and 2022?" Vector RAG might return passages, while a knowledge graph can provide the precise numerical answer.

12. RAG vs. KAG: When to Use Which

• Simple RAG: For product information queries (using Chroma DB with an LLM agent).
• KAG: For complicated questions like "How can I beat my competition based on my current market share?" (using a knowledge graph with a graph DB, Cypher query, and a RAG trained for multi-hop queries).

13. Knowledge Graph Extraction

• Hybrid Model (Recommended): Use an LLM to extract the graph, then have an expert prune the graph to remove irrelevant relationships.

14. Benchmark Results of KAG

• Accuracy: 91%
• Flexibility: 85%
• Reproducibility: Deterministic
• Traceability
• Scalability

15. Conclusion

By leveraging the structured nature of wisdom and knowledge, we can significantly enhance the quantitative capabilities of KAG systems and provide more accurate and insightful responses to complex queries. Wisdom-driven systems can potentially surpass the intelligence of the initial expert they are meant to serve.

16. Call to Action

Talk to Jesus (from Neo4j) at their booth for help building knowledge graphs. They have an LLM graph RAG stack on GitHub sponsored by Neo4j that allows you to spin up a Docker container and convert text to a graph for pruning.