Exploring AI Frameworks and Models with Jonathan Hooker at RedHat Summit 2025

Key Concepts:

• Agentic AI
• LLMs (Large Language Models)
• LangChain & LangGraph
• OpenShift AI
• vLLM
• OpenAI API compliance
• MCP (presumably Model Control Plane)
• RAG (Retrieval Augmented Generation)
• Vector Store
• Hugging Face
• Model Selection
• Llama 3.2, Llama 4

Building AI Solutions: A Deep Dive with Jonathan from CDW

1. Initial Inspiration and Tool Selection:

• Jonathan from CDW discusses the process of building AI solutions, emphasizing a practical, hands-on approach.
• The journey began with collaboration with CDW's Chief AI Architect, Nathan Cartwright, focusing on industry-standard tools.
• Key tools include:
  • bitHuman: Used for creating realistic AI avatars with accurate facial movements. The avatars are entirely AI-generated, not based on a single person.
  • LangChain and LangGraph: Frameworks for building agentic AI, simplifying the creation of complex AI solutions with a comprehensive toolkit. LangGraph is used to connect data sources.
  • OpenShift AI: A container platform for abstracting and sharing expensive hardware resources like GPUs and HPUs.
  • vLLM: Used for hosting LLMs, chosen for its OpenAI-compliant API.

2. Architecture and Data Integration:

• The architecture follows a front-end/back-end model with REST APIs and WebSockets, adhering to industry best practices for application development.
• Data Sources: Data sources are presented to LangGraph as functions. A function is created to query CDW.com in a specific way, then passed into a LangGraph function to build the AI tool.
• LLM Interaction: LangChain is used to call an LLM hosted on vLLM. vLLM is Red Hat's standardized inference service product.
• API Design:
  • vLLM uses an OpenAI-compliant API (e.g., /v1/chat/completions).
  • The front-end REST API currently uses a simple JSON query (e.g., /requests), but there's a plan to transition it to OpenAI compliance for broader compatibility.

3. MCP and Anthropic's Influence:

• LangGraph is already prepared to connect with an MCP (Model Control Plane) server, aligning with Red Hat's integration of MCP.
• Jonathan reveals that a significant portion (75%) of Hope's (the AI assistant) code was generated using Anthropic's Claude.
• He emphasizes the speed and efficiency of iterating on code with Claude.

4. RAG Implementation and Vector Stores:

• CDW utilizes a RAG model, incorporating embeddings into a vector store.
• Process:
  1. Traditional documents (marketing materials, PowerPoints) are uploaded to a vector store.
  2. These documents are embedded to create context for the AI.
  3. The embedding is used as a data source for Hope, allowing her to answer questions based on the uploaded documents.
• Currently, only marketing documents are used, excluding financial data.

5. Model Selection and Evaluation:

• Model selection is likened to hiring an employee, emphasizing the importance of finding the right fit for specific use cases.
• Hugging Face: Used as a marketplace to search for LLMs, evaluate their training data, and review benchmarks.
• Benchmarking: LLMs are evaluated based on benchmarks and scores to compare their performance.
• Currently, CDW is using Llama 3.2 and plans to transition to Llama 4.
• Cost and size of LLMs are also considered, with smaller models being suitable for tasks like translation or database queries.
• Defining the use case and end goal is crucial for selecting the appropriate LLM.

6. Notable Quotes:

• "Model selection is like interviewing a new employee."
• "Defining your use case first and understanding what your end goal is. And then kind of backing into, who are we going to hire really is a great way to ultimately pick an LLM."

7. Conclusion:

• The conversation highlights the practical steps and considerations involved in building AI solutions from the ground up.
• Key takeaways include the importance of selecting the right tools, designing a robust architecture, leveraging RAG for context, and carefully evaluating LLMs based on specific use cases and performance benchmarks.
• The discussion emphasizes the iterative nature of AI development and the value of collaboration and experimentation.