Stanford CS230 | Autumn 2025 | Lecture 8: Reinforcement Learning

Key Concepts

• Augmenting LLMs: Enhancing the capabilities of base Large Language Models beyond their pre-trained knowledge.
• Prompt Engineering: Designing effective prompts to elicit desired outputs from LLMs.
• Chain of Thought (CoT): A prompting technique that encourages LLMs to think step-by-step.
• Prompt Chaining: Breaking down complex tasks into a sequence of smaller, interconnected prompts.
• Few-Shot Prompting: Providing LLMs with a few examples within the prompt to guide their responses.
• Fine-tuning: Modifying the weights of an LLM to specialize it for a particular task.
• Retrieval Augmented Generation (RAG): Integrating external knowledge sources with LLMs to improve accuracy and provide sourcing.
• Agentic AI Workflows: Creating autonomous systems that perform multi-step tasks using LLMs, tools, and memory.
• Vector Databases: Databases optimized for storing and retrieving high-dimensional vector embeddings.
• Model Context Protocol (MCP): A protocol for efficient communication between LLMs and external services.
• Evaluation (Eval): Methods for assessing the performance and quality of LLM applications.
• Multi-Agent Systems: Systems composed of multiple interacting agents, often for parallelism or reusability.
• Fuzzy Engineering: Designing software that handles unstructured and dynamic inputs, contrasting with deterministic engineering.
• LLM Traces: Logs that provide visibility into the internal workings of LLM systems for debugging.

Challenges and Opportunities for Augmenting LLMs

The lecture begins by identifying limitations of using base pre-trained LLMs, such as GPT-3.5 Turbo or GPT-4. These limitations include:

• Domain Knowledge Gaps: LLMs may lack specialized knowledge for niche applications (e.g., autonomous farming, medical diagnosis).
• Outdated Information: Models are trained on data up to a certain point and cannot access real-time information or new trends (e.g., new slang words like "kof").
• Lack of Precision for Narrow Tasks: While broad, LLMs might not perform adequately on highly specific, well-defined tasks requiring high precision.
• Difficulty in Control: LLMs can be unpredictable and generate controversial or undesirable content, as seen with Microsoft's Tay chatbot or debates about political bias in LLMs.
• Inconsistencies in Style and Format: LLMs may not adhere to specific stylistic or formatting requirements crucial for domains like legal writing.
• Limited Context Handling: The context window of LLMs, even for advanced models, is finite (e.g., around 200,000 tokens, equivalent to two books), making it difficult to process very large documents or complex data. The "needle in a haystack" problem highlights the difficulty LLMs face in recalling specific facts from large contexts.
• Hallucinations: LLMs can generate factually incorrect information, which is particularly problematic in high-stakes fields like medicine or education.
• Lack of Sourcing: Base LLMs often fail to provide sources for their information, making it difficult to verify claims.

These challenges present opportunities for augmenting LLMs through various techniques, which can be broadly categorized into improving the foundation model itself (e.g., moving from GPT-3.5 to GPT-4) or engineering how the LLM is leveraged. This lecture focuses on the latter.

Prompt Engineering

Prompt engineering is presented as the first line of optimization for LLMs. A study involving BCG consultants demonstrated that prompt engineering training significantly improved performance compared to using LLMs without training. The study also identified two interaction styles:

• Centaurs: Delegate larger tasks to the AI and return to review.
• Cyborgs: Work in a rapid, back-and-forth manner with the AI.

Basic Prompt Design Principles:

• Specificity: Instead of a generic "Summarize this document," a more effective prompt includes details like "Summarize this 10-page scientific paper on renewable energy in five bullet points focusing on key findings and implications for policy makers." This specifies the format, length, and focus.
• Role-Playing: Instructing the LLM to "act as a renewable energy expert giving a conference at Davos" can improve output quality.
• Chain of Thought (CoT): Encouraging the model to "think step by step" by breaking down the task into explicit steps (e.g., "Step one: Identify the three most important findings. Step two: Explain how key each finding impacts renewable energy policy."). This technique, popularized in a 2023 paper, is crucial for controlling LLMs.
• Prompt Templates: Reusable prompt structures that can be personalized with specific data. For example, a template for a career assessment tool could incorporate user metadata like name, role, and preferred language.
• Zero-Shot vs. Few-Shot Prompting:
  • Zero-Shot: The LLM is asked to perform a task without any examples.
  • Few-Shot: The prompt includes a few examples to guide the LLM's understanding and alignment with the desired output. For instance, classifying the tone of a product review can be improved by providing examples of positive, negative, and neutral classifications. This is a quick way to align an LLM without modifying its parameters.

Prompt Chaining:

This technique involves breaking down a complex task into a sequence of independent prompts, allowing for better debugging and optimization.

• Example: A customer support response workflow can be broken down into:
  1. Extract Key Issues: Identify concerns from a customer review.
  2. Draft Outline: Create an outline for a professional response based on the identified issues.
  3. Write Full Response: Generate the final response based on the outline.
• Benefits: Enables isolated testing and improvement of each step, making it easier to pinpoint and fix issues. It also allows for tracking intermediate outputs to understand performance gains.
• Latency Consideration: Chaining can introduce latency, which needs to be managed for real-time applications.

Testing Prompts:

• Manual Rating: Humans rate the outputs of different prompts or workflows.
• Automated Testing Platforms: Tools like Prompt Fu can help automate prompt testing.
• LLM Judges: Using LLMs to evaluate the quality of other LLM outputs. This can involve:
  • Pair-wise Comparison: An LLM judge compares two outputs and determines which is better.
  • Single Answer Grading: An LLM judge rates a single output based on a rubric.
  • Reference-Guided Comparison: LLM judges use a rubric and reference examples to evaluate outputs.

Fine-tuning

Fine-tuning involves modifying an LLM's weights to specialize it for a task. However, the speaker expresses reservations due to:

• Data Requirements: Typically requires substantial labeled data.
• Overfitting: Fine-tuned models may overfit to specific data, losing general utility.
• Time and Cost Intensive: The process is resource-heavy, and by the time it's completed, newer, better base models may be available.
• Example: A humorous example of a fine-tuned Slack bot that adopted a conversational, non-compliant persona instead of performing its intended task.

Fine-tuning is still valuable for tasks requiring repeated high-precision output or when general-purpose LLMs struggle with domain-specific language.

Retrieval Augmented Generation (RAG)

RAG addresses several LLM limitations by integrating external knowledge sources.

• How it Works:
  1. Knowledge Base: A collection of documents (e.g., PDFs, databases).
  2. Embedding: Documents are converted into lower-dimensional vector representations using embedding models.
  3. Vector Database: These embeddings are stored in a specialized database for efficient retrieval.
  4. Query Embedding: User queries are also embedded.
  5. Retrieval: The system finds the most relevant document embeddings based on similarity to the query embedding.
  6. Augmented Prompt: The retrieved document snippets are added to the user's original query as context.
  7. LLM Generation: The LLM generates an answer based on the augmented prompt.
• Benefits:
  • Accuracy and Up-to-date Information: Answers are grounded in external, potentially up-to-date data.
  • Sourcing: RAG allows for citing the sources of information.
  • Targeted Customization: Enables customization without retraining the model.
  • Reduced Hallucinations: By grounding responses in retrieved documents.
• Advanced RAG Techniques:
  • Chunking: Breaking down large documents into smaller, manageable chunks for embedding.
  • Hypothetical Document Embeddings (HyDE): Generating a hypothetical document from the user query to improve retrieval accuracy, especially when query language differs from document language.

Agentic AI Workflows

Agentic AI workflows enable LLMs to perform multi-step, autonomous tasks. Andrew Ng coined the term to distinguish these complex workflows from simpler LLM applications.

• Core Components of an Agent:
  • Prompts: The instructions given to the LLM.
  • Memory: Stores information about the user and past interactions. This can include:
    • Working Memory: For fast access to recent, highly relevant information.
    • Archival Memory: For long-term storage of less frequently accessed information.
  • Tools: APIs or functions that the agent can use to interact with the external world (e.g., flight search API, database lookup). LLMs are adept at reading API documentation.
• Degrees of Autonomy:
  • Least Autonomous: Hardcoded steps and tools.
  • Semi-Autonomous: Hardcoded tools, but the agent determines the steps.
  • Most Autonomous: The agent decides steps, can create tools, and potentially write code.
• Model Context Protocol (MCP): An alternative to direct API calls, MCP facilitates more efficient and scalable communication between LLMs and services by defining a protocol for interaction.
• Example Workflow (Travel Agent):
  1. User Request: "I want to plan a trip to Paris from December 15th to 20th with flights, hotels near the Eiffel Tower, and an itinerary."
  2. Agent Planning: The agent breaks down the task into finding flights, hotels, generating recommendations, validating preferences, and booking.
  3. Execution: The agent uses its tools (APIs) to gather information, potentially interacts with the user for validation, and books the trip.
  4. Memory Update: The agent learns from the interaction (e.g., user preference for direct flights) and updates its memory.

Evaluation (Eval) of Agentic Workflows

Evaluating agentic workflows is crucial for ensuring they function correctly and efficiently.

• Methods:
  • End-to-End Metrics: User satisfaction ratings.
  • Component-Based Approach: Analyzing the performance of individual tools or LLM steps.
  • Objective vs. Subjective Evaluation:
    • Objective: Checking for factual correctness (e.g., correct order ID extracted).
    • Subjective: Assessing qualities like politeness, tone, or user preference alignment, often requiring human raters or LLM judges.
  • Quantitative Metrics: Success rates, latency, cost.
  • Qualitative Metrics: Error analysis, identifying hallucinations, tone mismatches, user confusion.
• LLM Judges: Can be used to automate subjective evaluations by rating outputs against a rubric or performing comparisons.
• Error Analysis: Manually reviewing interactions to identify patterns of failure.

Multi-Agent Systems

Multi-agent systems involve multiple agents working together, offering benefits like:

• Parallelism: Tasks can be executed concurrently, saving time.
• Reusability: Specialized agents (e.g., a design agent) can be used across different teams or workflows.
• Hierarchical or Flat Structures: Agents can be organized in a hierarchy or a flat, interconnected network.
• Example (Smart Home Automation): Agents for climate control, lighting, security, energy management, entertainment, and an orchestrator agent to manage user interactions and delegate tasks.

Future Trends in AI

The lecture concludes by discussing emerging trends:

• Plateauing LLM Improvement: While LLMs have advanced rapidly, the rate of improvement might slow down. Future gains may come from architectural search and novel approaches beyond current transformer models.
• Multimodality: Integrating different data types (text, images, audio, video) enhances LLM capabilities across modalities, leading to more sophisticated applications like robotics.
• Harmony of Methods: Future AI systems will likely combine various learning paradigms (supervised, unsupervised, reinforcement learning, etc.) to achieve faster learning, lower latency, and greater efficiency.
• Human-Centric vs. Non-Human-Centric Approaches: Research inspired by the human brain is valuable, but exploring computational approaches beyond biological limitations could unlock further advancements.
• Velocity of Change: The AI field is evolving at an unprecedented pace, emphasizing the importance of understanding broad concepts and developing the ability to quickly learn specific, in-demand techniques.

The lecture stresses that the half-life of skills in AI is low, encouraging a focus on breadth of knowledge and the ability to deep-dive as needed.