How to Build Planning Agents without losing control - Yogendra Miraje, Factset

Key Concepts:

• LLMs (Large Language Models): Limited by training data, enhanced with tools.
• Augmented LLM: LLM + Tools + Memory.
• Workflow: Augmented LLM on a static, predefined path.
• Agent: Augmented LLM with high autonomy and feedback loops.
• Workflow Agent: Predefined workflow run by an agent (workflow is in control, static).
• Agentic Workflow: Workflow planned and run by an agent (agent is in control, dynamic).
• Planning by Subgoal Division (Task Decomposition): Breaking down a goal into simpler steps.
• Blueprint: A series of steps for a workflow as per tool capabilities in natural language.
• MCP (Minimum Common Platform): Standardized tool server for consistent tool access.
• Evals: Evaluation frameworks for assessing agent performance.

1. Introduction and the Need for Context

• Yogi from FactSet discusses building agents, highlighting the rapid growth of AI.
• AI application development feels like "driving a monster truck through a crowded mall with tiny joysticks," indicating difficulty in controlling AI.
• A key reason for agents' unpredictable behavior is the lack of proper context, especially enterprise-specific workflows.

2. Defining Key Terms: LLMs, Augmented LLMs, Workflows, and Agents

• LLMs: Limited by their knowledge at the time of training.
• Augmented LLMs: LLMs enhanced with tools and memory.
• Workflows: Augmented LLMs following a static and predefined path.
• Agents: Augmented LLMs with high autonomy and feedback loops.
• Workflows are controllable and reliable, while agents are flexible and autonomous.

3. Agentic Workflows: Combining the Best of Both Worlds

• Agentic workflows plan and execute workflows based on goal context and feedback.
• Distinction between Workflow Agent and Agentic Workflow:
  • Workflow Agent: Workflow is predefined and in control.
  • Agentic Workflow: Agent plans and controls the workflow dynamically.
• Agentic workflows have more "agenticness" on the agentic spectrum.
• Agentic workflows enable automation at scale, leveraging existing enterprise microservices.

4. Moving from Reactive to Proactive Agents

• The focus needs to shift from "react-based agents" to "proactive agents."
• Building agentic workflows requires tools, memory, reflection, and the "planning by subgoal division" design pattern (task decomposition).
• Specific agentic architectures and research papers are available (LangChain provides code examples).

5. FactSet's Agentic Workflow Architecture

• FactSet is adopting the LLM compiler architecture.
• Components:
  • Blueprint Generator: Creates a high-level plan (blueprint).
  • Planner: Low-level task planner.
  • Executor: Executes the plan.
  • Joiner: Combines outputs and determines replanning or termination.
• Each component is a node on the LangGraph.
• Most time is spent building tools around microservices.
• The relationship between tools and microservices is not one-to-one; it's up to the designer.
• Key point: Design tools from the agent's perspective, ensuring the agent understands which tool to use.

6. Tool Design and MCP

• Follow standard MCP (Minimum Common Platform) for tool servers.
• Provide tool purpose, description, and input/output contracts for each tool.
  • Tool Purpose: Helps in tool selection.
  • Tool Description: Indicates when to invoke the tool.
  • Input/Output Contracts: Specifies how to use the tool.
• Add validation checks as "brakes" for the agent.

7. The Role of Blueprints

• Blueprint: A series of steps for a workflow as per tool capabilities in natural language.
• Blueprints reduce the cognitive load on the planner.
• Benefits of using blueprints:
  • Final control over task planning.
  • Limits in-context tools for the planner.
  • Helps in interpreting agentic behavior.
  • Facilitates collaboration with non-technical people.

8. Example: Preparing for a Company's Earning Call

• Simplified workflow for preparing for NVIDIA's earning call.
• Blueprint includes tools and tasks (e.g., summarizing previous earning calls, gathering financial data, suggesting questions, generating a report).
• The plan contains tool and function calls.
• Agentic workflow provides a more structured and workflow-aware response compared to a vanilla response.

9. The Importance of Evals

• Invest in building and maintaining a proper evaluation (eval) framework.
• Include component and end-to-end evals.
• Use techniques like code-based evals, LLM judges, and human-in-the-loop evaluations.
• Write evals for metrics that matter.
• Aspect-based evals (e.g., checking if a blueprint resembles a golden blueprint) are useful.

10. When Not to Use Agentic Workflows

• Fixed and repetitive tasks (use ETL pipelines).
• Workflows that cannot be easily captured.
• Deterministic outcomes are paramount (strict compliance, safety-critical contexts).
• Low latency and cost environments.

11. Key Learnings and Takeaways

• Start with simple blueprints and gradually increase complexity.
• Build a robust RAG (Retrieval-Augmented Generation) system for blueprints.
• Use blueprints to reduce in-context tools and provide a high-level plan to the planner.
• Design tools from the agent's perspective.
• Aim for tool usage simplicity.
• Implement safety guardrails and prioritize evals and observability.
• Agentic workflow is planned and run by an agent.
• Agentic workflows bring reliability at scale.
• Planning by subgoal division is a key design pattern.
• Plan and execute is a key agentic architecture.
• Build tools to complement microservices.
• Treat evals as first-class citizens.

12. Q&A

• LangChain is recommended as a starting point for code examples related to plan and execute agents.
• MCP helps provide a standard across the organization, enabling "build once, use everywhere" functionality.
• LangGraph is useful for orchestration, but the optimal framework depends on the specific use case.

Conclusion:

Agentic workflows offer a powerful approach to automating complex tasks by combining the flexibility of agents with the reliability of workflows. Key to success is a well-defined architecture, careful tool design, and a robust evaluation framework. The blueprint concept is crucial for managing complexity and enabling collaboration. While not suitable for all scenarios, agentic workflows can significantly enhance automation capabilities in enterprise environments.