Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Key Concepts

AI Agents, Knowledge Work Automation, Unstructured Data, Assistive Agents, Automation Agents, Agent Architecture, Document Toolbox, Document MCP Server, Complex Documents, Document Parsing, Excel Agent, Agent Orchestration, Constrained Architectures, Unconstrained Architectures, Financial Due Diligence, Enterprise Search, Technical Data Sheet Injection.

Building AI Agents that Actually Automate Knowledge Work

Introduction

The talk focuses on how AI agents can automate knowledge work, particularly with unstructured data, moving beyond simple Retrieval-Augmented Generation (RAG) chatbots. The speaker, Jerry, co-founder and CEO of LlamaIndex, emphasizes that 90% of enterprise data is unstructured (PDFs, PowerPoints, Word, Excel) and traditionally requires human review for decision-making. AI agents can now reason, act, analyze, research, and synthesize insights from this data to automate end-to-end processes.

Types of AI Agents

Two main categories of AI agents are discussed:

• Assistive Agents: These agents are designed to help humans get information faster, typically through a chat interface. They act as co-pilots, assisting in tasks but requiring human guidance.
• Automation Agents: These agents automate routine tasks, running in the background with less human intervention. They handle operational tasks and can take actions independently, often with batch review at the end.

The Stack for Building AI Agents

The stack consists of two main components:

• Tools: These are interfaces that allow agents to interact with the external world, surface relevant context, and take actions.
• Agent Architecture: This involves reasoning loops (general or constrained) that encode business logic to achieve specific tasks.

Building a Document Toolbox

This section focuses on creating tools that enable AI agents to interact with unstructured documents effectively.

• Data Pre-processing Layer: This layer is crucial for ensuring data quality. It involves:
  • Data Connectors: Sync data from sources like SharePoint, Google Drive, S3, and Confluence, including permissions and metadata.
  • Document Parsing and Extraction: Accurately understand documents, including tables, charts, and other complex elements.
  • Indexing: Index data for efficient retrieval using methods like vector indexing, SQL tables, or graph databases.
• Document MCP Server: This is a set of tools that equip an AI agent to understand and manipulate documents. It includes:
  • Semantic Search: Fuzzy find relevant data sources.
  • File Lookup: Look up file metadata.
  • Manipulation: Perform operations on files.
  • Structure Querying: Query structured databases for aggregate insights.
• Complex Documents: Handling complex documents (PDFs with embedded tables, charts, images, irregular layouts) is critical. LLMs can be used for document understanding, interleaving them with traditional parsing techniques and adding test-time tokens for validation and reasoning. LlamaIndex's cloud service outperforms existing parsing benchmarks.

Excel Capabilities

LlamaIndex has introduced new Excel capabilities to complement the document toolbox.

• Excel Agent: This agent transforms unnormalized Excel spreadsheets into normalized 2D formats and allows for agentic QA.
• Problem: Traditional RAG and text-to-CSV techniques are ineffective on complex Excel spreadsheets with gaps in rows and columns.
• Solution: The Excel agent deeply understands the semantic structure of the spreadsheet and uses specialized tools to answer questions.
• Performance: Achieves 95% accuracy on synthetic Excel sheets, surpassing human baselines (90%) and LLM with code interpreter (70-75%).
• How it Works:
  • Structure Understanding: Uses reinforcement learning (RL) to learn a semantic map of the sheet.
  • Specialized Tools: Translates the semantic map into specialized tools for the agent to reason over the Excel spreadsheet.

Agent Design Patterns

This section discusses different agent architectures and their corresponding use cases.

• Agent Orchestration: Ranges from constrained (explicitly defined control flow) to unconstrained (e.g., ReAct loop, function calling).
• Assistive UXs (Chat-Oriented):
  • Input: Natural language.
  • Architecture: Unconstrained (ReAct loop).
  • Human in the Loop: High degree of human guidance.
  • Goal: Help humans surface information.
• Automation UXs:
  • Input: Batch of inputs.
  • Architecture: Constrained.
  • Human in the Loop: Less human intervention at each step, batch review at the end.
  • Output: Structured results, API integrations, decision-making after approval.
• Automation Agents as Backend: Automation agents can serve as a backend for data ETL and transformation, providing tool interfaces for assistant agents.

Document Agent Use Cases

This section presents real-world examples of document agents automating knowledge work.

• Financial Due Diligence (Carl): Combines automation and assistant UXs for end-to-end leverage buyout analysis. It involves inhaling unstructured financial data, bespoke extraction algorithms, human review, and a co-pilot interface for analysts.
• Enterprise Search (SEMX): Uses task-specific agentic RAG chatbots over defined data collections, adding an agentic reasoning layer to break down queries and answer questions.
• Technical Data Sheet Injection: Automates the processing and review of technical data sheets for a global electronics company, transforming weeks of manual work into an automated extraction interface.

Conclusion

LlamaIndex is a platform for automating document workflows with agentic AI. The focus has evolved from broad RAG to a more specialized approach for automating knowledge work with unstructured data. The key is to build a comprehensive document toolbox, understand different agent architectures, and apply them to real-world use cases.