Rethinking how we Scaffold AI Agents - Rahul Sengottuvelu, Ramp

Key Concepts

• AI Agents
• LLMs (Large Language Models)
• Bit or Lesson (Systems that scale with compute beat systems that don't)
• Exponential Trends
• Classical Compute vs. Fuzzy Compute (LLM-based)
• Switching Report (CSV parsing agent)
• Code Interpreter
• Backend Architecture
• Request-Response Model

Ramp's AI Agent Evolution: A Case Study

The speaker, head of AI at Ramp, discusses the evolution of AI agents, particularly focusing on a "switching report" agent designed to parse CSV files from various third-party card providers. The goal is to onboard users to Ramp by importing their transaction history from other platforms.

1. Initial Approach: Manual Coding

• Description: Manually write code to handle the 50 most common third-party card vendor CSV formats.
• Pros: Direct, deterministic, and initially functional.
• Cons: Requires constant maintenance as vendors change formats, not scalable, and labor-intensive.

2. Hybrid Approach: Classical + Fuzzy Compute

• Description: Use LLMs to classify columns in the CSV (date, transaction amount, merchant name, etc.) and map them to a standardized schema.
• Process:
  • Take each column in the CSV.
  • Use an embedding model or semantic similarity to classify the column type.
  • Map the classified columns to Ramp's internal schema.
• Pros: More general than the manual approach.
• Cons: Still relies heavily on classical scripting, and the LLM component is limited to classification.

3. LLM-Centric Approach: Code Interpreter & Parallel Execution

• Description: Give the entire CSV to an LLM with a code interpreter, allowing it to write and execute code (e.g., using Pandas) to transform the data into the desired format.
• Process:
  • Provide the LLM with the CSV data.
  • Give the LLM access to a code interpreter with libraries like Pandas.
  • Provide a unit test or verifier to check the output.
  • Run the process multiple times in parallel (e.g., 50 times).
• Pros: Highly generalizable, leverages the LLM's reasoning and coding abilities.
• Cons: Computationally expensive (10,000x more compute than the first approach), but potentially cheaper than engineering time and the cost of failed CSV imports.
• Key Argument: While computationally intensive, this approach is more cost-effective due to reduced engineering effort and improved reliability.

The "Bit or Lesson" and Exponential Trends

• Core Idea: Systems that scale with compute beat systems that don't.
• Explanation: Exponential trends are rare and powerful. LLMs are improving exponentially, so building systems that leverage them will benefit from this improvement.
• Historical Examples: Chess, Go, Computer Vision, Atari games. In each case, brute-force approaches (scaling compute) eventually outperformed hand-crafted, rule-based systems.
• Application to Ramp: By increasing the "blue arrow" (fuzzy compute) portion of their systems, Ramp benefits directly from the ongoing improvements in LLMs.

Rethinking Backend Architecture: The LLM as Backend

• Traditional Web App Model: Frontend (JavaScript, HTML, CSS) makes requests to a backend, which interacts with a database.
• Proposed LLM-Centric Model: The LLM is the backend. It has access to tools like a code interpreter, network requests, and a database.
• Demo: LLM-Powered Email Client:
  • The speaker demonstrates an email client where the LLM renders the UI and handles user interactions.
  • When the user logs in, the Gmail token is sent to the LLM.
  • The LLM decides how to render the UI and responds to user actions (e.g., clicking on an email).
  • The LLM can perform actions like marking emails as unread or deleting them.
• Challenges: The demo is slow and not fully functional, indicating that the technology is still in its early stages.
• Future Potential: The speaker suggests that this type of architecture may become more viable as LLMs continue to improve.

Conclusion

The speaker advocates for building AI agents and backend systems that leverage the power of LLMs and scale with compute. By embracing "fuzzy compute" and allowing LLMs to handle more of the decision-making and execution, companies can benefit from the exponential improvements in AI technology. While the LLM-centric approach may not be fully mature yet, it holds significant potential for the future of software development.