Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo

Key Concepts:

• Ideal Customer Profile (ICP) analysis
• Large Language Models (LLMs)
• Hallucination rate in AI models
• Retrieval Augmented Generation (RAG)
• Prompt engineering (Chain of Thought prompting)
• JSON structured outputs
• Prompt caching
• Extended outputs
• AI engineering
• Data as an asset

1. The Problem: Analyzing a Large Volume of Sales Calls

• The speaker's company, Pulley, wanted to refine their Ideal Customer Profile (ICP) beyond "venture-backed startups" to a more specific target, like "CTO of an early-stage venture-backed crypto startup."
• They had thousands of hours of sales calls, representing a wealth of customer data.
• Manually analyzing 10,000 sales calls (assuming 30 minutes per call) would take approximately 625 days (nearly 2 years) of continuous work, making it impractical.
• Traditional approaches were either high-quality but unscalable (manual analysis) or fast but lacking context (keyword analysis).

2. The Solution: Leveraging Large Language Models (LLMs)

• LLMs are well-suited for analyzing unstructured data and recognizing patterns.
• The initial challenge was selecting the right model. GPT-4 and Claude 3.5 Sonnet were the most intelligent but also the most expensive and slowest.
• Smaller, cheaper models produced too many false positives (hallucinations), misclassifying transcripts based on superficial keywords (e.g., classifying a transcript as crypto-related because "blockchain" was mentioned).
• The decision was made to use Claude 3.5 Sonnet due to its acceptable hallucination rate, despite the higher cost.

3. Multi-Layered Approach to Reduce Hallucinations

• The process wasn't as simple as feeding transcripts into the model and extracting answers.
• A multi-layered approach was developed:
  • Raw Transcript Data: Starting with the original sales call transcripts.
  • Retrieval Augmented Generation (RAG): Enriching the data with information from third-party sources and internal data.
  • Prompt Engineering (Chain of Thought): Using techniques like chain-of-thought prompting to guide the model towards more reliable results.
  • Structured JSON Outputs: Ensuring the model produced structured JSON outputs to facilitate citations and verification.
• This system reliably extracted accurate company details and meaningful insights with a verifiable trail back to the original transcripts.

4. Cost Optimization Strategies

• Analyzing transcripts with low error rates significantly increased costs, especially when hitting the 4,000-token output limit of Claude 3.5 Sonnet.
• Two experimental features were leveraged to dramatically lower costs:
  • Prompt Caching: Caching transcript content reduced costs by up to 90% and latency by up to 85% because the same transcripts were repeatedly used for metadata and insight extraction.
  • Extended Outputs: Using Claude's experimental feature flag to double the output context allowed for complete summaries in single passes, avoiding multiple requests.
• These optimizations reduced the cost of analyzing 10,000 sales calls from $5,000 to $500 and the processing time from weeks to days.

5. Unexpected Wide-Ranging Impact

• The analysis had a broader impact than initially anticipated.
• The marketing team used the data to identify customers for branding and positioning exercises.
• The sales team automated transcript downloads, saving dozens of hours per week.
• Teams started asking new questions that were previously considered too daunting to investigate manually.
• The project transformed unstructured data from a liability into an asset.

6. Key Takeaways

• Models Matter: Claude 3.5 and GPT-4 could handle tasks that smaller models couldn't. The right tool isn't always the most powerful but the one that best fits specific needs.
• Good Engineering Still Matters: AI engineering involves building effective systems around LLMs, integrating them thoughtfully into existing architectures, and leveraging techniques like JSON structured output and good database schemas.
• Consider Additional Use Cases: Building a flexible tool with features like search filters and exports transformed a one-off project into a company-wide resource.

7. The Promise of AI

• AI can transform seemingly impossible tasks into routine operations.
• It's about augmenting human analysis and removing bottlenecks, not replacing humans.
• Tools like Claude, ChatGPT, and Gemini unlock entirely new possibilities.

8. Call to Action

• The speaker challenges viewers to identify the customer data they are currently sitting on (sales calls, support tickets, product reviews, user feedback, social media interactions).
• These data sources are valuable but often untouched.
• The tools and techniques exist to turn this data into valuable insights.

Synthesis/Conclusion:

The speaker details a project where AI, specifically Claude 3.5 Sonnet, was used to analyze 10,000 sales calls to refine the company's ICP. The project highlights the importance of choosing the right AI model, employing robust engineering practices to minimize hallucinations and optimize costs, and considering the broader organizational impact of AI-driven insights. The key takeaway is that AI, when implemented thoughtfully, can transform unstructured data into a valuable asset, unlocking new possibilities and augmenting human capabilities.