What Lies Beneath the API — Benjamin Cowen, Modal

Key Concepts

• Frontier APIs: Large-scale, general-purpose LLM services (e.g., OpenAI, Anthropic) that offer rapid development but limited customization.
• Fine-tuning: The process of taking a pre-trained model and training it further on a specific dataset to improve performance for a niche domain.
• Serverless Compute: A cloud execution model where the provider manages the infrastructure, allowing developers to run code without provisioning or managing servers.
• Reinforcement Learning (RL): A machine learning paradigm where an agent learns to make decisions by performing actions and receiving feedback (rewards).
• Rollouts: In RL, the process of running many parallel simulations to gather data on how an agent performs in various scenarios.
• Inference: The process of using a trained model to make predictions or generate content.

---

1. The Shift Toward Domain-Specific Models

Ben Cohan argues that as AI products mature, companies inevitably reach a point where general-purpose "Frontier APIs" are no longer sufficient. While these APIs are excellent for rapid prototyping, they lack the ability to optimize for specific business logic, latency requirements, or cost-efficiency.

• The "Caveman Mode" Limitation: Using prompt engineering (e.g., "speak like a caveman") to reduce token usage is a temporary fix that does not scale for high-growth startups.
• Economic Signals: If a company’s API costs exceed the revenue generated by the product, or if evaluation metrics (evals) have plateaued, it is a strong indicator that the product has entered a "custom domain" phase where fine-tuning is necessary.

2. The Middle Ground: Accessible Fine-Tuning

Traditionally, fine-tuning required massive infrastructure, dedicated AI/infrastructure engineers, and complex cluster management. Cohan highlights a shift where modern serverless platforms and open-source libraries have democratized this process.

• Algorithm Control vs. Infrastructure Management: Modern tools allow developers to maintain full control over the training algorithm without needing to manage the underlying hardware clusters.
• Fast Iteration: The goal is to retain the speed of the "Frontier API" development cycle while gaining the performance benefits of a custom-trained model.
• Real-World Success: Companies like Intercom and Decagon have successfully moved to custom models, achieving superior performance at a fraction of the cost of general-purpose APIs.

3. Prerequisites for Training

Cohan emphasizes that companies often already possess the necessary components to begin training without realizing it:

• Data Collection: The "garbage in, garbage out" rule applies. If a company has been building an "agent harness" or tracking product performance, they have already been curating the training data required for fine-tuning.
• Mature Evals: Before training, a company must have a robust system for evaluating model performance. Without clear metrics, training is ineffective.

4. Methodologies and Technical Implementation

• Supervised Fine-Tuning: Can now be implemented in as little as 300 lines of Python code using modern open-source libraries.
• Hyperparameter Tuning: Serverless platforms allow for "fan-out" architectures where multiple containers are spun up on-demand to test different parameters, killing off underperforming runs immediately to save costs.
• Reinforcement Learning (RL) at Scale: RL requires massive parallelization. Cohan notes that customers are now scaling to 50,000–100,000 sandboxes to perform "rollouts," a task made feasible by unified APIs that treat GPU clusters and sandboxes as interchangeable resources.
• Inference Serving: Once trained, models can be deployed using tools like vLLM, SG Lang, or Triton Inference Server, which allow for auto-scaling based on real-time traffic.

5. Notable Quotes

• "The frontier labs probably don't have the exact same goal as you... they want their models to win on everything possible. And we want our models to win at our business logic." — Cohan, citing a perspective from the company Decagon.
• "If you've built an agent harness, then you have what you need to have a new model learn through reinforcement learning how to provide your service."

6. Synthesis and Conclusion

The transition from general-purpose APIs to custom-trained models is a natural evolution for successful AI products. The barrier to entry for this transition has dropped significantly due to serverless compute and accessible open-source frameworks.

Main Takeaways:

1. Don't wait: Start preparing for custom training now by focusing on data collection and building robust evaluation frameworks.
2. Leverage existing work: If you have an agent-based product, you already have the data and the evaluation loop needed for reinforcement learning.
3. Infrastructure is no longer a blocker: With serverless platforms, the complexity of managing clusters is abstracted away, allowing developers to focus on the algorithm and the business logic.