20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Key Concepts

• State of the Art (SOTA): A relative term in AI that often lacks a singular definition, leading to confusion when selecting models.
• Public Leaderboards: Platforms (e.g., LM Arena, Artificial Analysis) that rank models based on aggregated performance scores.
• Elo Score: A rating system used to calculate the relative skill levels of models based on head-to-head "battles."
• Pareto Front: A graphical representation showing the trade-off between two variables (e.g., quality vs. efficiency), identifying the most optimal models.
• Model Compression: Techniques to improve efficiency, including Quantization (reducing precision), Pruning (removing unnecessary components), and Distillation/Caching (reducing inference steps).
• Inference Latency: The time taken for a model to generate an output (e.g., an image or video).

---

1. The Problem with Current SOTA Evaluation

The speaker argues that relying on "lazy" solutions—simply picking the top-ranked model from a public leaderboard—is often ineffective for specific production needs.

• Inconsistency: Different leaderboards use different methodologies, resulting in conflicting rankings for the same models.
• Lack of Specificity: Aggregated scores mask performance on niche tasks (e.g., text rendering vs. object removal). A model might be "SOTA" overall but perform poorly on a specific user requirement.
• Statistical Insignificance: Many leaderboards are built on a few thousand samples, which is insufficient compared to the millions of inferences a production application might handle daily.

2. Pitfalls of Manual and Automated Benchmarking

• Manual Inspection Bias: Human evaluation is highly subjective. The speaker demonstrated that users have varying preferences based on the specific prompt and the limited number of samples viewed.
• Metric Misalignment: Standard metrics like CLIP score often show negligible variations between models, making it difficult to distinguish performance. Metrics must be chosen based on the specific use case (e.g., using text-rendering-specific metrics for text-heavy tasks).

3. The Efficiency-Quality Trade-off

A critical argument presented is that quality is often driven by excessive compute, which may not be cost-effective or sustainable.

• Case Study: Evaluating a model via 26,000 battles took 20 days of compute, cost $5,000, and consumed energy equivalent to running 400 marathons.
• Alternative: Using optimized, smaller models can achieve similar results in 7 hours, costing only $265 and requiring the energy equivalent of just 4 marathons.
• Actionable Insight: Developers should prioritize Pareto plots to visualize the balance between quality (y-axis) and efficiency/cost (x-axis). This reveals that multiple models can be "SOTA" depending on the user's budget and latency requirements.

4. Methodologies for Optimization

To achieve high performance without relying on massive foundation models, the speaker suggests:

• Quantization: Applying different levels of precision to specific modules within a model.
• Pruning: Removing redundant components that do not contribute significantly to output quality.
• Step Reduction: Reducing the number of denoising steps (e.g., from 50 to 4) through distillation or caching methods, which significantly lowers inference time.

5. Notable Quotes

• "It's not because ChatGPT image is ranked top one on the leaderboard that it means that it's the best overall."
• "People tend to just look at quality, but it's important not to look only at quality, but also at efficiency."
• "There is not one single state of the art model, but there are actually multiple of them."

---

Synthesis and Conclusion

The concept of "State of the Art" is not a static label but a dynamic balance between task-specific performance and operational efficiency. The speaker concludes that benchmarking is not dead, but it must be performed with rigor:

1. Evaluate on large, relevant datasets that mirror actual production conditions.
2. Use multiple metrics that are specifically aligned with the target use case.
3. Prioritize efficiency by using Pareto analysis to find the "sweet spot" between quality and compute cost.
4. Adopt compression techniques (quantization, pruning, caching) to make smaller, specialized models perform at a level comparable to massive foundation models.