What Happens When GPT, Claude, and Gemini Rank Each Other Blindly?

Key Concepts

• LLM Council: A multi-agent framework where multiple AI models provide independent responses to a single prompt to improve accuracy through diverse perspectives.
• AI Gateway: A middleware service (specifically Vercel’s) that provides a unified API endpoint to access multiple model providers, handling routing, failover, and analytics.
• Ensemble Learning: A machine learning technique where multiple "weak learners" are combined to outperform a single "strong learner."
• Answer Synthesis: The process of aggregating multiple model outputs into a final, cohesive response that highlights consensus and dissent.
• Latency & Cost: The primary trade-offs of using a multi-model approach, as running multiple concurrent requests increases both time-to-response and token consumption.

---

1. The LLM Council Framework

The "Council" approach, inspired by Andrej Karpathy, moves away from relying on a single biased model. The process follows a three-stage methodology:

1. Independent Generation: Multiple models (e.g., GPT, Claude, Gemini) receive the same prompt simultaneously. Each is assigned a specific persona (e.g., Skeptic, Domain Expert, Contrarian) to ensure diverse viewpoints.
2. Blind Ranking: Models review and rank each other’s responses anonymously to prevent bias.
3. Chairman Synthesis: A final model (the "Chairman") analyzes the ranked responses, identifies areas of consensus and dissent, and generates a final, synthesized answer with a confidence score.

2. Technical Implementation & Infrastructure

• Vercel AI Gateway: This is the backbone of the project. It allows developers to swap models by changing a single string in the code. It provides:
  • Unified API: OpenAI-compatible endpoint.
  • Observability: A dashboard for tracking latency, request volume, and costs.
  • Resilience: Automatic retries and failover if a provider hits rate limits or goes down.
  • Pricing: No markup on token costs; the gateway service is effectively free.
• Parallel Execution: The system fires requests to all models in parallel to minimize the total time taken for the council to reach a conclusion.

3. Experimental Findings & Benchmarks

The author tested the Council against individual models using a blind, independent judge. The results were nuanced:

• Performance: The Council only outperformed individual models in open-ended design questions.
• Failure Points: It performed poorly on factual questions (where individual models were already accurate) and risk-related questions.
• Format Efficiency: The study found that "plain" final answers consistently scored better than verbose, full write-ups of the council's deliberation process.
• Key Takeaway: The Council is a "judgment tool," not a "calculator."

4. Strategic Application: When to Use

The author provides a framework for deciding when to implement a multi-model council:

| Use Case | Recommendation | Reason | | :--- | :--- | :--- | | High-Stakes Strategy | Use | Benefits from diverse perspectives and second opinions. | | Open-Ended Design | Use | Multiple viewpoints help identify trade-offs. | | Simple Lookups | Avoid | Unnecessary cost and latency. | | Latency-Sensitive Apps | Avoid | Parallel processing is inherently slower than a single call. | | Factual/Checkable Data | Avoid | Single strong models are sufficient; the council adds no value. |

5. Notable Quotes

• "Multiple weak learners are better than a single strong learner." (Referencing the ensemble learning principle).
• "The way to think about the council is that it's a judgment call. It's not a calculator."

6. Synthesis/Conclusion

The LLM Council is a powerful architectural pattern for complex, subjective tasks where the cost of a wrong decision is high. However, it is not a universal solution. Developers should prioritize "workhorse" models for standard production tasks and reserve the Council pattern for scenarios requiring deep reasoning, strategy, or the mitigation of model-specific biases. The use of an AI Gateway is essential for managing the complexity of such a multi-model architecture, providing the necessary observability to justify the increased cost and latency.