Building a Chess Coach — Anant Dole and Asbjorn Steinskog, Take Take Take

Key Concepts

• Stockfish: A high-performance, classical chess engine used for move calculation and position evaluation.
• Maya: A neural network-based chess engine trained to predict human-like moves rather than just optimal moves.
• LLM Hallucination: The tendency of Large Language Models to generate plausible but factually incorrect information, particularly problematic in chess where precise calculation is required.
• Context Extraction: The process of identifying tactical and positional themes (e.g., forks, pins, skewers, doubled pawns) to ground LLM outputs.
• Autonomous Agents: AI systems capable of performing tasks (like debugging code or refining prompts) with minimal human intervention.
• Latency vs. Quality Trade-off: The balance between generating fast, real-time feedback for users and the high-compute requirements of complex reasoning models.

---

1. The Play Magnus AI Coaching System

The Play Magnus application provides users with an automated "Game Review" feature. After a game, the system analyzes moves and provides natural language commentary explaining the "why" behind specific tactical or positional decisions.

• Functionality: It identifies "brilliant" moves, explains threats, and provides insights into game phases, player ratings, and opening depth.
• Technical Architecture: The system separates the data pipeline (Stockfish/Maya) from the language generation (LLM). The LLM is restricted to translating pre-calculated data into human-readable English to prevent hallucinations.

2. History of Chess AI

• Type A vs. Type B Engines: Claude Shannon (1949) proposed "Type A" (brute-force search) and "Type B" (selective, intuitive search) engines.
• Evolution: For decades, Type A engines dominated, culminating in Deep Blue’s victory over Kasparov in 1997.
• The Neural Shift: DeepMind’s AlphaGo and AlphaZero introduced the "intuitive" neural network approach, which proved superior for complex games.
• The LLM Problem: While LLMs are excellent at language, they struggle with chess because they lack inherent calculation capabilities, often leading to "hallucinations" in move sequences.

3. Methodology: The AI Pipeline

The team employs a multi-step framework to generate accurate coaching feedback:

1. Calculation: Run Stockfish to determine the objective "best" move.
2. Contextualization: Use detectors to identify tactical themes (forks, pins, etc.) and Maya to understand the probability of a human player finding that move at a specific rating level.
3. Translation: Feed the structured data (JSON) into an LLM (e.g., Gemini 1.5 Flash) to generate the explanation.
4. Human-in-the-loop: When a user reports a bad comment, an autonomous agent (using an MCP server) triggers a triage skill to investigate, modify prompts, and propose a fix via GitHub.

4. Latency and Quality Optimization

• Performance Targets: The team aims for sub-3-second latency for real-time user feedback.
• Model Selection:
  • Gemini 1.5 Flash: Chosen for its speed (Time to First Token ~1s) and reliability in the current production pipeline.
  • Reasoning Models: While more accurate, they are currently too slow for the "instant" feedback required in the app, though they are planned for future "chat with your coach" features.
• Evaluation (Evals): The team maintains 16 distinct chess scenarios (tactics, blunders, etc.) to benchmark models. They use "LLM-as-a-judge" and manual verification by domain experts (the presenters) to ensure quality.

5. Key Learnings and Actionable Insights

• Decouple Logic from Language: Do not ask an LLM to "play" chess; ask it to "explain" data provided by a specialized engine.
• Iterative Context Pruning: Start with a large JSON context file and prune it step-by-step to optimize for both quality and token efficiency.
• Autonomous Debugging: Closing the loop with autonomous agents (e.g., using Slack/GitHub integration) significantly accelerates the development and maintenance cycle.
• Domain Expertise: Rely on Subject Matter Experts (SMEs) to define the "ground truth" for evaluations, even if those experts are not the ones writing the code.

Conclusion

The Play Magnus team successfully bridged the gap between classical chess engines and generative AI by treating the LLM as a translator rather than a calculator. By grounding the AI in verified data from Stockfish and Maya, and utilizing autonomous agents for continuous improvement, they have created a scalable, production-ready coaching tool that balances the strict requirements of chess accuracy with the need for low-latency user experience.