Plan with Claude Opus, Build with Kimi K2.6? LIVE Mixed-Provider Benchmark

Key Concepts

• Archon: An orchestration framework that acts as a "harness" for AI agents, allowing for the creation of deterministic, modular, and reproducible workflows (DAGs) that can mix different LLM providers.
• Dark Factory: A codebase that autonomously evolves, triages issues, tests, and merges code without human intervention.
• Provider Mixing: The strategy of using different LLM providers (e.g., Anthropic’s Claude Opus vs. Kimmy K 2.6) for specific nodes in a workflow to optimize for cost, token efficiency, and reasoning capability.
• Second Brain: An AI-driven management layer that orchestrates the execution of Archon workflows and evaluates their output.
• Benchmark Evaluator: A structured assessment tool that scores pull requests across seven dimensions (e.g., root cause analysis, scope discipline, code quality, and plan-implementation fidelity).

---

1. Main Topics and Methodology

The video focuses on live-benchmarking different combinations of LLMs within Archon workflows to determine the most cost-effective and high-quality configuration for autonomous coding.

• The Workflow Matrix: The creator tested four combinations of models (Opus and Kimmy) across three distinct GitHub issues:
  • OO: Opus for planning and implementation.
  • OK: Opus for planning, Kimmy for implementation.
  • KO: Kimmy for planning, Opus for implementation.
  • KK: Kimmy for planning and implementation.
• The Evaluation Framework: Each workflow was scored on a 70-point scale (7 dimensions, 10 points each). Key metrics included:
  • Scope Discipline: Ensuring the agent only touches necessary files (avoiding "sprawl").
  • Plan-Implementation Fidelity: Comparing the final diff against the initial plan to see if the implementation followed the blueprint.
  • Subtle Correctness: Handling edge cases and async operations.

2. Step-by-Step Process

1. Issue Identification: The "Second Brain" identifies GitHub issues in the repository.
2. Exploration Node: Uses a model (e.g., Sonnet) to explore the codebase and gather context.
3. Planning Node: Generates a markdown-based "blueprint" for the fix.
4. Implementation Node: Executes the code changes based on the plan.
5. Self-Review/Validation: The agent reviews its own work before submitting a Pull Request (PR).
6. Evaluation: The evaluator compares the PR against the original plan and the seven-dimension rubric.

3. Key Arguments and Findings

• Reasoning vs. Execution: The primary finding is that reasoning capability is most critical during the planning phase.
• The "Opus-Planning" Advantage: The data suggested that using a high-reasoning model (Opus) for planning, followed by a cheaper, faster model (Kimmy) for implementation, yields results nearly identical to using Opus for everything.
• Diminishing Returns: Upgrading the implementation model to Opus provided minimal gains compared to the significant boost gained by upgrading the planning model to Opus.
• Reliability Issues: The creator noted that Kimmy’s API occasionally suffers from "tool edit" failures and hanging, which can cause workflow bottlenecks compared to the more stable Claude SDK.

4. Notable Quotes

• "Archon acts as the product manager or orchestrator holding these models to structured boundaries."
• "Spend Opus on planning and let Kimmy implement."
• "Garbage in, garbage out." (Regarding the necessity of a high-quality plan for a successful implementation).

5. Technical Observations

• Token Efficiency: By offloading implementation to cheaper models, the creator significantly reduced costs while maintaining high code quality.
• Worktree Isolation: Archon uses git worktree to create isolated environments for each workflow, ensuring that concurrent runs do not interfere with one another.
• Anti-Gravity Experience: The creator experimented with the "Anti-Gravity" tool, noting that while Gemini 3.5 Flash is exceptional at UI/frontend design, it is prone to hallucinations regarding technical documentation and suffers from restrictive rate limits.

6. Synthesis and Conclusion

The experiment confirms that for autonomous coding workflows, the quality of the plan is the primary determinant of success. By utilizing a "Planning-Heavy" architecture—where a frontier model like Opus creates a detailed, structured blueprint—developers can safely delegate the actual coding and validation to more cost-effective models like Kimmy. This approach maximizes token efficiency and allows for scaling autonomous development without hitting the rate limits or high costs associated with using top-tier models for every step of the software development lifecycle.