Shocking New AI Just Hit 12 Million Tokens With 1000x Less Compute

Key Concepts

• Sub Quadratic Sparse Attention (SSA): A novel attention mechanism that computes only the most relevant token relationships, enabling linear scaling of compute costs.
• Quadratic Scaling Bottleneck: The inherent limitation of standard Transformer models where doubling input length quadruples computational cost.
• Needle in a Haystack (NIAH): A benchmark testing a model's ability to retrieve a specific piece of information from a massive dataset.
• RAG (Retrieval-Augmented Generation): The current industry-standard "scaffolding" used to bypass context limits by searching and feeding fragments of data to a model.
• Linear Scaling: The goal of SSA, where computational cost grows proportionally with input length rather than exponentially.

---

1. The Core Problem: The Quadratic Bottleneck

Modern AI models rely on Transformers, which use "attention" to compare every token against every other token. This creates a quadratic scaling issue: increasing context from 10,000 to 1 million tokens increases compute requirements by roughly 10,000 times.

To manage this, the industry has built complex infrastructure—vector databases, chunking pipelines, and RAG frameworks. These systems force models to work with fragments rather than the "whole artifact," which is problematic for tasks like legal analysis or code debugging where context is distributed across disparate sections of a document.

2. The Solution: Sub Quadratic Sparse Attention (SSA)

Sub Quadratic claims to have solved this by replacing dense attention with SSA.

• Mechanism: Unlike previous sparse methods that used fixed patterns (e.g., local windows) or compression (e.g., Mamba, RWKV), SSA learns which relationships matter dynamically based on content meaning.
• Linearity: Crucially, both the selection of tokens and the attention computation scale linearly. The company highlights that other systems (like Deepseek’s indexer) fail because their "selector" remains quadratically expensive, effectively just moving the bottleneck rather than removing it.

3. Performance and Benchmarks

Sub Quadratic released the SubQ 1.1 small model, which demonstrates significant efficiency gains:

• Needle in a Haystack: Achieved 100% accuracy at 1M and 2M tokens, and 98% at 6M and 12M tokens, despite being trained primarily on 1M–2M token contexts.
• Efficiency: At 1 million tokens, SSA requires only 3.9 petaflops compared to 252 petaflops for dense attention—a 64x reduction.
• Flash Attention Comparison: SSA reaches parity with Flash Attention 2 at 16,000 tokens and becomes 56x faster at 1 million tokens (966ms vs. 54,164ms on an H100).
• Automation Bench Finance: In a real-world test involving 500 API endpoints, SubQ 1.1 small scored 13%, outperforming models like Sonnet 4.6 (8%) and Haiku 4.5 (3%), and trailing only the largest frontier models.

4. Methodology and Development

• Iterative Training: Because SSA is so efficient, the team ran over 100 long-context experiments across six model generations—a feat impossible with dense attention.
• Architecture: They did not train from scratch; they took existing open-weight frontier models, replaced the dense attention layers with SSA, and extended context using YaRN (Yet another RoPE for Transformers) for positional scaling.
• Trade-offs: The team noted that improving long-context retrieval often degraded "knowledge-heavy reasoning," requiring careful tuning to maintain balance.

5. Notable Quotes and Perspectives

• The "Scaffolding" Argument: The video posits that "the current architecture of enterprise AI is built around a bottleneck that does not have to exist."
• Skepticism: The industry remains cautious, with some comparing the claims to "AI Theranos" due to past failures in the long-context space (e.g., magic.dev). The company has addressed this by seeking third-party verification from Appen.

6. Synthesis and Conclusion

Sub Quadratic’s technology represents a potential paradigm shift in AI architecture. By removing the quadratic cost of attention, they aim to make RAG and complex retrieval pipelines obsolete for "bounded artifacts" like legal contracts or codebases.

Main Takeaway: If Sub Quadratic’s larger models scale as effectively as the 1.1 small model, the industry may move away from "fragmented" AI (searching for pieces) toward "holistic" AI (reading the entire document). The success of this technology will be determined not by benchmarks, but by how well it performs on the "messy" data of real-world production environments.