Everything I Learned Training Frontier Small Models — Maxime Labonne, Liquid AI

Key Concepts

• Edge Models: Small-scale AI models (350M to 24B parameters) optimized for on-device deployment (phones, cars).
• Memory Bound: A constraint where model performance is limited by hardware memory capacity, necessitating smaller parameter counts.
• Doom Looping: A failure mode where models repeat sequences of words indefinitely; particularly prevalent in small models attempting complex reasoning.
• Short Convolutions (ShortCom): A high-throughput, low-latency architectural component used in LFM2 models.
• Effective Parameters: The portion of a model's parameters dedicated to reasoning and knowledge, excluding large, inefficient embedding layers.
• Agentic Workflows: Using small models as "agents" that utilize external tools (web search, Python) to compensate for limited internal knowledge capacity.

---

1. Characteristics of Small Models

Maxim Labon emphasizes that small models are not merely "scaled-down" versions of large models; they possess unique constraints and requirements:

• Memory Bound: Hardware limitations (e.g., mobile devices) restrict model size, leading to lower knowledge capacity.
• Task Specificity: Because they lack the capacity for general-purpose knowledge, they are best utilized for narrow, high-performance tasks like summarization or tool use.
• Latency Sensitivity: They require high throughput to be viable for real-time applications.

2. Architectural Innovations

Labon contrasts standard architectures (Gemma 3, Qwen 3.5) with Liquid AI’s LFM2 architecture:

• Embedding Layer Inefficiency: Many small models (e.g., Gemma 3 270M) allocate a disproportionate amount of parameters (up to 63%) to the embedding layer, which does not contribute to reasoning.
• LFM2 Optimization: By utilizing Short Convolutions, the LFM2 architecture achieves higher throughput and lower memory usage compared to sliding window attention or gated linear attention.
• On-Device Profiling: The architecture was refined through real-world testing on hardware like the AMD Ryzen Max Plus and Samsung Galaxy S25 Ultra, rather than relying solely on theoretical scaling laws.

3. Training Methodology

The LFM 2.5 training recipe challenges traditional "Chinchilla" scaling laws:

• Token Scaling: Despite being small (350M parameters), the model was pre-trained on 28 trillion tokens. Labon notes that performance continues to scale with token count, even at small sizes.
• Post-Training Pipeline:
  1. Supervised Fine-Tuning (SFT): Focuses on narrow, task-specific data.
  2. Preference Alignment: Uses an "on-policy length-normalized DPO" algorithm.
  3. Reinforcement Learning (RL): Highly effective for small models to improve generalization across diverse tasks.

4. Solving "Doom Looping"

Doom looping occurs when small models face tasks exceeding their reasoning capacity. Liquid AI employs two primary solutions:

• DPO Data Generation: During preference alignment, the team generates multiple rollouts using temperature sampling. A "jury" LLM identifies the best (chosen) and worst (rejected/doom-looping) responses, training the model to avoid the latter.
• RL with Verifiable Rewards: For math or logic tasks, the model is rewarded only if it reaches a correct final answer. Adding an n-gram repetition penalty further discourages repetitive loops.
• Result: These techniques reduced the doom-loop ratio from ~16% to near-zero, whereas models like Qwen 3.5 (treated as simple scaled-down versions) suffer from >50% doom-loop rates in reasoning modes.

5. Strategic Application: The Agentic Approach

Labon argues that the future of small models lies in agency rather than raw knowledge:

• Tool Use: Since small models hallucinate due to low knowledge capacity, they should be paired with web search tools to retrieve accurate information.
• Recursive Environments: For long-context limitations, models can use Python or recursive environments to process data, effectively bypassing the need for massive context windows.
• Use Cases: Small models are ideal for regulated environments (healthcare/finance), offline scenarios (in-car), and latency-critical applications.

Synthesis

The presentation concludes that small models are a distinct scientific and production category. By moving away from the "scaled-down" mindset and focusing on architectural efficiency (ShortCom), aggressive token pre-training, and agentic tool integration, developers can create highly capable, low-latency models that outperform larger, general-purpose models in specific, real-world tasks.