Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel

Key Concepts

• Data Quality: The foundational premise that the quality of training data directly dictates the performance and reliability of AI models.
• Agentic Tasks: AI tasks involving autonomous agents that interact with environments (e.g., terminal-based tasks) to achieve specific goals.
• Task Quality: A metric defined by achievability, non-triviality, functional correctness, and environmental reliability.
• Reinforcement Learning (RL) Training: The process of improving model performance through iterative feedback loops based on task outcomes.
• Expert-in-the-Loop: The methodology of using human expertise to curate, verify, and scale high-quality datasets.
• Failure Modes: Categorization of model errors into "logical errors" (model failure) versus "environmental noise" (task/infrastructure failure).

---

1. The Core Thesis: Data Quality as Task Quality

Kobe Crawford, a developer advocate at Snorkel, emphasizes that in the context of Agentic AI, "task quality" and "data quality" are effectively synonymous. Snorkel’s research focuses on providing empirical evidence that high-quality tasks—those that are well-defined and verifiable—are essential for "hill climbing" (improving) foundation models.

2. Defining and Verifying Task Quality

To ensure rigor, Snorkel evaluates tasks based on four criteria within containerized, isolated environments:

1. Achievability: The task can be completed by a capable model.
2. Non-triviality: The task requires meaningful reasoning and tool usage.
3. Functional Correctness: The logic of the task executes as expected.
4. Environmental Reliability: The infrastructure supporting the task is stable and reproducible.

Tasks that pass these criteria are labeled "Accepted," while those that fail are "Rejected." This binary classification allows for a comparative analysis of how task quality influences model training.

3. Empirical Findings: Accepted vs. Rejected Tasks

Using models like Claude 3.5 and Codex, Snorkel compared performance on accepted versus rejected tasks:

• Complexity: Accepted tasks required twice as many tool calls and more output tokens, indicating they were intrinsically more difficult and required deeper reasoning.
• Failure Analysis: The team categorized failures to distinguish between "meaningful" model failures (logical errors) and "degenerate" failures (environmental issues). Accepted tasks produced "cleaner" failures, which provide better signals for model improvement.
• Performance Uplift: In RL training runs with identical compute budgets, models trained on high-quality (accepted) tasks saw a 6% improvement, whereas models trained on low-quality (rejected) tasks saw only a 1% improvement. This 5x performance gap serves as strong evidence for the necessity of data curation.

4. Methodologies and Challenges

• The "Noise" Problem: Crawford notes that low-quality tasks often act as "noise" in the training process. If a task is under-specified or contains hidden dependencies, the model cannot learn effectively, and the failure provides no useful feedback.
• Human-in-the-Loop: Snorkel utilizes human experts to define rubrics and ground truth. This human expertise is scaled using LLM judges, with the team constantly monitoring inter-annotator agreement to ensure consistency between human experts and automated evaluators.
• Future Directions: The team is expanding into "fuzzy" domains where verification is not as straightforward as coding or math. They are currently working on benchmarks that incorporate human-centric, emotional, or long-horizon tasks where multiple outcomes might be valid.

5. Notable Quotes

• "The quality of data is critical... task quality and data quality are largely the same thing."
• "Accepted tasks are producing cleaner failures... this is a test that would be actually very useful for the model to be able to hill climb on."
• "The improvement was about a 6% improvement with the higher-quality tasks... that uplift of the 5x uplift difference based on quality is really striking."

6. Synthesis and Conclusion

The presentation concludes that data quality is the primary driver of model improvement in agentic workflows. By implementing a rigorous, containerized, and expert-verified framework for task selection, developers can significantly accelerate model performance. The key takeaway is that investing in the curation of high-quality, verifiable tasks—and filtering out noisy, under-specified data—yields a disproportionately higher return on investment during the training phase compared to simply increasing the volume of data.