Full Workshop: Build Your Own Deep Research Agents - Louis-François Bouchard, Paul Iusztin, Samridhi

Key Concepts

• AI Slop: Generic, shallow, or hallucinated content generated by AI that lacks human value.
• Autonomy Slider: A framework for balancing system complexity, cost, and control (ranging from simple prompting to agentic workflows).
• Workflow vs. Agent: Workflows are deterministic, sequential processes; Agents are autonomous systems capable of planning, tool use, and reacting to environmental feedback.
• Context Rot: The degradation of model performance as context windows grow, often caused by the "lost in the middle" phenomenon.
• MCP (Model Context Protocol): An open standard for connecting AI agents to tools, data, and resources.
• Evaluator-Optimizer Pattern: A loop where an LLM generates content, and a separate "reviewer" LLM provides structured feedback to improve the output.
• Observability: Using tools (e.g., Opik) to track traces, latency, cost, and token usage for debugging.
• AI Evals: The process of quantifying system performance using a labeled dataset, a judge model, and F1 scores.

---

1. The Problem: AI Engineering Constraints

AI engineers face unique constraints compared to traditional software engineers:

• Cost per task: Varies significantly based on model architecture.
• Latency: Critical for reasoning models.
• Context Management: Managing the "context budget" to keep inputs lean and relevant.
• Reliability: Avoiding hallucinations and "AI slop" (e.g., overused phrases like "rapidly evolving" or "it's not about X, but Y").

2. System Architecture: Research & Writing

The workshop demonstrates an end-to-end system for producing high-quality technical articles, split into two distinct phases:

A. The Research Agent (Exploratory)

• Goal: High precision and recall in gathering information.
• Methodology: Uses Fast MCP to expose tools.
• Tools:
  • Deep Research Tool: Performs grounded Google searches via Gemini API.
  • YouTube Analyzer: Extracts transcripts and summaries directly from URLs using Gemini’s multimodal capabilities.
  • Compile Tool: Aggregates findings into a research.md file.
• Key Insight: The agent acts as the "brain" that identifies gaps in information and decides when to pivot or perform additional searches.

B. The Writing Workflow (Deterministic)

• Goal: Consistent, high-quality content that avoids AI-typical patterns.
• Framework:
  • Guideline: User-defined input (topic, angle, key points).
  • Profiles: Static markdown files defining structure, terminology (banned words), and character style.
  • Few-Shot Examples: High-quality representative posts to guide the model.
• Evaluator-Optimizer Loop:
  1. Writer: Generates the first draft.
  2. Reviewer: Uses a separate context window to critique the draft against guidelines and profiles, outputting structured Pydantic objects (location, comment, severity).
  3. Editor: Applies the feedback to refine the post.

3. Observability and Evaluation

To ensure the system is production-ready, the team emphasizes:

• Monitoring: Using Opik to visualize traces, token usage, and cost.
• AI Evals:
  • Dataset Creation: Building a labeled dataset (Pass/Fail + critique) from real-world examples.
  • Judge Calibration: Testing the "Judge LLM" against a dev/test split and calculating the F1 Score to ensure it aligns with human domain experts.
  • Regression Testing: Ensuring new features do not break existing functionality.

4. Notable Quotes

• "Most people miss it... [AI] generates a lot of meaningless and shallow content. What we need here is proper research and proper writing." — Lu Frana
• "Skills are a very clean way of doing [tasks]... it is only loaded when needed and wiped off the context afterward, avoiding context rot." — Samrudi
• "Never ask the LLM to directly generate the output [for evals]. Always ask it to help you generate the input, but never the output." — Paul

5. Synthesis and Conclusion

The workshop highlights that effective AI engineering is not about choosing the most complex agentic system, but about choosing the simplest solution that works. By combining deterministic workflows for writing with agentic systems for research, and wrapping them in a rigorous evaluation and observability framework, developers can move beyond "AI slop" to create reliable, high-value technical content. The core takeaway is that data quality (the evaluation dataset) is more important than the prompt engineering itself.