Anthropic Doesn't Trust An Agent to Both Write and Cite. You Shouldn't Either.

Key Concepts

• Hallucination: Generating outputs not supported by retrieved data.
• Overextension: Stating a narrow claim from a source with broader, unjustified confidence.
• Conflation: Fusing multiple retrieved chunks into a single, unsupported claim.
• Citation Mismatch: Citing a real source that does not actually support the associated text.
• Attribution Hallucination: The model incorrectly generating or mapping citation markers/coordinates.
• Faithfulness vs. Truth: Faithfulness ensures the output matches the retrieved source; Truth ensures the source itself is accurate.
• Agentic Verification: Designing systems that force the AI to prove its claims through structured workflows.

---

1. Challenges in Knowledge Agents

As LLMs improve, errors become more subtle and confidently stated, making them harder to detect. The video identifies three primary failure modes in RAG (Retrieval-Augmented Generation) systems:

• Overextension: Exaggerating the scope of a retrieved fact.
• Conflation: Mixing disparate data chunks into a false synthesis.
• Citation Mismatch: Attaching valid sources to irrelevant or incorrect paragraphs.

2. Verification Methodologies

A. Notebook LM Style (Inline Citations & Bounding Boxes)

• Mechanism: The agent embeds citation markers (e.g., S1, S2) in the output. The UI maps these to specific bounding boxes in the source document.
• Technical Detail: Avoid asking the LLM to calculate coordinates, as this leads to "attribution hallucination." Instead, maintain a backend mapping of evidence markers to pre-calculated bounding boxes.
• Tools: DocLine, Data Labs (Marker), Microsoft Azure Document Intelligence, and Google Cloud Document AI are used to extract text with precise spatial coordinates.

B. Semantic Verification (Faithfulness Checker)

• Process: A secondary, specialized LLM (e.g., Gemini 1.5 Flash) acts as a "faithfulness checker."
• Workflow: It receives the generated claim and the cited evidence, then returns a status (Verified, Contradicted, or Unclear) with a confidence score.
• Implementation: This is often triggered as a user-initiated action rather than on every generation to manage latency and costs.

C. Dedicated Citation Agent

• Architecture: Inspired by Claude’s "Deep Research," this involves a multi-agent system where a lead orchestrator delegates research to sub-agents. Once the report is generated, a final "Citation Agent" reviews the text to insert accurate source markers.

D. Evidence-First & Claim-First Architectures

• Evidence-First: A research LLM collects raw evidence paths/IDs first; a separate generator agent then writes the response using only that curated context, reducing noise.
• Claim-First: The system decomposes a request into atomic claims, verifies each against evidence, and then compiles them into a final output.

3. Fact-Checking and Truth Verification

Faithfulness checking only ensures the AI didn't lie about its sources. To ensure the truth of the information, the system must validate against authoritative external sources.

• Fixed Style Harness: A structured, multi-stage process (e.g., for legal briefs) that parses documents, extracts specific entities (case law, statutes), and cross-references them against external legal databases.
• Dynamic Harness: Uses runtime planning to fan out tasks to parallel agents, including "adversarial agents" tasked with attempting to refute the findings.

4. Notable Quotes

• "As models get bigger, the errors don't necessarily disappear, they just get more subtle and more confidently stated, which actually makes them harder to catch."
• "This system is just checking the faithfulness to the source. It's not checking the truth of the source. So if the retrieved document is old or wrong, you're going to get a faithfully verified wrong answer."

5. Synthesis and Conclusion

Building trustworthy AI agents requires moving beyond "single-shot" generation. Developers must implement verification loops that treat citations as a first-class citizen of the UI. Whether through inline bounding boxes, secondary semantic verification, or multi-agent citation architectures, the goal is to reduce the "surface area" of potential errors. For high-stakes domains like law or finance, developers should implement fixed or dynamic verification harnesses that validate claims against authoritative external ground truth, ensuring that the system provides due diligence rather than just fluent text.