The Miranda Hypothesis: How Hamilton Poisoned Persona Evals - Jacob E. Thomas, Results Gen

Key Concepts

• Role-Playing Language Agent (RPLA): A system designed to instantiate a persona (real or fictional) to reason and speak as that individual.
• Miranda Hypothesis: The theory that culturally dominant representations of a figure (e.g., Hamilton the musical) saturate training corpora, causing models to produce "composites" that overwrite the actual documentary record.
• Cognitive Simulation: The current industry standard for modeling personality through psychological frameworks and motivational chains.
• Epistemic Simulation: A proposed fourth paradigm that constrains AI reasoning to specific primary documents, temporal anchors, and expert-led evaluation.
• Algorithmic Sycophancy: The tendency of models to provide answers that align with the user's or rater's existing biases and cultural myths rather than historical fact.
• Context Window Architecture: Using retrieval-augmented generation (RAG) to keep source documents intact and inspectable, as opposed to "dissolving" them into model weights via fine-tuning.

---

1. The Problem: The "Mask and the Mirror"

The speaker argues that current AI evaluation benchmarks (like the "in-character" benchmark) are fundamentally flawed. They measure fluency and personality consistency (the "Mask") rather than historical fidelity (the "Mirror").

• The Failure Mode: Models often produce "anachronistic compositing," where a figure (e.g., Alexander Hamilton) speaks with the moral framework and vocabulary of a modern pop-culture depiction rather than their own historical record.
• The Metric Gap: Current benchmarks score these fluent, anachronistic outputs highly, meaning the evaluation tools are structurally incapable of detecting the most common failure in persona-based AI.

2. The Miranda Hypothesis: Why Models Fail

The speaker posits that frontier models are trained on massive datasets where recent, culturally dominant narratives (like Broadway musicals or viral tweets) vastly outweigh the primary documentary record of historical figures.

• Mechanism: Auto-regressive token prediction compresses these sources into parameters without the ability to distinguish between a 1789 letter and a 2019 review.
• Result: The model defaults to a "salience-weighted composite." For example, the model's "Hamilton" knows he is the subject of a musical, and the model's "Lincoln" has already read the Gettysburg Address before he has written it.

3. Proposed Solution: Epistemic Simulation

To move beyond "convincing" but inaccurate personas, the speaker proposes a shift from Cognitive Simulation to Epistemic Simulation, defined by three commitments:

1. Corpus-Bounded: Reasoning is licensed only by specific primary documents.
2. Temporally Anchored: The persona is fixed to a specific moment in time; knowledge post-dating that moment is out of bounds.
3. Expert-Loop Evaluated: Outputs are judged by domain experts (historians, theologians, etc.) who understand the evidentiary record.

4. Methodology: The Prism Instrument

The speaker introduces a pre-registered evaluation instrument designed to "refract" the composite persona into distinct historical moments.

• The Experiment: Using Abraham Lincoln as the subject, the experiment tests four distinct "moments" (1847, 1858, 1860, 1862–65) across three conditions:
  • Bare Model (Control): No anchor, prone to Miranda distortion.
  • Primary Sources (The Prism): Anchored in specific documents.
  • Biography: A modern interpretive text (a "clouded" prism).
• The Rubric: A three-axis scoring system:
  • Anachronism Detection (40%): Does it avoid later moral logic?
  • Documentary Consistency (35%): Does it track the seated sources?
  • Contextual Plausibility (25%): Does it show awareness of the figure's actual knowledge at that time?
• Crucial Note: "Rhetorical authenticity" (sounding like the person) is explicitly excluded as a primary criterion to prevent rewarding "fluent but wrong" outputs.

5. Architectural Recommendations

• Context Window vs. Fine-Tuning: The speaker argues against fine-tuning for personas, as it "dissolves" the archive into weights, breaking the chain of provenance. Keeping documents in the context window preserves the "archival virtue" of the system, making it auditable and reversible.
• The Role of the Expert: The domain expert is not a runtime cost but a "build-time gate." They create the rubric and the "gold set" of vignettes used to validate the model before deployment.

6. Synthesis and Conclusion

The speaker concludes that for role-playing systems, the persona is not a property of the model weights, but an event that occurs when the model, the document, and the human convene.

Key Takeaway: "A persona system without a domain expert in its evaluation loop is a thermometer that cannot read temperature." To build responsible AI, developers must stop treating historians as "adjacent" to the process and instead bring language models into the archive, ensuring that the output is measured by its truth rather than its convincingness.