Build a Basic LLM Judge

Key Concepts

• Automated Judge: A system designed to evaluate code or content outputs based on predefined criteria.
• Zod: A TypeScript-first schema declaration and validation library.
• LLM-as-a-Judge: Using Large Language Models to evaluate subjective qualities like brand fit or toxicity.
• Few-Shot Prompting: Providing a model with examples of inputs and desired outputs to improve performance.
• Chain of Thought (CoT): A prompting technique requiring the model to explain its reasoning process before providing a final answer.
• Temperature: A hyperparameter that controls the randomness of model output; lower values (e.g., 0) increase consistency.

---

1. Architecture of an Automated Judge

The system is divided into two primary evaluation layers:

• Objective Checks: Utilizes Zod for strict JSON schema validation and standard utility functions for hard-coded rules. These result in binary pass/fail outcomes.
• Subjective Checks: Employs an LLM to evaluate nuanced criteria such as brand alignment or toxicity.

2. Model Selection and Configuration

Choosing the right model involves balancing reasoning capability, speed, and cost.

• Strategy: Developers can scale down models for routine checks or use a "hybrid" approach—cheap models for daily CI/CD checks and powerful models for final release testing.
• Model Choice: The creators utilize Gemini 1.5 Flash (noted as "Gemini 3 Flash" in transcript) for its efficiency.
• Configuration:
  • Temperature: Set to 0 for standard models to ensure deterministic, consistent results.
  • Reasoning Models: For models with native reasoning capabilities, keep the default temperature but set the "thinking level" to high.

3. Evaluation Methodology

To avoid the "politeness bias" (where humans and LLMs tend to cluster numeric scores in the middle), the system strictly uses a binary pass/fail output rather than a 1–10 scale.

• Data Structure: A unified TypeScript EvalResult type is used for both rule-based and LLM-based evaluations. It contains:
  • label: The pass/fail status.
  • rational: A field for the judge to provide a textual explanation of its decision.

4. Prompt Engineering Framework

To ensure the LLM acts as an effective judge, the following framework is recommended:

• Persona Assignment: Define the LLM as an expert in the specific domain being evaluated.
• Strict Rubric: Provide clear, non-ambiguous grading criteria.
• Chain of Thought: Explicitly instruct the model to output its reasoning before the final label.
• Few-Shot Prompting: Include specific examples of pass and fail outputs within the prompt to guide the model.
  • Crucial Constraint: Ensure that few-shot examples are strictly separated from the actual test data to prevent "cheating" or data leakage.

5. Synthesis and Conclusion

The automated judge framework provides a scalable, consistent way to evaluate software outputs by combining rigid schema validation with intelligent, LLM-driven subjective analysis. By prioritizing binary outcomes and structured reasoning (Chain of Thought), developers can reduce ambiguity in testing. The next logical step in this workflow, as noted by the creators, is alignment—verifying that the automated judge’s decisions correlate accurately with human judgment.