Agent Evaluation
Three Grader Types: Code, Model, Human
Graders are the judges of your evaluation system. Choose the wrong Grader and your results become unreliable. Production experience shows that combining all three Grader types — each playing to its strengths — yields the best outcomes. There's no silver bullet, but there is an optimal combination.
Grader 1: Code Grader
Code Grader
Automatically evaluates correctness using programmatic logic
| Dimension | Details |
|---|---|
| Methods | String matching, regex, static analysis (AST), unit test pass/fail, tool-call validation (correct API and parameters) |
| Pros | Fast (millisecond-level), near-zero cost, fully objective and reproducible, ideal for CI/CD automation |
| Cons | Too strict on valid variants (e.g., fails on different variable names), lacks semantic understanding, cannot evaluate subjective quality |
| Best for | Tasks with clear correct answers: code compilation, API return value checks, file format compliance, mathematical accuracy |
Grader 2: Model Grader (LLM-as-Judge)
Model Grader (LLM-as-Judge)
Uses another LLM to score against a rubric
| Dimension | Details |
|---|---|
| Methods | Feed the Agent's output and a scoring Rubric to a judge LLM, which returns a score and rationale based on predefined criteria |
| Pros | Can evaluate subjective quality (writing style, logic, creativity); understands intent beyond literal matching; flexibly adapts to different task types |
| Cons | Higher cost (each evaluation consumes tokens); potential scoring bias; requires carefully designed rubrics; results not fully reproducible |
| Best for | Open-ended tasks: research report quality, code style assessment, conversational naturalness, summary completeness and accuracy |
Key Tip: Define Your Rubric Score-by-Score
Vague criteria like "rate output quality from 0 to 1" are nearly useless. A good Rubric should spell out:
0 = Completely fails to answer the question, or contains serious factual errors
0.3 = Answers the question but misses critical information
0.7 = Complete and accurate, but poorly organized or redundant
1 = Complete, accurate, concise, and clearly structured
0 = Completely fails to answer the question, or contains serious factual errors
0.3 = Answers the question but misses critical information
0.7 = Complete and accurate, but poorly organized or redundant
1 = Complete, accurate, concise, and clearly structured
Grader 3: Human Grader
Human Grader
Direct evaluation by domain experts
| Dimension | Details |
|---|---|
| When to use | Calibrating scoring criteria early in evaluation; when LLM judgment shows clear blind spots; high-stakes domains requiring expert judgment (medical, legal, financial) |
| Pros | Highest-quality feedback; can identify blind spots in automated methods; provides deep insights for improvement |
| Cons | Not scalable (human time is limited); slow (hours to days); expensive; inter-rater disagreement |
| Best for | Periodic spot-checks to calibrate Model Grader accuracy; establishing evaluation criteria for new domains; final validation for critical product decisions |
Hybrid Strategy: Recommended Workflow
Three-Layer Evaluation System
1
Code Grader as Foundation
Cover all deterministic scenarios first: compilation, format validation, API call accuracy. Fast and cheap.
2
Model Grader for Extension
Cover subjective scenarios: output quality, logical coherence, user experience. Well-designed Rubrics are key.
3
Human Periodic Calibration
Periodically spot-check whether Model Grader scores are drifting, correct biases, and ensure the evaluation system remains trustworthy.
Core Logic: Code Grader sets the floor (no catastrophic failures), Model Grader raises the ceiling (output quality), Human Grader calibrates the judges (ensures fairness). All three are indispensable.
Real-World Cases
Descript
Video Editing Agent
Descript's evaluation system is built around three dimensions, each scored independently and combined for overall assessment:
Did Not Break -- Video intact after edit
Did the Right Thing -- Instructions correctly executed
Did It Well -- Professional editing quality
Bolt AI
Code Generation Agent
Bolt AI combines all three Grader types to form a complete evaluation pipeline:
- Static Analysis (Code Grader): checks whether generated code compiles and passes lint
- Browser Agent Testing (Code Grader): automatically opens the generated page to verify the UI matches expectations
- LLM Judge (Model Grader): evaluates code quality, readability, and adherence to best practices
There's no silver bullet — combining all three Grader types works best. Code Grader holds the floor, Model Grader raises the ceiling, Human Grader calibrates accuracy. This is the best practice validated by multiple top-tier Agent teams.