Advanced · What You Can Do Now

Stop saying “it feels better”: build your first eval set

The first three squares you checked by eye. Fine at small scale. But you’re about to hit the classic trap: you change a prompt, these three cases get better, those two quietly get worse, and you don’t notice. Anthropic says “not evaluating is running naked.” This square puts clothes on your Agent.

Hands-on track, fourth square: it can work now — make every change evidence-based.

Know what you want it to do for you

Talks like a human, reliably

Actually gets work done

Better or worse, measurably

Runs long without losing memory

The process is reproducible

See it first · One change, five cases, five different fates

An eval set isn’t mysterious. It’s a pile of fixed inputs, each with a “what counts as pass” criterion. The magic is the moment you score: what you thought was an across-the-board win is often three up, two down. Without an eval set, those two drops stay invisible forever.

Same cases, before and after a prompt change

Regular weekly report, three-part structure correct

Pass

Monthly report with numbers, all numbers correct

Pass

Extra-long doc, missed a key conclusion in the second half

Fail

Short doc, normal summary

Pass

Missing info, made up a number

Fail

3 / 5 cases passed

Baseline of the first prompt: 3 / 5. Long-doc and missing-info cases failed. Remember this number — it’s the reference for every later change.

Three eval pitfalls this chapter already covered

First, quietly editing cases while you score. Change a case and the before/after scores aren’t comparable. Adding cases is fine; changing cases means re-running the full baseline.

Second, looking only at the total. Is 4/5 better than 3/5? First check whether the failing case used to pass.

Third, writing the criterion as “high-quality answer”. That can’t be judged. Write “includes next week’s plan” or “numbers match the source” — sentences you can decide at a glance. Only when you can’t judge that way does LLM-as-Judge come in.

Hands-on checklist · Pick one, start it, tick it off

This Chapter's Hands-on Checklist

0 / 3 done

Collect 10 real cases

15 min Everyone

Pick 10 inputs you’ve actually used and save them in a file, one per line. The mix matters: seven everyday, three nasty. Pull the nasty ones from the M1 five-run streak and the M2 stuck records — the inputs that already crashed it are the most valuable.

What counts as done

The file exists, all 10 are real inputs (not ones you just made up), and at least 2 are ones that already crashed it.

Write a pass criterion for each, run a baseline

1 hour If you want changes to be evidence-based

Each case gets one “what counts as pass,” written so you can decide at a glance: “output is exactly three sections,” “the conclusion includes XX,” “no numbers invented beyond the source.” Then run all 10 and write down the first baseline score.

What counts as done

Anyone with your criteria and the output would judge the same as you. Baseline is recorded, e.g. 6 / 10.

Change one prompt version, let the score talk

Half a day If you want to keep iterating

Change the prompt against the failing cases. Don’t touch a single case. Re-run the full set and compare. Watch two things: how much the total moved, and whether any case that used to pass now fails. A regression isn’t shameful — write down which case and why. You’re already doing the real engineering Anthropic talks about.

What counts as done

You can say a sentence like: “This change moved the score from 6/10 to 8/10, at the cost of #4 going from pass to fail, because the new rule is too conservative.” If you can say that, M3 is standing.

Log the baseline in the Build Log

For M3, record: where the eval set lives, what the baseline is, and what the first comparison run concluded. From this square on, when you say “it got better,” you have a number in hand.

Fill in M3