Runbook — authoring a golden set

Audience: analysts / production managers (who define "a good answer") + engineers (who wire it up). Governing decision: ADR-035. Format = Option 2 (agreed). Where cases live: agent/evals/domain/golden/<role>.json, run by the domain-eval harness (#3324).

A golden set is a set of test cases for one Specialist role. Each case is a synthetic client message + a reference of the ideal reply + a checkable rubric. We run the Specialist against them and an LLM-as-judge scores the replies, so we can prove a change (new prompt, retrieval policy, model) helps before shipping.

Synthetic only. Cases are hand-authored and PII-free (ADR-035). Do not paste real customer conversations or client KB. Use real cases only as inspiration for what to write.

Where does my case go? Behavior (clarify-vs-proceed, don't fabricate, escalate when out of scope) → golden/policy_behaviors.json — it runs against every role automatically, don't duplicate it per role. Domain knowledge (facts, policies, red lines specific to the role) → the role file. Analyst sessions therefore only need to author facts + red lines.

1. The case format (Option 2)

{
  "role": "ecommerce_ops",
  "cases": [
    {
      "id": "eco-001-order-not-shipped",
      "question": "I ordered 3 days ago and it still says processing. When will it ship?",
      "golden_answer": "Acknowledge the delay, explain 'processing' = confirmed but not yet handed to the carrier, give the stated handling/dispatch window, and offer to check the specific order. Do NOT invent a ship date or tracking number.",
      "must_include": ["processing means not yet shipped", "offer to check the order"],
      "must_not_include": ["a fabricated tracking number", "a guaranteed exact ship date with no data"]
    }
  ]
}

Field	What it is	Who authors
`id`	stable `role-NNN-slug`	eng
`question`	the client's message, phrased the way a real client would	analyst
`golden_answer`	the professional intent of a correct reply (not verbatim)	analyst
`must_include`	points a correct reply must cover (rubric gate)	analyst
`must_not_include`	claims/actions a reply must never make (red lines)	analyst
seeded KB / mocked tools / persona	the sandbox fixtures around the case	eng (facts from analyst)

Option 2 means: analysts provide the job, the context, the canonical facts, realistic phrasing, and the rubric. We only write full ideal-reply transcripts for the ~10–15 judge spot-check cases (they're costly and go stale when the KB changes).

2. Rubric-quality checklist (the part that decides trust)

The judge scores against your rubric, so ambiguous rubrics = noisy, untrustworthy scores. (Empirically, cases like eco-015 (now pol-009) with vague rubrics swung wildly across identical runs — see the noise characterization.) For every case:

Each must_include item is one concrete, checkable idea ("offer to check the order"), not a vague quality ("be helpful").
Each rubric item is independently judgeable yes/no — the judge is literally asked "does the reply do X?" per item, so an item that bundles two ideas ("apologizes and offers a refund") must be split.
must_not_include items are specific red lines ("a fabricated tracking number"), not general ("anything wrong").
The golden_answer describes intent, and every must_include item is actually satisfiable from it.
A human grader would agree on pass/fail without debate. If two people could disagree, tighten it.
Compliance / safety cases: put the red line in must_not_include (this is where must-not-say lands).

3. What is testable now (and how to choose)

✅ Rubric gates (single-turn): clarify-vs-proceed behavior, must-not-say / compliance red lines, "don't fabricate numbers when you have no data." Score meaningfully with no extra setup.
✅ Grounding faithfulness via context (injection): put the synthesized canonical fact(s) in the case's context field; it's injected into the prompt as a knowledge block. Tests whether the model uses the given fact and doesn't fabricate. See mkt-003 in golden/marketing_content.json. Right choice for ephemeral/real-time facts (live campaign numbers, today's status) that wouldn't live in a KB.
✅ Seeded-KB via REAL retrieval (#3873, shipped): the role declares kb_dir (a folder of fixture docs, {doc_id,title,description,filename,content} JSON) and the case sets kb_gold_docs — the harness seeds a real pgvector and runs the production retrieval pipeline; the draft is judged on what retrieval actually surfaced, and failures are split into retrieval_miss (right doc in kb/, not retrieved) vs hallucination (doc retrieved, claim unsupported). Right choice for document-shaped knowledge (policies, terms, product facts). This is also what back-tests retrieval changes (HyDE / rerank / chunking). See archetype 4 in golden/TEMPLATE.json.example and the README's "Seeded-KB mode" section.
⏳ Still future — tool mocks: cases that depend on tool/integration calls (order lookups, live API data) are not testable yet; don't author them.

Rule of thumb: doc-shaped fact → kb/ fixture + kb_gold_docs; real-time figure → context; behavior → policy suite.

4. Add a role & run it

Start from the template — don't author from a blank page:

# 0. Copy the annotated template (or the campaign starter, if that's your role)
cp agent/evals/domain/golden/TEMPLATE.json.example agent/evals/domain/golden/<role>.json.example
#    TEMPLATE.json.example        — generic, one annotated case per archetype
#    hp_campaign_ops.json.example — Amy/Capital Launchpad starter (3 JTBDs pre-structured)
# Fill every [PLACEHOLDER], delete the _comment/_HOW_TO_USE keys, THEN rename
# .json.example -> .json. The runner executes every *.json in golden/ (and the
# policy suite auto-expands to the new role), so only rename when facts are real.

What the current harness tests per archetype: task-success and must-not-say cases run as-is; ephemeral facts go in context (injection — tests faithful use); doc-shaped facts go in kb/ fixtures with kb_gold_docs (real retrieval — also catches retrieval misses; see §3). Tool/integration behavior is not testable yet — don't author cases that depend on tool calls.

# 1. Create the golden file
$EDITOR agent/evals/domain/golden/<role>.json      # role + cases[]

# 2. Run locally (needs a key; skips cleanly without one)
cd agent
export OPENROUTER_API_KEY=...
python -m evals.domain.run --role <role>

# 3. Or in CI: workflow_dispatch the `domain-eval` workflow with input role=<role>,
#    or label a PR `run-domain-eval`.

For a baseline or a policy A/B decision, run with the judge denoised and repeat:

DOMAIN_EVAL_JUDGE_VOTES=5 python -m evals.domain.run --role <role>   # ×3, then compare

5. Reading results (do this right or the numbers lie)

Trust mean_correctness averaged over K≥3 runs, not a single run's pass_rate — single-run pass counts are noisy (see findings doc).
For A/B: run both arms on the same set with the same pinned judge, and trust the delta, not absolute scores (consistent judge bias cancels from the difference).
pass/fail per case is for diagnostics (which cases regressed), not the headline metric.

6. Back-testing a prompt change (the PM workflow)

You changed (or want to change) a Specialist prompt and want evidence before shipping.

Edit the prompt on a branch. Prompts live in agent/prompts/<role>.txt (e.g. ecommerce_ops.txt). Easiest non-eng path: edit the file in the GitHub web UI — it creates a branch + PR for you.
Run BOTH arms in the same batch (levels drift day-to-day; only same-batch comparisons are valid). From a terminal, or ask any engineer:
```
gh workflow run domain-eval.yml --ref dev           -f judge_votes=5   # baseline
gh workflow run domain-eval.yml --ref <your-branch> -f judge_votes=5   # candidate
```
(Or add the run-domain-eval label to your PR for the candidate arm.)
Download the two domain-eval-report artifacts (each run's page → Artifacts, or gh run download <run-id> -n domain-eval-report).
Let the referee decide:
```
cd agent
python -m evals.domain.compare baseline/report.json candidate/report.json
```
Read the verdict: IMPROVED / REGRESSED / INCONCLUSIVE (95% CI on the mean paired Δ), plus blockers — any case that newly violates a must-not-say rule blocks regardless of the average. INCONCLUSIVE at small deltas is a real answer: the change doesn't move quality measurably — decide on other grounds.

For a high-stakes call, repeat step 2–4 (3 pairs of runs) and check the verdicts agree. Never decide from two job summaries eyeballed side by side.

7. Grow the set

Target 50–100 cases/role, ≥500 aggregate before trusting absolute averages. Prioritise from real friction (high-editRatio requests, #3323) — as inspiration for what to author, not as data to copy in.

See also: ADR-035 · noise characterization · harness README.

1. The case format (Option 2)​

2. Rubric-quality checklist (the part that decides trust)​

3. What is testable now (and how to choose)​

4. Add a role & run it​

5. Reading results (do this right or the numbers lie)​

6. Back-testing a prompt change (the PM workflow)​

7. Grow the set​