ADR-035 — Eval Sandbox & Golden Sets (metrics-first quality)

Status: Accepted (2026-07-09; P4.1 wires the minimal hallucination gate) Date: 2026-06-30 Epic: scopes a slice of #3322 (ADR-033 "Professional Specialist") Authors: Alexander K — seeking input from technical & production managers and analysts: Alex O, Alex D, Tom, Paul Related ADRs: ADR-033 (professional Specialist), ADR-032 (retrieval engine), ADR-020 (Org×Specialist isolation), ADR-008 (KB K1/K2)

Thesis: We cannot improve what we cannot measure, and we cannot trust an improvement we cannot back-test. ADR-033 is a solid theory of a professional Specialist (faithfulness gates, agentic RAG, planner/executor, calibration). This ADR argues the cornerstone must come first: an evaluation sandbox and a small set of golden scenarios. Every other ADR-033 layer is a hypothesis until eval tells us it helps. Simple things often work surprisingly well; SOTA often works surprisingly badly. Metrics are paramount; features are downstream of them.

TL;DR for managers & analysts (the ask)

We are building a repeatable "test track" for the AI Specialist: a set of golden scenarios the agent is run against, scored automatically, so that before we ship any change (e.g. "add HyDE to improve search") we can prove it helps and doesn't regress.

We need your help to define what "a good answer" is — that is domain knowledge engineers cannot fake. Concretely, two decisions for this meeting:

What format should a golden scenario take? A spectrum from "just the job + context" to "full ideal dialogue transcripts." Our recommendation is in §7. We want your call on the effort/fidelity trade.
Which 1–3 jobs-to-be-done do we seed first? Derived from docs/user-stories.md, the product manifest, and Slack threads. We propose you nominate the highest-value, most error-prone client requests.

Everything below is the reasoning. The two boxed questions in §7 and §9 are what we actually need answered.

Context

Where this sits relative to ADR-033

ADR-033 lays out 7 layers (L0 identity → L7 eval/observability) and a 4-phase roadmap. Its P0 ("看得见" — make it visible) is already shipped: edit-diff capture (#3323), a domain golden-set + LLM-as-judge skeleton (#3324), confidence calibration (#3394), correction few-shot (#3406). This ADR does not rebuild those. It takes the position that L7 (Eval & Observability) is the cornerstone the rest of the roadmap should be sequenced behind — not because the other layers are wrong, but because without eval we are guessing whether they help. We treat ADR-033 as reference, not scripture.

What exists today (build on, don't duplicate)

agent/evals/domain/ (#3324): an LLM-as-judge harness scoring correctness vs a golden answer + must-include/exclude rubric. Seed set: 11 cases (golden/ecommerce_ops.json); runner in run.py / eval_runner.py. Opt-in CI (.github/workflows/domain-eval.yml). Limitation — this is single-turn, not agentic: it takes one completion from the production system prompt (temperature 0.3) with no retrieval, no tools, no multi-turn — the harness's own docstring calls full-pipeline retrieval+tools "a future upgrade." So it measures the model's intrinsic prompt+domain correctness, not the agentic flow (retrieval + tool-calling + dialogue) an Expert actually reviews. Closing that gap is the point of this ADR — "the golden-set harness shipped" is true; "we can benchmark agentic flows" is not yet.
rag/evals/retrieval/ (#3091): retrieval-ranking eval (Hit@k/MRR). Synthetic ground truth, runs in CI against a pgvector service container. This is the pattern we extend.
rag/evals/ragas_smoke.py: a mocked RAGAS faithfulness stub — not production, not in CI.
Edit-diff capture (#3323): editRatio/wasEdited stamped on the Expert's sent message; original draft + final reply both persisted as Message rows. This is real-traffic signal we use as triage, not as committed golden data.

The driving use case

"I want to add HyDE to improve search. Should I? It might help; it might ruin retrieval. Every policy change should be tested — back-tested even better."

This single question is the product spec for the sandbox: run the scenario set with the change OFF vs ON, diff the scorecard, decide on evidence.

Decision

1. Build an in-agent eval sandbox (the cornerstone)

A Python harness (extending agent/evals/) runs the agent's real chat → retrieval → tool loop against a fixed scenario, deterministically:

Layer	In the sandbox	Why
External integrations (Shopify/Amazon/etc.)	mocked (canned responses)	not under test; non-reproducible
Retrieval (Haystack + pgvector + keyword)	real, over a seeded local KB	this is exactly what HyDE/rerank/chunking changes touch — must be real or the back-test is meaningless
LLM generation	real via OpenRouter, `temperature=0`	the thing under test for prompt/model changes; CI already has `OPENROUTER_API_KEY`. The sandbox pins `temp=0` (the current #3324 harness uses 0.3) to cut avoidable run-to-run variance

Rule of thumb: never mock the thing you want to back-test. All fixtures are synthetic → CI-native, leak-free, no customer PII (see ADR-020).

2. A golden scenario is a fixture bundle, not a Q→A pair

scenario/
  jtbd.md            # the job-to-be-done + business context + definition of done   ← ANALYST
  conversation.json  # the multi-turn user messages (how a client really phrases it) ← ANALYST
  kb/                # seeded canonical documents the Specialist may retrieve         ← ANALYST (content) + ENG (load)
  rubric.json        # ideal-outcome points + must-say / must-not-say                 ← ANALYST
  tools.mock.json    # canned integration responses                                   ← ENG (+ prod-mgr for realism)
  persona.json       # Specialist config under test                                   ← ENG (from existing Specialist)
  eval.json          # judge instructions derived from the rubric                     ← ENG

Reuse, don't fork (per team feedback). This bundle evolves the existing #3324 case schema (golden_answer + must_include / must_not_include) rather than replacing it — that (input, ideal-answer, rubric) tuple stays the canonical unit the judge scores against, and new roles (e.g. Kaito) land as new golden files under the same runner + domain-eval.yml. The sandbox does, however, need new execution paths around that core — KB seeding, tool-mock injection, and a multi-turn driver — so this is the existing harness substantially extended, not a trivial field addition and not a second harness.

3. Metrics — the product-success scorecard (full taxonomy)

Axis	Primary metric	Tier	In v1 sandbox?	Role
Task success	JTBD achieved (graded `[0,1]` + pass)	Leading	✅	Client
Groundedness	% claims entailed by retrieved KB; 0 fabrications	Leading	✅	Client / safety
Tool correctness	right tool + args + result used	Leading	✅	Client
Safety / scope	0 must-not-say / infra-disclosure / overreach	Leading	⏳ deferred (prod deny-list #3054 is the floor; re-enters via the spot-check set)	Compliance
Expert leverage	send-verbatim rate, `editRatio` (#3323)	Lagging	❌ (prod)	Expert
Autonomy	auto-resolve rate vs escalation	Lagging	❌ (prod)	Production mgr
Client satisfaction	CSAT / re-ask rate / resolution time	Lagging	❌ (prod)	Client
Calibration	confidence vs actual edit rate (#3394)	Lagging	❌ (prod)	Production mgr / Superadmin
Cost / latency	$/resolved-turn, p95; tokens/turns	Both	📊 readout only (not a gate)	Superadmin

v1 leading axes scored in the sandbox: task success, groundedness, tool correctness. Rationale for the rest is in §8.

4. Rigorous metric definitions + pass rules (the pass rule is the metric)

Axis	Definition	Computation	v1 pass rule
Task success	Did the draft accomplish the scenario's JTBD?	LLM-judge, rubric-point coverage	graded + advisory (report score/delta; human decides while we calibrate)
Groundedness	Is every factual claim supported by the retrieved context?	claim decomposition + entailment (RAGAS `faithfulness`)	HARD GATE on fabrication — a draft inventing facts never ships. Unsupported-because-retrieval-missed is reported as a retrieval gap, not a generation failure (must be distinguished, or the metric lies).
Tool correctness	Right tool(s), right args, result used?	deterministic structural-trajectory + exact/normalized arg match; optional judged check for result-use	graded + advisory

This is the hybrid pass-rule philosophy: gate the one unambiguous failure (hallucination), advise on the judgment calls until we've built confidence in the judge (§5), then tighten to gates as confidence grows.

5. Trust comes from the human rubric + judge consistency — not a statistical gate

Calibration lives primarily in the human-authored golden set: the golden_answer + must_include / must_not_include rubric turn scoring from opinion into concrete, checkable criteria. That is what makes the target trustable. The judge is an LLM applying that rubric, so:

Lean on the rubric gates (must_include / must_not_include) over the fuzzy correctness float — the rubric is the part humans make genuinely checkable.
Require judge consistency, not perfection: pin the judge model + version + rubric, because we trust deltas between policy arms, not absolute scores — a consistent bias cancels out of the difference (see §6).
Keep a lightweight spot-check: on ~10–15 cases, confirm the judge grades them the way a human would; re-check whenever the judge model or prompt changes.

No heavy statistical calibration gate in v1 — the controlled-experiment design (§6) makes deltas trustworthy without one. (This answers "manual or LLM-as-judge?" — humans author the rubric, the LLM applies it, humans spot-check it.)

6. Back-test decision rule

Run the scenario set with a policy OFF (baseline) and ON. Report per-axis deltas with a noise band (LLM nondeterminism makes tiny deltas noise). Ship rule: improves ≥1 axis, regresses none beyond the band. No single composite score — a balanced per-axis diff so a groundedness regression can't be hidden by a task-success bump.

Why a noisy judge is still enough — the controlled-experiment argument. Both policy arms run over the same scenarios with the same pinned judge, so any consistent judge bias hits both arms equally and cancels out of the difference. The delta (A vs B) is therefore trustworthy long before the absolute score is — you need a judge that is consistent, not perfectly accurate. This is precisely why synthetic scenarios + an uncalibrated-but-consistent LLM judge are enough to experiment with agentic policies today, before we have production outcome data. Corollary: trust deltas now; earn absolute claims ("the agent is X% good") only as the set grows and is validated against real signal.

Tooling positions

RAGAS: adopt narrowly as the groundedness scorer behind a stable interface (build on the existing stub). It does not own task-success or tool-correctness. Pin the version. It is itself an uncalibrated judge → subject to the same rubric-gating + spot-check (§5).
LangFuse: adopted as a sync-copy + runner/dashboard layer (P4.1). The golden truth source stays first-party — the platform-global EvalGoldenScenario table (ADR-020); a derived copy may sync to a Langfuse dataset for run_experiment + trend dashboards. The PII/data-sovereignty objection is addressed by keeping the truth source in-DB — Langfuse would hold only a synced copy. It remains an experiment-tracker/dashboard layer, not a metric and not the gate authority (the publish-blocking state machine + EvalRun record live API-side; Langfuse is at most a runner/score layer it calls). It is not required for the v1 gate (design D-3: the eval-gate worker judges directly off the /eval/run CaseResults; the Langfuse dataset sync + run_experiment + trend dashboards are deferred advisory, langfuse_experiment_run_id may be null). What P4.1 actually wires is the minimal hallucination gate — the single unambiguous hard gate (§5) plus the gate infrastructure (EvalRun table, eval-gate worker, publish state-machine PASS/BLOCK/ERROR fork); the fuller Langfuse dashboard / experiments UI is deferred (non-goal), not shipped.

7. What we need from managers & analysts (golden-set format)

This is the headline discussion item. What is the unit a domain expert contributes? Three points on a spectrum:

Option	Analyst provides	Eng provides	Effort	Fidelity
1 — Job + context + rubric	JTBD, business context, "what good looks like" bullets, red lines	conversation, seeded KB, mocks, judge	Low	Medium (rubric-based)
2 — + canonical facts + realistic phrasing (recommended unit)	everything in 1, plus the actual KB facts the Specialist must rely on and realistic multi-turn user messages	mocks, judge wiring	Medium	High
3 — Full golden dialogue pairs	complete multi-turn transcripts with the ideal Specialist reply per turn	mocks	High	Highest (enables reference scoring) — but brittle: a KB update can invalidate the "ideal" reply

❓ Decision 1: golden-set format

Recommendation: Option 2 as the standard contribution unit (job + context + canonical facts + realistic user phrasing + rubric), and reserve Option 3 (full ideal replies) only for the ~10–15 spot-check cases (§5), where verbatim references help confirm the judge grades like a human.

Why: the rubric + the canonical facts + how-clients-actually-ask are exactly what only domain experts can supply and engineers cannot fake. Full verbatim transcripts everywhere are expensive and brittle (they rot when the KB changes), and we only strictly need them where we calibrate the judge.

We want your verdict on this trade.

Resolved (2026-07-09, P4.1 — closes OD-13): the light "job + context + rubric" unit was adopted. The 3 existing golden JSON sets (ecommerce_ops / marketing_content / policy_behaviors) are that shape — {role, cases[{question, golden_answer, must_include, must_not_include, context?}]} — and are now seeded into the platform-global eval_golden_scenarios table. Full-dialogue transcripts (Option 3) were not chosen (they rot when the KB changes; reserved only for the future judge-anchor tier). See the P4.1 eval gate.

Where your input maps to existing artifacts (so this isn't from scratch): JTBDs come from docs/user-stories.md; conversation flows/decision-points from docs/design/USER_FLOWS.md; canonical facts from real KB + Slack threads; the backlog/priority from docs/FUTURE_REQUIREMENTS.md.

Triage signal we already have: real {agent_draft → expert_sent} pairs with high editRatio (#3323) tell us which requests the agent struggles with — i.e. where a golden scenario is most worth your time. Traffic prioritizes the curation backlog. For v1, scenarios are authored synthetically (PII-free, committed, CI-native). Seeding cases directly from real corrections/traces is attractive but carries customer-PII / client-KB exposure under ADR-020, so it is out of scope for v1 and deferred to a later decision (see §9).

8. Why these metrics and not the others

The lagging metrics (expert leverage, autonomy, CSAT, calibration) are not excluded from the product — they are already (partially) instrumented (#3323, #3394, /ops/analytics). They are excluded from the v1 sandbox for three reasons:

Not back-testable. You only observe CSAT / leverage / autonomy after shipping to real traffic over weeks. They cannot answer "should I merge HyDE?" before the merge. The sandbox exists precisely to give a pre-ship signal.
Confounded. Real-world outcomes depend on Expert behavior, client mix, channel, season — you cannot attribute a CSAT move to one agent policy change. The sandbox isolates the agent so a delta means what it says.
Leading vs lagging is the whole design. The sandbox's job is to produce leading indicators that predict the lagging ones. A deliberate later step is to validate that linkage (does a sandbox groundedness win actually move prod editRatio?). Folding lagging metrics into v1 would conflate the predictor with the outcome.

Safety/scope specifically is deferred from v1 scoring (not from the taxonomy): production already enforces a floor via the deny-list / infra-disclosure guard (#3054), and safety is the natural first addition to the hand-curated spot-check set. Cost/latency is a readout in the sandbox (tokens/turns/$), useful for comparing policies, but not a gate.

9. Open questions for discussion

❓ Decision 2: which JTBDs seed the first 1–3 scenarios?

Nominate the highest-value, most error-prone client requests (ideally ones with high real-traffic editRatio). Each becomes one fixture bundle.

Who owns the spot-check labels? The judge is trusted via a lightweight human spot-check (~10–15 cases graded by a domain expert, §5), not a heavy calibration run. Roughly how much analyst time can we get?
Cadence: how often do scenarios get reviewed/refreshed as the KB and product change (to avoid golden-set rot)?
Seeding from real data (deferred — decide later): the cheapest realistic cases are approved/edited Expert replies (#3380 corrections) joined via kb_retrieval_event (#3189), with session traces backfilling the multi-turn side. Attractive as a future source, but it involves customer PII + client KB under ADR-020 (anonymization? where does the data live? can it be committed?). v1 is synthetic-only; the real-data path and its PII handling are a separate, later decision.

Scope & phasing

Phase 1 (this ADR): the sandbox (Approach A) + 1–3 hand-built golden scenarios + calibrated judge on the 3 leading axes. The deliverable bar is "a golden scenario the Specialist reliably passes," not feature completeness.
Phase 2: an automated scenario generator with human curation (lower the per-scenario cost).
Phase 3+ (validated by Phase 1, not assumed): faithfulness gate as an online L4 check, agentic/HyDE retrieval, planner/executor — each shipped only if the sandbox shows it helps. Confidence calibration, HITL learning, RL are explicitly secondary to reaching golden-set objectives.

Known limitations & risks

Named deliberately — an eval whose weak points are explicit is more trustworthy than one that hides them.

The proxy is not yet validated. That sandbox (leading) metrics predict production (lagging) outcomes is a hypothesis, not a proven fact. Mitigation: a first linkage check — correlate judge scores against real editRatio (#3323) on a sample of logged questions — before treating the sandbox as authoritative. Until it passes, the sandbox advises, it does not gate.
v1's small N cannot decide policy questions. With 1–3 scenarios and a noisy judge, a HyDE-sized delta is within noise. Mitigation: v1 proves the machinery runs and discriminates; grow the set (toward the 50–100/role the golden set already targets) and size N from the effect we want to detect before claiming decision authority.
Not fully deterministic. Even at temperature=0, OpenRouter routing and the external embedding/rerank hops introduce variance. Mitigation: pin model IDs + versions, snapshot the seeded-KB embeddings, and run each config K times reporting mean ± spread — that measured spread is the noise band of §6.

Consequences

Positive: a pre-ship, evidence-based gate for every quality change; a shared definition of "good" co-owned by domain experts; reuse of the #3091/#3324 chassis; no new PII surface.
Costs: real analyst / manager time to author scenarios + spot-check labels (the deliberate investment); sandbox maintenance as the agent evolves; golden-set rot if not refreshed.

Non-goals

Not a production observability platform (that's the existing PG event store + Prometheus).
Not full-stack e2e testing (the NestJS↔agent contract is out of scope — it isn't where policy changes live).
Not fine-tuning / RLHF (per ADR-033 non-goals).
Not building from real client data in v1 (PII).

TL;DR for managers & analysts (the ask)​

Context​

Where this sits relative to ADR-033​

What exists today (build on, don't duplicate)​

The driving use case​

Decision​

1. Build an in-agent eval sandbox (the cornerstone)​

2. A golden scenario is a fixture bundle, not a Q→A pair​

3. Metrics — the product-success scorecard (full taxonomy)​

4. Rigorous metric definitions + pass rules (the pass rule is the metric)​

5. Trust comes from the human rubric + judge consistency — not a statistical gate​

6. Back-test decision rule​

Tooling positions​

7. What we need from managers & analysts (golden-set format)​

❓ Decision 1: golden-set format​

8. Why these metrics and not the others​

9. Open questions for discussion​

❓ Decision 2: which JTBDs seed the first 1–3 scenarios?​

Scope & phasing​

Known limitations & risks​

Consequences​

Non-goals​