Eval instrument v2 — design

Date: 2026-07-02 · Status: approved (brainstorm with Alex) Context: ADR-035 · Phase-0 noise characterization · PRs #3752 (merged), #3757 (open, this stacks on it)

Problem

Phase-0 characterized the domain-eval harness and left four structural weaknesses that patching (K-vote) mitigates but doesn't fix:

1. The judge grades the raw agent JSON envelope ({reply, confidence, …}), not the reply a customer would read — a suspected contributor to the "intrinsic" noise floor we measured.
2. Our doctrine is "trust paired deltas," but there is no tool that computes them — back-tests are eyeballed aggregates.
3. The judge's holistic correctness float is the least reliable unit of LLM judgment; per-item binary checks are the reliable unit, but a pure item-fraction KPI would be too coarse (2-item rubrics → {0, ½, 1}).
4. 10 of 16 "ecommerce" cases actually test role-agnostic behavior (clarify-vs-proceed, no-fabrication, escalation) — conflating policy with domain knowledge and bloating every future role file.

Plus one open review finding on #3757: with transient judge failures, majority is computed over surviving votes, so one noisy vote can fail a case ("reduced quorum becomes majority").

Scope

Step	What	Lands in
0	Quorum guard	push to #3757 (open review response)
A	Judge the extracted reply, not the envelope; file the eco-010/016 bug	PR-1
C	Judge protocol v2: item-level binary gates + graded alignment KPI (hybrid)	PR-1
B	compare.py — paired-delta back-test tool	PR-2
D	Policy/role suite split	PR-2
V	Validation: re-baseline 3× K=5 on both roles + policy suite	after PR-1+2 merge
Docs	Post-validation docs & guides finalization	small docs PR after V

Grouping rationale: PR-1 contains everything that changes score semantics → one re-baseline. PR-2 is tooling/content with no semantic impact; compare.py consumes the v2 report schema so it follows PR-1.

Non-goals (deferred): adaptive K-vote (cost optimization — matters at 50+ cases); linkage retrospective (judge scores vs editRatio — separate project, still the deepest untested assumption); KB-slot prompt fidelity (mirror production KB formatting — belongs to the ADR-035 sandbox work); RAGAS integration.

Design

0. Quorum guard (#3757)

More than half of the configured _JUDGE_VOTES must return a valid dict, else the case becomes an explicit error (error="judge quorum not met (1/3)"). Run continues (fail-open preserved). Prevents a single surviving vote from acting as a majority.

A. Envelope extraction

Before judging: try json.loads(candidate); if the result is a dict with a "reply" key, the judge receives str(reply). The raw envelope is kept in CaseResult.candidate for diagnostics. Any parse failure → judge the raw string (today's behavior, fail-open).

Bug filing: GitHub issue for the over-clarification behavior (eco-010/eco-016 fail 12/12 runs at 0.0–0.5 correctness; agent interrogates fully-specified briefs; suspect clarify-first policy prompt; evidence = noise-characterization doc).

C. Judge protocol v2 (hybrid — approved with the design)

One judge call per vote returns structured item-level output:

{
  "items":      {"offer to check the order": true, "explain processing": false},
  "violations": {"a fabricated tracking number": false},
  "alignment":  0.85,
  "rationale":  "one sentence"
}

K-vote aggregation (over surviving votes, quorum enforced):

• must_include item covered iff a majority of votes says true; must_not_include violated iff majority says true.
• alignment → median ⇒ becomes correctness (the continuous KPI; K-averaging stays smooth).
• verdict = all items covered ∧ no violations ∧ median alignment ≥ _PASS_THRESHOLD.

Report schema is a superset of v1: missing_required / forbidden_present keep their names (derived from item majorities); a new items block carries the per-item detail. Cost unchanged — same call count, bigger response.

Why hybrid: binary per-item questions are the reliable unit of LLM judgment; the graded alignment float preserves the averaging-friendly continuous KPI Phase-0 proved out. Pass/fail authority moves entirely onto the reliable binary gates.

Consequence: v2 correctness (alignment) is not comparable with Phase-0 numbers → re-baseline required (step V); old numbers marked non-comparable in the findings doc.

B. compare.py

python -m evals.domain.compare baseline.json candidate.json [--role R] [--fail-on-regression]

• Pairs cases by (role, id); intersection only; warns on set mismatch.
• Per-case Δcorrectness table; counts improved/regressed/tied.
• Bootstrap 95% CI on the mean paired Δ (10k resamples, fixed seed, stdlib-only).
• Gate-flips (pass→fail / fail→pass) listed; a new rubric-violation failure is a hard blocker regardless of the mean.
• Verdict per ADR-035 §6: IMPROVED / REGRESSED / INCONCLUSIVE (CI vs 0).
• Reads v1 reports too (no items field → v1 mode), so historical artifacts stay mutually comparable.
• Output: markdown to stdout; --fail-on-regression → exit 1 on REGRESSED or gate regression (CI-gateable later).

D. Policy/role suite split

• golden/policy_behaviors.json with "suite": "policy": the role-agnostic behavior cases move here — eco-007…016 → pol-001…010, plus mkt-002 (proceed-on-complete-brief).
• Role files keep domain cases only: ecommerce_ops.json → eco-001…006; marketing_content.json → mkt-001 (compliance) + mkt-003 (grounded).
• Runner expands the policy suite against every role discovered in golden/ (a behavior is a property each persona must satisfy; catches role-prompt regressions), reported as policy@<role>. --roles flag restricts for cheap runs. Decision: all-roles default (approved with the design).
• Runbook gains a "where does my case go?" rule: behavior → policy suite; facts/red-lines → role file. Analyst sessions (e.g. Veena/campaign) then only author facts + red lines.

V. Validation

After PR-1 and PR-2 merge: 3 × domain-eval runs at DOMAIN_EVAL_JUDGE_VOTES=5 covering both roles + policy suite. Success = spread of run-level mean correctness ≤ Phase-0's v1 K-vote band (0.031) and no case erroring on quorum. These numbers become the v2 baseline.

Docs & guides finalization (post-validation — explicit user requirement)

Ships as a small docs PR once V's numbers exist:

• noise-characterization doc: "protocol v2 baseline" section with the new numbers; v1 tables marked non-comparable (holistic-judge era).
• harness README: v2 judge schema, policy-suite semantics, compare.py usage, updated env-knob table.
• golden-set runbook: "where does my case go?" decision rule; item-writing guidance strengthened (each item must be independently judgeable yes/no); compare-based back-test workflow (baseline → change → compare → verdict).
• Non-number-dependent doc edits ship inside PR-1/PR-2 themselves; only the baseline-numbers section waits for V.

Testing (all tokenless, mocked)

• Envelope extraction: JSON-with-reply / plain prose / malformed JSON.
• v2 aggregation: item majorities, violation majorities, median alignment, quorum failure → case error, even-K ties (split item flag ≠ majority).
• compare.py: synthetic report fixtures → IMPROVED / REGRESSED / INCONCLUSIVE / gate-flip blocker / v1-schema tolerance / disjoint case sets.
• Runner policy expansion: policy suite × discovered roles; --roles filter.
• Existing mocked tests migrated to the v2 judge return shape.

Error handling

• Malformed judge JSON in one vote → vote dropped, counts against quorum.
• All votes failed → case error, run continues (fail-open).
• Compare with disjoint sets → warn + intersect; empty intersection → exit with message.
• Policy suite with zero role files → clear error.

Sequencing & branches

1. Now: quorum guard → feat/domain-eval-context-grounding (#3757).
2. PR-1 feat/domain-eval-judge-v2 (this branch; stacked on #3757, rebases onto dev when #3757 merges): A + C + tests + non-number doc edits.
3. PR-2 feat/domain-eval-compare-policy (stacked on PR-1): B + D + tests + non-number doc edits.
4. V via workflow_dispatch ×3; docs PR with v2 baseline.