Domain-eval noise characterization (Phase 0)

Date: 2026-07-02 · v2 protocol baseline appended 2026-07-03 — see the final section. Context: ADR-035 argues we must characterize the eval instrument before trusting its numbers. Before investing in new golden sets, we ran the existing domain-eval harness (agent/evals/domain, #3324) repeatedly to measure its run-to-run noise and decide which metric to trust.

⚠️ All tables in the Phase-0 sections below are from the v1 holistic-judge era (single holistic correctness float; pre-#3855). The judge protocol v2 (item-level rubric gates + median-alignment KPI) changed score semantics — v1 numbers are not comparable with v2 numbers. The authoritative reference numbers for compare.py baselines are in the v2 baseline section at the end.

What we ran (v1 protocol)

12 live CI runs of the domain-eval workflow on the ecommerce_ops golden set (16 cases), in 4 conditions × 3 identical repeats, changing one determinism lever at a time:

Condition (3 runs each)	mean_correctness (per run)	3-run avg	spread	pass/16
baseline (generator temp=0.3)	0.65 / 0.68 / 0.71	0.681	0.063	5–6
generator temp=0	0.63 / 0.66 / 0.71	0.667	0.075	4–7
temp=0 + OpenRouter provider pinned (OpenAI-only)	0.61 / 0.64 / 0.73	0.656	0.120	2–4
temp=0 + pinned + judge 5-vote (median)	0.64 / 0.68 / 0.65	0.656	0.031	4–6

Findings

1. The noise is intrinsic — not fixable by determinism knobs. Pinning generator temperature=0 did not reduce the band; pinning the OpenRouter provider (OpenAI-only, no fallback) did not either. eco-015 swung correctness 0.0 → 0.0 → 0.8 even with both pinned. The residual variance is inherent model + judge nondeterminism (server batching, MoE routing, float non-associativity — present even at temp=0).

2. Judge K-vote is the one effective lever. Judging each candidate 5× and taking the median correctness + majority verdict roughly halved the mean_correctness band (0.063 → 0.031) — ~4× tighter than provider pinning. The judge's per-call scoring variance was a real, and the only fixable, contributor.

3. Single-run pass_rate is unreliable; the continuous metric is not. pass swung 2/16 → 7/16 (a 3.5× range) across runs that were supposed to be identical — because thresholding correctness at 0.7 (+ rubric gates) turns tiny wobbles near the cutoff into verdict flips. The K-vote median mean_correctness, by contrast, converged to ≈ 0.66 ± 0.015.

   KPI decision: track K-vote median mean_correctness, averaged over K≥3 runs, and trust deltas between policy arms on the same set — not single-run pass_rate. Use pass/fail for per-case diagnostics only.

4. Confirmed product signal (stable across all 12 runs):

   • eco-001…006 (real ecommerce domain cases) fail every run — expected, because the harness is single-turn with no seeded KB; the model is asked policy specifics it was never given. This is the empirical case for the ADR-035 sandbox.
   • eco-010 and eco-016 (fully-specified "just proceed" briefs) fail every run (0.0–0.5): the model over-clarifies / mishandles briefs it should execute — a genuine prompt bug the eval surfaced, independent of KB.

5. Two cases are pathologically noisy — eco-014 (0.9/0.5/0.5) and eco-015 (0.0/0.5/0.0) resist even 5-vote stabilization. Their rubrics are ambiguous to score; they are noise sources and should be rewritten or retired (see the authoring runbook rubric-quality checklist).

Recommendations

• Report K-vote median mean_correctness over K≥3 runs; set DOMAIN_EVAL_JUDGE_VOTES=5 for baselines and policy A/B decisions.
• Do not chase determinism knobs (temp/provider) — the floor is intrinsic.
• Rewrite/retire ambiguous cases (eco-014, eco-015; audit eco-005).
• Grow N — 16 cases with this noise floor cannot resolve small deltas.
• Seed KB (the ADR-035 sandbox) before the domain cases (eco-001…006) can be scored meaningfully.
• File the eco-010/016 over-clarification bug against the clarify-first prompt.

Provenance

Runs on dev / exp/domain-eval-temp0 via the domain-eval workflow, 2026-07-02. Report artifacts (report.json) attached to each GitHub Actions run. Harness knobs (DOMAIN_EVAL_GEN_TEMP, DOMAIN_EVAL_PROVIDER, DOMAIN_EVAL_JUDGE_VOTES) landed with this document.

Protocol v2 baseline (2026-07-03 — post item-level judge)

Judge protocol v2 (PR stack #3757 → #3855 → #3861) changed what correctness means: it is now the K-vote median of the graded golden-intent alignment, with pass/fail authority on item-level rubric majorities, judging the fence-tolerantly extracted reply (not the JSON envelope). The golden suite was simultaneously split into role files + a shared policy suite. Nothing above is comparable with the numbers below — and pre-split reports won't even pair in compare.py (case ids changed).

3 × identical full-suite runs (judge_votes=5, all roles + policy expansion) on dev @ dc16ca33 (runs 28653202903 / 28653204828 / 28653206991):

group	mean_correctness (3 runs)	spread	pass
ecommerce_ops (6 domain cases, no KB)	0.600 / 0.600 / 0.600	0.000	0/6 stable
marketing_content (2 domain cases)	0.900 / 0.900 / 0.900	0.000	1/2 stable
policy@ecommerce_ops (11 behavior cases)	0.800 / 0.809 / 0.727	0.082 ⚠	8/8/7
policy@marketing_content (11 behavior cases)	0.827 / 0.818 / 0.827	0.009	9/9/9

Zero case errors, zero quorum failures, zero judge key-mismatches across all 120 case-runs — the v2 robustness machinery is quiet, i.e. the judge complies with the item-level protocol.

Verdict: instrument accepted. Three of four groups sit at or near zero spread — far inside the v1 K-vote band (0.031). The one out-of-band group localized entirely to pol-008/pol-009; see the refresh below.

Also confirmed under v2 (stable across all runs): the over-clarification bug #3854 (pol-004 fails at 0.0, pol-010 at 0.5), and the KB-less domain cases fail uniformly (the ADR-035 sandbox motivation).

Baseline refresh after the pol-009 rewrite (2026-07-03)

v2's per-item observability corrected the Phase-0 diagnosis: the "pathologically noisy rubrics" were largely the agent flip-flopping between behaviors on identical input — clarify vs refuse (pol-009), ask-the-type vs presume-a-presentation (pol-008) — which the v1 holistic judge blurred into apparent scoring noise. The v2 judge is deterministic given the candidate.

• pol-009 rewritten: the polysemous referent 'the model' was out-of-domain for the personas under test and partly provoked the refusal branch. It is now 'the report' (plausible for every persona), with one concrete item (asks which report is meant) and refusal-instead-of-clarifying named as an explicit violation. Result: 1.0 pass, deterministic, in both policy groups, 3/3 runs.
• pol-008 kept unchanged as a behavioral sentinel: its 1.0↔0.7 wobble is the agent genuinely presuming the deliverable type on some runs — product signal, not instrument noise.

3 × full-suite runs (judge_votes=5) on the rewrite branch (runs 28654261960 / 28654264231 / 28654266492) — these supersede the table above as the compare.py reference baseline:

group	mean_correctness (3 runs)	spread	pass
ecommerce_ops	0.517 / 0.500 / 0.500	0.017	0/6 stable
marketing_content	0.850 / 0.900 / 0.850	0.050*	1/2 stable
policy@ecommerce_ops	0.845 / 0.845 / 0.818	0.027 ✅	9/9/8
policy@marketing_content	0.818 / 0.818 / 0.827	0.009	9/9/9

* a 2-case group magnifies alignment decimals (mkt-001 jittered 0.7↔0.8 with a stable fail verdict); grows out as the role file grows. All groups now within or explained; zero errors / quorum failures / key mismatches. Residual policy@ecommerce_ops spread is exactly the pol-008 sentinel.

Two further observations from this batch:

• Cross-batch level drift is real: the KB-less ecommerce_ops alignments shifted 0.600 → ~0.51 between the two batches (verdicts unchanged) — upstream model/serving drift. Compare policy arms within one batch (same day, same runs); never read absolute means across batches as a trend.
• Provided-context grounding works: mkt-003 (injected 30-day refund policy) passed 1.0 deterministically in every run across both batches.

Seeded-KB re-characterization (2026-07-04 — #3873)

Seeded-KB mode (ADR-035 §1) adds a real retrieval hop before generation for cases that declare kb_gold_docs: fixture docs are indexed into a live pgvector and the production retrieval pipeline (rag/pipelines/retrieval) runs per question, so the draft is judged on what retrieval actually surfaced. This introduces two new variance sources absent from the injection (context) path and therefore requires its own characterization before its numbers are trusted:

1. Embedding + rerank nondeterminism — the ingestion embedder and the Cohere-shaped reranker are extra live OpenRouter hops (same intrinsic-noise class as generation/judge).
2. A new metric axis — retrieval_hit / failure_class (retrieval-miss vs hallucination). Its stability is what a retrieval-policy A/B (HyDE on/off) reads.

Protocol (identical shape to the v2 baseline above). Run the seeded suite 3× identically and read the spread of each seeded group's mean_correctness and its seeded_kb rollup (retrieval_hit_rate, retrieval_misses, hallucinations):

# 3 identical runs — dispatch three times, download each report.json
gh workflow run domain-eval-seeded-kb.yml --ref dev -f role=ecommerce_ops -f judge_votes=5

Read: (a) does retrieval_hit_rate hold steady run-to-run on a fixed corpus (it should — the seeded docs and query are fixed; only embedding float-wobble moves it); (b) is the mean_correctness spread within the v2 K-vote band (±≈0.03)?

Baseline table — PENDING the first live 3× dispatch (needs the OPENROUTER_API_KEY secret + the pgvector service; not runnable from this PR's sandbox). Fill on first run:

group	mean_correctness (3 runs)	spread	retrieval_hit_rate	misses / halluc.
ecommerce_ops (seeded: eco-007, eco-008)	_ / _ / _	_	_	_

Until this row is filled from a real dispatch, treat seeded-KB numbers as machinery-verified but not yet noise-characterized — the tokenless wiring (classification, report shape, skip-when-unavailable) is proven by agent/evals/test_domain_seeded_kb.py, but the live noise band is not yet measured. The A/B design still holds regardless: both retrieval-policy arms run over the same seeded corpus with the same pinned pipeline, so a consistent embedding/rerank bias cancels out of the paired compare.py delta (ADR-035 §6).

Progress log (agent quality over time)

Running ledger of benchmark batches. Method: 3 × full-suite runs at judge_votes=5, three paired compare.py verdicts against the previous accepted baseline. Read verdicts and blockers, not raw means — levels drift across batches (see above); the paired referee is the signal.

Date	dev @	vs baseline	Referee verdicts (3 pairs)	Blockers	Notes
2026-07-03	dc16ca33→refresh	— (baseline)	—	—	v2 protocol baseline + pol-009 refresh (runs 28654261960/…4231/…6492)
2026-07-14	98df7afd8+	07-03 refresh	INCONCLUSIVE ×3 (Δ −0.01/+0.02/+0.02, n=30)	1 technical: pol-004 gained a 2nd violation label — behavior morphed refuse→ask-for-missing-attachment within an already-0.0 case (#3854); no new red-line behavior	Flat. +642 lines of prompt-assembly changes (directives era) moved answer quality neither way on this set. pol-008 sentinel wobbled fail→pass in pair 3, as documented. Runs 29251166426/…9317/…1918
2026-07-14	fix branch #4416	same-day dev batch	B1: IMPR/INC/INC · B2: blocked (safety) · B3: INC ×3 · B4: IMPR/INC/INC, no blockers	B2 caught eager cancellation (eco-005); B4 fixed a Greptile-caught precedence contradiction	First intervention A/B (#3854, PR #4416): 4 iterations in ~2h; stall→extract-specifics→safety-fenced→precedence-fixed. eco-005 flips fail→pass 2/3 (0.2→0.8) — first passes ever; guard clean ×9; pol-004/010 residuals on the issue (pol-010 = product-stance question)