Skip to main content

Eval Instrument v2 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Harden the domain-eval harness into a trustworthy back-testing instrument: judge the reply (not the JSON envelope), item-level hybrid judge protocol, paired-delta compare tool, and a policy/role golden-suite split.

Architecture: Two stacked PRs per the approved spec (docs/superpowers/specs/2026-07-02-eval-instrument-v2-design.md). PR-1 (feat/domain-eval-judge-v2) contains all score-semantics changes (envelope extraction + judge protocol v2 + quorum carried from #3757). PR-2 (feat/domain-eval-compare-policy) adds compare.py and the policy-suite split. Validation re-baseline + numbers-dependent docs land after merge.

Tech Stack: Python 3.11 stdlib only (no new deps), pytest + pytest-asyncio (mocked, tokenless), GitHub Actions domain-eval.yml.

Worktrees / branches:

PhaseWorktreeBranch
0~/.config/superpowers/worktrees/humanwork/domain-eval-contextfeat/domain-eval-context-grounding (#3757)
1~/.config/superpowers/worktrees/humanwork/eval-judge-v2feat/domain-eval-judge-v2 (stacked on #3757)
2~/.config/superpowers/worktrees/humanwork/eval-compare-policyfeat/domain-eval-compare-policy (stacked on PR-1)

Test environment: local venv may lack agent deps. Per task: try cd <worktree>/agent && python3 -m pytest evals/test_domain_harness.py -q. If imports fail (prompts_loader chain), create a scratch venv once: python3 -m venv /tmp/claude-1000/-home-alexdel-Projects-humanwork/evalvenv && /tmp/claude-1000/-home-alexdel-Projects-humanwork/evalvenv/bin/pip install pytest pytest-asyncio pyyaml httpx openai and use that python. If deps still missing, fall back to python3 -m py_compile <files> and rely on regular CI (these tests run tokenless in ci.yml). Never run the full repo suite locally.

File structure (whole initiative):

  • Modify: agent/evals/domain/harness.py — quorum (Phase 0), _extract_reply, _JUDGE_SYS v2, _normalize_vote, run_case v2 aggregation, CaseResult.items/violations
  • Modify: agent/evals/test_domain_harness.py — migrate fakes to v2 shape, new tests
  • Modify: agent/evals/domain/run.py — plan_runs() + policy-suite expansion
  • Create: agent/evals/domain/compare.py — paired-delta comparison CLI
  • Create: agent/evals/test_domain_compare.py
  • Create: agent/evals/domain/golden/policy_behaviors.json (cases moved from role files)
  • Modify: agent/evals/domain/golden/ecommerce_ops.json (drops 10 behavior cases), golden/marketing_content.json (drops mkt-002)
  • Modify: .github/workflows/domain-eval.yml — judge_votes input
  • Modify: agent/evals/domain/README.md, docs/runbooks/golden-set-authoring.md
  • Post-validation: docs/testing/domain-eval-noise-characterization.md (v2 baseline section)

Phase 0 — quorum guard on #3757​

Task 1: Quorum guard in run_case​

Files:

  • Modify: ~/.config/superpowers/worktrees/humanwork/domain-eval-context/agent/evals/domain/harness.py

  • Test: ~/.config/superpowers/worktrees/humanwork/domain-eval-context/agent/evals/test_domain_harness.py

  • Step 1: Write the failing test — append to test_domain_harness.py:

@pytest.mark.asyncio
async def test_run_case_quorum_not_met_is_case_error(monkeypatch):
    # Greptile: with K=3 and 2 transient judge failures, ONE surviving vote must
    # not act as a majority — the case becomes an explicit error instead.
    monkeypatch.setattr(harness, "_JUDGE_VOTES", 3)
    calls = {"n": 0}

    async def fake_generate(_client, _role, _q, context=None):
        return "answer"

    async def fake_judge(_client, **_kw):
        calls["n"] += 1
        if calls["n"] <= 2:
            raise RuntimeError("judge 429")
        return {"correctness": 0.9, "missing_required": [], "forbidden_present": []}

    monkeypatch.setattr(harness, "generate_answer", fake_generate)
    monkeypatch.setattr(harness, "judge", fake_judge)

    r = await run_case(None, "ecommerce_ops", {"id": "q", "question": "q", "golden_answer": "g"})
    assert r.verdict == "fail"
    assert r.error and "quorum" in r.error
  • Step 2: Run test, verify it fails

Run: cd <worktree-0>/agent && <python> -m pytest evals/test_domain_harness.py::test_run_case_quorum_not_met_is_case_error -q Expected: FAIL — currently 1 surviving vote proceeds (verdict pass, no error).

  • Step 3: Implement. In harness.py run_case, replace the current empty-votes guard:
        votes = [v for v in raw if isinstance(v, dict)]
        # Quorum: more than half of the CONFIGURED votes must have succeeded,
        # else one noisy survivor could act as a "majority" (Greptile #3757).
        quorum = _JUDGE_VOTES // 2 + 1
        if len(votes) < quorum:
            first_err = next((v for v in raw if isinstance(v, BaseException)), None)
            raise RuntimeError(
                f"judge quorum not met ({len(votes)}/{_JUDGE_VOTES})"
            ) from first_err

(Delete the old if not votes: ... raise first_err or RuntimeError("all judge votes failed") block — quorum subsumes it.)

  • Step 4: Run the harness test file

Run: cd <worktree-0>/agent && <python> -m pytest evals/test_domain_harness.py -q Expected: all pass — note test_run_case_tolerates_partial_vote_failure (K=3, 1 failure → 2 survivors ≥ quorum 2) still passes.

  • Step 5: Commit + push to #3757, reply to Greptile
cd ~/.config/superpowers/worktrees/humanwork/domain-eval-context
git add agent/evals/domain/harness.py agent/evals/test_domain_harness.py
git commit -m "fix(domain-eval): judge quorum — majority of configured votes must survive (Greptile)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
git push origin feat/domain-eval-context-grounding
gh pr comment 3757 --body "Third finding fixed in latest commit: quorum guard — more than half of the *configured* votes must succeed (\`judge quorum not met (1/3)\` case error otherwise), so a reduced quorum can never act as a majority. Test: \`test_run_case_quorum_not_met_is_case_error\`."

Task 2: Rebase PR-1 branch onto updated #3757​

  • Step 1:
cd ~/.config/superpowers/worktrees/humanwork/eval-judge-v2
git fetch origin feat/domain-eval-context-grounding
git rebase origin/feat/domain-eval-context-grounding

Expected: clean rebase (spec + plan commits only on this branch). If #3757 has merged to dev by execution time, rebase onto origin/dev instead and set PR-1's base to dev.

Phase 1 — PR-1: envelope extraction + judge protocol v2​

All tasks in ~/.config/superpowers/worktrees/humanwork/eval-judge-v2.

Task 3: _extract_reply — judge the reply, not the envelope​

Files:

  • Modify: agent/evals/domain/harness.py

  • Test: agent/evals/test_domain_harness.py

  • Step 1: Write failing tests — append:

def test_extract_reply_from_envelope():
    env = '{"reply": "Hello, happy to help!", "confidence": 0.8, "flag_for_review": false}'
    assert harness._extract_reply(env) == "Hello, happy to help!"


def test_extract_reply_passthrough_prose_and_malformed():
    assert harness._extract_reply("plain prose answer") == "plain prose answer"
    assert harness._extract_reply('{"not json') == '{"not json'
    # dict without reply, or empty reply → raw string unchanged
    assert harness._extract_reply('{"foo": 1}') == '{"foo": 1}'
    assert harness._extract_reply('{"reply": "  "}') == '{"reply": "  "}'

  • Step 2: Run, verify fail — pytest evals/test_domain_harness.py -q -k extract_reply → FAIL (no attribute _extract_reply).

  • Step 3: Implement in harness.py (below _extra_body):

def _extract_reply(candidate: str) -> str:
    """The production prompt makes the model emit a JSON envelope
    ({reply, confidence, flag_for_review, ...}). Judge the prose a customer
    would actually see, not the envelope. Fail-open: anything that isn't a
    dict with a non-empty string "reply" is judged as-is."""
    try:
        parsed = json.loads(candidate)
    except (ValueError, TypeError):
        return candidate
    if isinstance(parsed, dict):
        reply = parsed.get("reply")
        if isinstance(reply, str) and reply.strip():
            return reply.strip()
    return candidate

And in run_case, judge the extracted text (raw envelope stays in the report):

        candidate = await generate_answer(
            client, role, case["question"], case.get("context")
        )
        judged_text = _extract_reply(candidate)

…and pass candidate=judged_text into every judge(...) call (the CaseResult(candidate=candidate) field keeps the raw envelope).

  • Step 4: Run pytest evals/test_domain_harness.py -q → all pass.

  • Step 5: Commit

git add agent/evals/domain/harness.py agent/evals/test_domain_harness.py
git commit -m "feat(domain-eval): judge the extracted reply, not the agent JSON envelope

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 4: Judge protocol v2 — prompt, schema, _normalize_vote​

Files:

  • Modify: agent/evals/domain/harness.py

  • Test: agent/evals/test_domain_harness.py

  • Step 1: Write failing tests for vote normalization:

def test_normalize_vote_defaults_and_clamping():
    items, violations, alignment = harness._normalize_vote(
        {"items": {"a": True}, "violations": {}, "alignment": 1.7},
        must_include=["a", "b"],
        must_not_include=["x"],
    )
    # omitted item -> NOT covered (conservative); omitted violation -> not violated
    assert items == {"a": True, "b": False}
    assert violations == {"x": False}
    assert alignment == 1.0  # clamped

    items, violations, alignment = harness._normalize_vote(
        {"alignment": "bad"}, must_include=["a"], must_not_include=[]
    )
    assert items == {"a": False} and alignment == 0.0

  • Step 2: Run, verify fail — -k normalize_vote → FAIL.

  • Step 3: Implement. Replace _JUDGE_SYS with the v2 prompt:

_JUDGE_SYS = (
    "You are a strict domain-quality judge for a professional support Specialist. "
    "Given a client QUESTION, a reference GOLDEN answer, the CANDIDATE reply, and "
    "two rubric lists, evaluate the candidate. Respond ONLY as JSON with keys: "
    '"items": object mapping EACH MUST_INCLUDE string (verbatim, exactly as given) '
    "to true if the candidate covers that point, else false; "
    '"violations": object mapping EACH MUST_NOT_INCLUDE string (verbatim) to true '
    "if the candidate commits that violation, else false; "
    '"alignment": number 0.0-1.0 — semantic alignment with the golden intent; '
    '"rationale": one sentence. '
    "Be conservative: an unclear or uncovered item is false; flag a violation only "
    "when it is actually present in the candidate."
)

Add below _extract_reply:

def _normalize_vote(
    vote: dict[str, Any], must_include: list[str], must_not_include: list[str]
) -> tuple[dict[str, bool], dict[str, bool], float]:
    """Coerce one judge vote to the v2 shape, keyed by the rubric strings.
    Omitted item -> False (not covered — conservative on quality);
    omitted violation -> False (not violated — conservative on blame)."""
    items_raw = vote.get("items") or {}
    viol_raw = vote.get("violations") or {}
    items = {k: bool(items_raw.get(k, False)) for k in must_include}
    violations = {k: bool(viol_raw.get(k, False)) for k in must_not_include}
    try:
        alignment = float(vote.get("alignment", 0.0))
    except (TypeError, ValueError):
        alignment = 0.0
    return items, violations, min(1.0, max(0.0, alignment))

  • Step 4: Run -k normalize_vote → PASS. (Other tests still pass — nothing else changed yet.)

  • Step 5: Commit

git add agent/evals/domain/harness.py agent/evals/test_domain_harness.py
git commit -m "feat(domain-eval): judge v2 prompt + vote normalization (item-level rubric)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 5: run_case v2 aggregation + migrate existing tests​

Files:

  • Modify: agent/evals/domain/harness.py (CaseResult + run_case)

  • Modify: agent/evals/test_domain_harness.py (all fake_judge fakes)

  • Step 1: Migrate existing fakes to the v2 shape and add the new aggregation tests. In test_domain_harness.py:

Replace every fake_judge return value:

  • test_run_case_pass: return {"items": {}, "violations": {}, "alignment": 0.9, "rationale": "ok"} (case unchanged, no rubric lists).
  • test_run_case_forbidden_forces_fail_even_if_correct: case becomes {"id": "y", "question": "q", "golden_answer": "g", "must_not_include": ["unconditional promise of a full refund"]}; fake returns {"items": {}, "violations": {"unconditional promise of a full refund": True}, "alignment": 0.95}; assertions unchanged (fail, forbidden_present).
  • test_run_case_missing_required_forces_fail: case gains "must_include": ["offer to check the order"]; fake returns {"items": {"offer to check the order": False}, "violations": {}, "alignment": 0.85}; assertions unchanged.
  • test_run_case_k_vote_median_and_majority: scores = iter([0.9, 0.9, 0.5]), fake returns {"items": {}, "violations": {}, "alignment": next(scores)}; assert r.correctness == 0.9 and r.verdict == "pass" (median alignment).
  • test_run_case_tolerates_partial_vote_failure: good votes return {"items": {}, "violations": {}, "alignment": 0.9}; assertions unchanged.
  • test_run_case_quorum_not_met_is_case_error (from Task 1): good vote returns the v2 shape too.
  • test_run_case_even_votes_no_tie_fail → rewrite with the v2 tie semantics (majority required to ASSERT; ties keep the default):
@pytest.mark.asyncio
async def test_run_case_even_vote_tie_semantics(monkeypatch):
    # Ties keep the default: a 1-of-2 violation flag is NOT a majority -> not
    # violated (no false accusation); a 1-of-2 item coverage is NOT a majority
    # -> item stays uncovered (conservative on quality).
    monkeypatch.setattr(harness, "_JUDGE_VOTES", 2)
    outs = iter([
        {"items": {"a": True}, "violations": {"v": True}, "alignment": 0.9},
        {"items": {"a": False}, "violations": {"v": False}, "alignment": 0.9},
    ])

    async def fake_generate(_client, _role, _q, context=None):
        return "answer"

    async def fake_judge(_client, **_kw):
        return next(outs)

    monkeypatch.setattr(harness, "generate_answer", fake_generate)
    monkeypatch.setattr(harness, "judge", fake_judge)

    r = await run_case(
        None, "ecommerce_ops",
        {"id": "t", "question": "q", "golden_answer": "g",
         "must_include": ["a"], "must_not_include": ["v"]},
    )
    assert r.forbidden_present == []          # tie -> not violated
    assert r.missing_required == ["a"]        # tie -> not covered
    assert r.verdict == "fail"

Also add an items-detail test:

@pytest.mark.asyncio
async def test_run_case_reports_item_detail(monkeypatch):
    monkeypatch.setattr(harness, "_JUDGE_VOTES", 1)

    async def fake_generate(_client, _role, _q, context=None):
        return "answer"

    async def fake_judge(_client, **_kw):
        return {"items": {"a": True, "b": False}, "violations": {"v": False}, "alignment": 0.8}

    monkeypatch.setattr(harness, "generate_answer", fake_generate)
    monkeypatch.setattr(harness, "judge", fake_judge)

    r = await run_case(
        None, "ecommerce_ops",
        {"id": "d", "question": "q", "golden_answer": "g",
         "must_include": ["a", "b"], "must_not_include": ["v"]},
    )
    assert r.items == {"a": True, "b": False}
    assert r.violations == {"v": False}
    assert r.missing_required == ["b"]
    assert "items" in r.as_dict()

  • Step 2: Run, verify the new/migrated tests fail — pytest evals/test_domain_harness.py -q → failures (run_case still aggregates v1 keys).

  • Step 3: Implement. In CaseResult add two fields (after forbidden_present):

    items: dict[str, bool] = field(default_factory=dict)
    violations: dict[str, bool] = field(default_factory=dict)

Replace the post-quorum aggregation in run_case with:

        must_include = case.get("must_include", [])
        must_not_include = case.get("must_not_include", [])
        normalized = [_normalize_vote(v, must_include, must_not_include) for v in votes]
        majority = len(votes) / 2
        # An assertion (covered / violated) needs a strict majority of surviving
        # votes; ties keep the default (uncovered / not violated).
        items = {
            k: sum(n[0][k] for n in normalized) > majority for k in must_include
        }
        violations = {
            k: sum(n[1][k] for n in normalized) > majority for k in must_not_include
        }
        correctness = statistics.median([n[2] for n in normalized])
        missing = [k for k, ok in items.items() if not ok]
        forbidden = [k for k, bad in violations.items() if bad]
        return CaseResult(
            id=case["id"],
            correctness=round(correctness, 4),
            verdict=_verdict(correctness, missing, forbidden),
            missing_required=missing,
            forbidden_present=forbidden,
            items=items,
            violations=violations,
            rationale=f"{len(votes)}/{_JUDGE_VOTES} votes ok; median alignment {round(correctness, 4)}",
            candidate=candidate,
        )

(The Counter import becomes unused — remove it. judge() itself is unchanged apart from the new _JUDGE_SYS; it still returns the parsed JSON dict.)

Update the module docstring pipeline description (step 2) to: "judge(...) — LLM-as-judge checks each rubric item yes/no and grades 0..1 alignment with the golden intent; K votes are aggregated (item majorities + median alignment)."

  • Step 4: Run the whole file — pytest evals/test_domain_harness.py -q → ALL pass.

  • Step 5: Commit

git add agent/evals/domain/harness.py agent/evals/test_domain_harness.py
git commit -m "feat(domain-eval): v2 aggregation — item majorities + median alignment KPI

BREAKING (score semantics): correctness is now the K-vote median of the
graded golden-intent alignment; pass/fail authority moves to item-level
rubric majorities. Not comparable with pre-v2 baselines — re-baseline
tracked in the spec's validation phase.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 6: judge_votes workflow input​

Files:

  • Modify: .github/workflows/domain-eval.yml

  • Step 1: Edit. Add under on.workflow_dispatch.inputs:

      judge_votes:
        description: "Judge votes per case (median/majority; use 5 for baselines)"
        type: string
        default: "1"

And in the job env block add:

      DOMAIN_EVAL_JUDGE_VOTES: ${{ github.event.inputs.judge_votes || '1' }}
  • Step 2: Validate + commit
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/domain-eval.yml'))" && echo YAML OK
git add .github/workflows/domain-eval.yml
git commit -m "ci(domain-eval): judge_votes workflow input for K-vote baselines

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 7: PR-1 doc edits (non-number)​

Files:

  • Modify: agent/evals/domain/README.md

  • Modify: docs/runbooks/golden-set-authoring.md

  • Step 1: README. In "What it does", replace the sentence describing the judge with:

…then score it with an LLM-as-judge that (a) checks each must_include / must_not_include rubric item yes/no, and (b) grades a 0–1 alignment with the reference (golden_answer). With DOMAIN_EVAL_JUDGE_VOTES=K, K votes are aggregated: an item is covered/violated only on a strict majority; correctness = the median alignment (the continuous KPI). A case passes when every required item is covered, no forbidden item is violated, and median alignment ≥ DOMAIN_EVAL_PASS_THRESHOLD (0.7). The judge scores the extracted reply text from the agent's JSON envelope, not the raw envelope.

  • Step 2: Runbook. In the rubric-quality checklist add one bullet:
  • Each rubric item is independently judgeable yes/no — the judge is literally asked "does the reply do X?" per item, so an item that bundles two ideas ("apologizes and offers a refund") must be split.
  • Step 3: Commit
git add agent/evals/domain/README.md docs/runbooks/golden-set-authoring.md
git commit -m "docs(domain-eval): describe v2 judge protocol + item-writing guidance

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 8: File the over-clarification bug​

  • Step 1:
cd /home/alexdel/Projects/humanwork
gh issue create \
  --title "Agent over-clarifies fully-specified briefs (clarify-first policy too aggressive)" \
  --body "## Evidence (from the domain-eval Phase-0 characterization)

Cases \`eco-010-fully-specified-deck\` and \`eco-016-fully-specified-product-update\` — briefs where audience, purpose, scope, tone (and source material where needed) are ALL provided, so the correct behavior is to proceed — **failed 12/12 runs** across every eval condition (temp 0.3 / temp 0 / provider-pinned / judge 5-vote), correctness 0.0–0.5. This is stable signal, not judge noise (see docs/testing/domain-eval-noise-characterization.md).

## Suspected cause
The clarify-first policy in the production system prompt (\`build_system_prompt\`) is too aggressive: it interrogates complete briefs instead of executing them. Note the mirror cases (\`eco-007..009\`, genuinely ambiguous asks) pass — the policy fires correctly on ambiguity but doesn't stand down on completeness.

## Repro
\`gh workflow run domain-eval.yml --ref dev -f role=ecommerce_ops\` → check \`eco-010\`/\`eco-016\` in the report artifact.

## Fix direction
Tune the clarify-first section of the role prompts to include an explicit 'if the brief specifies audience+purpose+scope+tone, proceed without questions' branch — then back-test with the eval before/after (compare.py, ADR-035 §6)."

Expected: issue URL printed. Note the issue number for the PR body.

Task 9: Push + open PR-1​

  • Step 1:
cd ~/.config/superpowers/worktrees/humanwork/eval-judge-v2
git push -u origin feat/domain-eval-judge-v2
gh pr create --base feat/domain-eval-context-grounding --head feat/domain-eval-judge-v2 \
  --title "feat(domain-eval): judge v2 — reply extraction + item-level hybrid protocol" \
  --body "$(cat <<'EOF'
**Stacked on #3757** (base set accordingly; will retarget to dev when it merges).

Implements PR-1 of the approved spec (docs/superpowers/specs/2026-07-02-eval-instrument-v2-design.md):

- **Judge the reply, not the envelope** — `_extract_reply` pulls `reply` from the agent's JSON envelope before judging (fail-open on prose/malformed).
- **Judge protocol v2 (hybrid)** — one call per vote returns per-rubric-item yes/no (`items`/`violations`) + a graded 0–1 `alignment`. K-vote aggregation: strict item majorities decide the gates; **median alignment becomes `correctness`** (the continuous KPI). ⚠️ Score semantics change → re-baseline planned (spec §V); pre-v2 numbers not comparable.
- **`judge_votes` workflow input** for K-vote baseline runs.
- Docs: README protocol description, runbook item-writing rule.
- Filed the over-clarification bug surfaced by Phase-0 (eco-010/016).

All tests tokenless/mocked as before.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Phase 2 — PR-2: compare.py + policy/role split​

Task 10: Worktree + compare.py core​

Files:

  • Create worktree: ~/.config/superpowers/worktrees/humanwork/eval-compare-policy, branch feat/domain-eval-compare-policy off feat/domain-eval-judge-v2

  • Create: agent/evals/domain/compare.py

  • Create: agent/evals/test_domain_compare.py

  • Step 1: Worktree

cd /home/alexdel/Projects/humanwork
git worktree add ~/.config/superpowers/worktrees/humanwork/eval-compare-policy \
  -b feat/domain-eval-compare-policy feat/domain-eval-judge-v2
  • Step 2: Write failing tests — agent/evals/test_domain_compare.py:
"""Tests for the paired-delta compare tool (ADR-035 §6). Tokenless."""
from __future__ import annotations

from evals.domain import compare


def _report(cases: dict[str, tuple[float, str, list[str]]], role="ecommerce_ops"):
    """cases: id -> (correctness, verdict, forbidden_present)"""
    return {
        "roles": {
            role: {
                "summary": {},
                "cases": [
                    {"id": cid, "correctness": c, "verdict": v, "forbidden_present": f}
                    for cid, (c, v, f) in cases.items()
                ],
            }
        }
    }


def _uniform(n, corr, verdict="pass", forbidden=None):
    return {f"c{i:02d}": (corr, verdict, forbidden or []) for i in range(n)}


def test_paired_cases_intersection_only():
    base = _report({"a": (0.5, "pass", []), "b": (0.5, "pass", [])})
    cand = _report({"b": (0.7, "pass", []), "c": (0.9, "pass", [])})
    pairs = compare.paired_cases(base, cand)
    assert [(p.case_id) for p in pairs] == ["b"]
    assert pairs[0].delta == 0.2


def test_verdict_improved():
    base = _report(_uniform(8, 0.5))
    cand = _report(_uniform(8, 0.7))
    result = compare.compare_reports(base, cand)
    assert result.verdict == "IMPROVED"
    assert not result.blockers


def test_verdict_regressed_and_fail_flag():
    base = _report(_uniform(8, 0.7))
    cand = _report(_uniform(8, 0.5))
    result = compare.compare_reports(base, cand)
    assert result.verdict == "REGRESSED"


def test_verdict_inconclusive_on_mixed_noise():
    base = _report({f"c{i}": (0.5, "pass", []) for i in range(8)})
    cand_cases = {f"c{i}": (0.5 + (0.01 if i % 2 else -0.01), "pass", []) for i in range(8)}
    result = compare.compare_reports(base, _report(cand_cases))
    assert result.verdict == "INCONCLUSIVE"


def test_new_violation_is_blocker():
    base = _report({"a": (0.9, "pass", []), "b": (0.9, "pass", [])})
    cand = _report({"a": (0.9, "pass", []), "b": (0.9, "fail", ["a fabricated tracking number"])})
    result = compare.compare_reports(base, cand)
    assert result.blockers and "b" in result.blockers[0]


def test_main_exit_codes(tmp_path, capsys):
    import json
    b, c = tmp_path / "b.json", tmp_path / "c.json"
    b.write_text(json.dumps(_report(_uniform(8, 0.7))))
    c.write_text(json.dumps(_report(_uniform(8, 0.5))))
    assert compare.main([str(b), str(c)]) == 0                       # report-only
    assert compare.main([str(b), str(c), "--fail-on-regression"]) == 1
    out = capsys.readouterr().out
    assert "REGRESSED" in out


def test_main_empty_intersection(tmp_path):
    import json
    b, c = tmp_path / "b.json", tmp_path / "c.json"
    b.write_text(json.dumps(_report({"a": (0.5, "pass", [])})))
    c.write_text(json.dumps(_report({"z": (0.5, "pass", [])})))
    assert compare.main([str(b), str(c)]) == 2

  • Step 3: Run, verify fail — pytest evals/test_domain_compare.py -q → FAIL (no module compare).

  • Step 4: Implement agent/evals/domain/compare.py:

"""
Paired-delta comparison of two domain-eval reports (ADR-035 §6).

Usage (from `agent/`):
    python -m evals.domain.compare baseline.json candidate.json [--role R] [--fail-on-regression]

Pairs cases by (role, id) — intersection only. Reports per-case Δcorrectness,
a bootstrap 95% CI on the mean paired Δ, gate flips, and a verdict:
IMPROVED (CI > 0) / REGRESSED (CI < 0) / INCONCLUSIVE. Any case that newly
fails on a rubric violation (forbidden_present) is a hard BLOCKER regardless
of the mean. Works on v1 and v2 report schemas (uses correctness/verdict/
forbidden_present only). Exit codes: 0 ok, 1 with --fail-on-regression on
REGRESSED or blocker, 2 usage/data error.
"""
from __future__ import annotations

import argparse
import json
import random
import sys
from dataclasses import dataclass, field
from pathlib import Path

BOOT_ITERS = 10_000
BOOT_SEED = 42


@dataclass
class Pair:
    role: str
    case_id: str
    base_corr: float
    cand_corr: float
    base_verdict: str
    cand_verdict: str
    new_violation: bool

    @property
    def delta(self) -> float:
        return round(self.cand_corr - self.base_corr, 4)


@dataclass
class CompareResult:
    pairs: list[Pair]
    mean_delta: float
    ci_low: float
    ci_high: float
    verdict: str  # IMPROVED | REGRESSED | INCONCLUSIVE
    gate_flips: list[str] = field(default_factory=list)  # "role/id pass->fail"
    blockers: list[str] = field(default_factory=list)


def paired_cases(base: dict, cand: dict, role: str | None = None) -> list[Pair]:
    pairs: list[Pair] = []
    roles = sorted(set(base.get("roles", {})) & set(cand.get("roles", {})))
    for r in roles:
        if role and r != role:
            continue
        bc = {c["id"]: c for c in base["roles"][r].get("cases", [])}
        cc = {c["id"]: c for c in cand["roles"][r].get("cases", [])}
        for cid in sorted(bc.keys() & cc.keys()):
            b, c = bc[cid], cc[cid]
            pairs.append(
                Pair(
                    role=r,
                    case_id=cid,
                    base_corr=float(b.get("correctness", 0.0)),
                    cand_corr=float(c.get("correctness", 0.0)),
                    base_verdict=b.get("verdict", ""),
                    cand_verdict=c.get("verdict", ""),
                    new_violation=bool(c.get("forbidden_present"))
                    and not b.get("forbidden_present"),
                )
            )
    return pairs


def bootstrap_ci(deltas: list[float], iters: int = BOOT_ITERS, seed: int = BOOT_SEED):
    rng = random.Random(seed)
    n = len(deltas)
    means = sorted(
        sum(rng.choice(deltas) for _ in range(n)) / n for _ in range(iters)
    )
    return means[int(0.025 * iters)], means[int(0.975 * iters)]


def compare_reports(base: dict, cand: dict, role: str | None = None) -> CompareResult:
    pairs = paired_cases(base, cand, role)
    if not pairs:
        raise ValueError("no overlapping (role, case id) pairs between the reports")
    deltas = [p.delta for p in pairs]
    mean_delta = sum(deltas) / len(deltas)
    lo, hi = bootstrap_ci(deltas)
    if lo > 0:
        verdict = "IMPROVED"
    elif hi < 0:
        verdict = "REGRESSED"
    else:
        verdict = "INCONCLUSIVE"
    gate_flips = [
        f"{p.role}/{p.case_id} {p.base_verdict}->{p.cand_verdict}"
        for p in pairs
        if p.base_verdict != p.cand_verdict
    ]
    blockers = [
        f"{p.role}/{p.case_id} newly fails on a rubric violation"
        for p in pairs
        if p.new_violation and p.cand_verdict == "fail"
    ]
    return CompareResult(
        pairs=pairs,
        mean_delta=round(mean_delta, 4),
        ci_low=round(lo, 4),
        ci_high=round(hi, 4),
        verdict=verdict,
        gate_flips=gate_flips,
        blockers=blockers,
    )


def _markdown(r: CompareResult) -> str:
    lines = [
        "# Domain-eval paired comparison",
        "",
        f"**Verdict: {r.verdict}** — mean Δcorrectness {r.mean_delta:+.4f} "
        f"(95% CI [{r.ci_low:+.4f}, {r.ci_high:+.4f}], n={len(r.pairs)})",
        "",
    ]
    if r.blockers:
        lines += ["## 🚫 BLOCKERS (new rubric-violation failures)"] + [
            f"- {b}" for b in r.blockers
        ] + [""]
    if r.gate_flips:
        lines += ["## Gate flips"] + [f"- {g}" for g in r.gate_flips] + [""]
    improved = sum(1 for p in r.pairs if p.delta > 0)
    regressed = sum(1 for p in r.pairs if p.delta < 0)
    tied = len(r.pairs) - improved - regressed
    lines += [f"Cases: {improved} improved · {regressed} regressed · {tied} tied", ""]
    lines += ["| role | case | base | cand | Δ |", "|---|---|---|---|---|"]
    for p in sorted(r.pairs, key=lambda p: p.delta):
        lines.append(
            f"| {p.role} | {p.case_id} | {p.base_corr} | {p.cand_corr} | {p.delta:+.4f} |"
        )
    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser(description="Paired-delta compare of two domain-eval reports")
    ap.add_argument("baseline")
    ap.add_argument("candidate")
    ap.add_argument("--role", default=None)
    ap.add_argument("--fail-on-regression", action="store_true")
    args = ap.parse_args(argv)
    try:
        base = json.loads(Path(args.baseline).read_text())
        cand = json.loads(Path(args.candidate).read_text())
        result = compare_reports(base, cand, args.role)
    except (OSError, ValueError) as exc:
        print(f"compare: {exc}", file=sys.stderr)
        return 2
    print(_markdown(result))
    if args.fail_on_regression and (result.verdict == "REGRESSED" or result.blockers):
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

  • Step 5: Run pytest evals/test_domain_compare.py -q → ALL pass. Also python3 -m py_compile agent/evals/domain/compare.py.

  • Step 6: Commit

git add agent/evals/domain/compare.py agent/evals/test_domain_compare.py
git commit -m "feat(domain-eval): compare.py — paired-delta back-test verdicts (ADR-035 §6)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 11: Policy/role golden-set split​

Files:

  • Create: agent/evals/domain/golden/policy_behaviors.json

  • Modify: agent/evals/domain/golden/ecommerce_ops.json, golden/marketing_content.json

  • Modify: agent/evals/test_domain_harness.py (fixture-shape test)

  • Step 1: Mechanical move script (run from the worktree root) — moves the 10 behavior cases + mkt-002 verbatim, renames ids:

python3 - <<'PY'
import json
from pathlib import Path

g = Path("agent/evals/domain/golden")
eco = json.loads((g / "ecommerce_ops.json").read_text())
mkt = json.loads((g / "marketing_content.json").read_text())

behavior_prefixes = tuple(f"eco-{n:03d}" for n in range(7, 17))  # eco-007..eco-016
moved, kept_eco = [], []
for c in eco["cases"]:
    (moved if c["id"].startswith(behavior_prefixes) else kept_eco).append(c)
assert len(moved) == 10, f"expected 10 behavior cases, got {len(moved)}"

kept_mkt = [c for c in mkt["cases"] if c["id"] != "mkt-002-seriesA-post-proceed"]
mkt002 = [c for c in mkt["cases"] if c["id"] == "mkt-002-seriesA-post-proceed"]
assert len(mkt002) == 1
moved.append(mkt002[0])

for i, c in enumerate(moved, start=1):
    slug = c["id"].split("-", 2)[2]  # keep original slug
    c["id"] = f"pol-{i:03d}-{slug}"

policy = {
    "suite": "policy",
    "description": (
        "Role-agnostic behavior suite: clarify-vs-proceed, no-fabrication, "
        "escalation. Run against EVERY role discovered in golden/ (a behavior "
        "is a property each persona must satisfy); reported as policy@<role>. "
        "Domain facts belong in the per-role files, not here."
    ),
    "cases": moved,
}
(g / "policy_behaviors.json").write_text(json.dumps(policy, indent=2, ensure_ascii=False) + "\n")
eco["cases"] = kept_eco
mkt["cases"] = kept_mkt
(g / "ecommerce_ops.json").write_text(json.dumps(eco, indent=2, ensure_ascii=False) + "\n")
(g / "marketing_content.json").write_text(json.dumps(mkt, indent=2, ensure_ascii=False) + "\n")
print("policy:", len(moved), "eco:", len(kept_eco), "mkt:", len(kept_mkt))
PY

Expected output: policy: 11 eco: 6 mkt: 2.

  • Step 2: Update the fixture-shape test — in test_golden_fixtures_well_formed, replace the role assertion:
        gs = json.loads(f.read_text())
        if gs.get("suite") == "policy":
            assert gs.get("cases"), f"{f.name} has no cases"
        else:
            assert gs.get("role"), f"{f.name} missing role"
            assert gs.get("cases"), f"{f.name} has no cases"

(keep the per-case field assertions running for both file kinds).

  • Step 3: Run pytest evals/test_domain_harness.py::test_golden_fixtures_well_formed -q → PASS.

  • Step 4: Commit

git add agent/evals/domain/golden/ agent/evals/test_domain_harness.py
git commit -m "refactor(domain-eval): split role-agnostic behavior cases into policy suite

eco-007..016 + mkt-002 -> golden/policy_behaviors.json (pol-001..011).
Role files now contain domain cases only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 12: Runner expands the policy suite per role​

Files:

  • Modify: agent/evals/domain/run.py

  • Test: agent/evals/test_domain_harness.py

  • Step 1: Write failing tests — append to test_domain_harness.py:

def test_plan_runs_expands_policy_per_role():
    from evals.domain.run import plan_runs

    goldens = [
        {"role": "ecommerce_ops", "cases": [{"id": "eco-001"}]},
        {"role": "marketing_content", "cases": [{"id": "mkt-001"}]},
        {"suite": "policy", "cases": [{"id": "pol-001"}, {"id": "pol-002"}]},
    ]
    runs = plan_runs(goldens)
    keys = sorted({k for k, _, _ in runs})
    assert keys == ["ecommerce_ops", "marketing_content",
                    "policy@ecommerce_ops", "policy@marketing_content"]
    # policy cases run once per discovered role, with that role's prompt
    assert sum(1 for k, r, c in runs if k == "policy@marketing_content") == 2
    assert all(r == "marketing_content" for k, r, c in runs if k == "policy@marketing_content")


def test_plan_runs_role_filter_restricts_everything():
    from evals.domain.run import plan_runs

    goldens = [
        {"role": "ecommerce_ops", "cases": [{"id": "e1"}]},
        {"role": "marketing_content", "cases": [{"id": "m1"}]},
        {"suite": "policy", "cases": [{"id": "p1"}]},
    ]
    runs = plan_runs(goldens, role_filter="ecommerce_ops")
    keys = sorted({k for k, _, _ in runs})
    assert keys == ["ecommerce_ops", "policy@ecommerce_ops"]

  • Step 2: Run, verify fail — -k plan_runs → FAIL (no plan_runs).

  • Step 3: Implement. In run.py, replace _load_goldens + _run with:

def _load_goldens() -> list[dict]:
    sets = []
    for f in sorted(GOLDEN_DIR.glob("*.json")):
        sets.append(json.loads(f.read_text()))
    return sets


def plan_runs(
    golden_sets: list[dict], role_filter: str | None = None
) -> list[tuple[str, str, dict]]:
    """(report_key, prompt_role, case) triples.

    Role files run under their own role. Policy-suite files (suite == "policy")
    are expanded against EVERY role discovered from the role files — a behavior
    is a property each persona must satisfy — reported as policy@<role>.
    role_filter restricts both kinds (cheap runs / CI role input).
    """
    role_sets = [g for g in golden_sets if g.get("suite") != "policy"]
    policy_sets = [g for g in golden_sets if g.get("suite") == "policy"]
    roles = [g["role"] for g in role_sets]
    if role_filter and role_filter not in roles:
        raise SystemExit(f"golden set not found for role: {role_filter}")
    runs: list[tuple[str, str, dict]] = []
    for g in role_sets:
        if role_filter and g["role"] != role_filter:
            continue
        runs.extend((g["role"], g["role"], c) for c in g.get("cases", []))
    for g in policy_sets:
        for r in roles:
            if role_filter and r != role_filter:
                continue
            runs.extend((f"policy@{r}", r, c) for c in g.get("cases", []))
    return runs


async def _run(role: str | None) -> dict:
    client = _client()
    grouped: dict[str, list[CaseResult]] = {}
    # Sequential keeps token spend predictable and avoids rate-limit bursts.
    for key, prompt_role, case in plan_runs(_load_goldens(), role):
        grouped.setdefault(key, []).append(await run_case(client, prompt_role, case))
    return {
        "roles": {
            k: {"summary": aggregate(v), "cases": [c.as_dict() for c in v]}
            for k, v in grouped.items()
        }
    }

(Update the module docstring usage note: --role ecommerce_ops now also runs policy@ecommerce_ops.)

  • Step 4: Run the full harness test file — pytest evals/test_domain_harness.py -q → ALL pass. Also python3 -m py_compile agent/evals/domain/run.py.

  • Step 5: Commit

git add agent/evals/domain/run.py agent/evals/test_domain_harness.py
git commit -m "feat(domain-eval): run policy suite against every discovered role

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"

Task 13: PR-2 docs + open PR​

Files:

  • Modify: agent/evals/domain/README.md, docs/runbooks/golden-set-authoring.md

  • Step 1: README. Add after the "Add cases" section:

Policy suite vs role files​

golden/policy_behaviors.json ("suite": "policy") holds role-agnostic behavior cases — clarify-vs-proceed, no-fabrication, escalation. It runs against every role in golden/ (reported as policy@<role>), so a role-prompt regression in any persona shows up. Role files hold domain cases only (facts + red lines). --role R restricts both kinds to role R.

Comparing two runs (back-testing a change)​

python -m evals.domain.compare baseline/report.json candidate/report.json [--fail-on-regression]

Pairs cases by (role, id), reports per-case Δ, bootstrap 95% CI on the mean paired Δ, gate flips, and a verdict (IMPROVED / REGRESSED / INCONCLUSIVE). A case that newly fails on a rubric violation is a hard blocker. Trust this paired verdict — not eyeballed aggregates.

  • Step 2: Runbook. Replace the "1. The case format" intro sentence with a decision rule box right above it:

Where does my case go? Behavior (clarify-vs-proceed, don't fabricate, escalate when out of scope) → golden/policy_behaviors.json — it runs against every role automatically, don't duplicate it per role. Domain knowledge (facts, policies, red lines specific to the role) → the role file. Analyst sessions therefore only need to author facts + red lines.

  • Step 3: Commit + push + PR
git add agent/evals/domain/README.md docs/runbooks/golden-set-authoring.md
git commit -m "docs(domain-eval): policy-suite semantics + compare.py back-test workflow

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
git push -u origin feat/domain-eval-compare-policy
gh pr create --base feat/domain-eval-judge-v2 --head feat/domain-eval-compare-policy \
  --title "feat(domain-eval): compare.py paired deltas + policy/role suite split" \
  --body "$(cat <<'EOF'
**Stacked on the judge-v2 PR** (base set accordingly; retarget to dev as the stack merges).

Implements PR-2 of docs/superpowers/specs/2026-07-02-eval-instrument-v2-design.md:

- **compare.py** — the missing paired-delta tool (ADR-035 §6): per-case Δ, bootstrap 95% CI on the mean paired Δ (stdlib-only), gate flips, IMPROVED/REGRESSED/INCONCLUSIVE verdict, new-rubric-violation = hard blocker, `--fail-on-regression` exit code. Reads v1 and v2 report schemas.
- **Policy/role split** — the 10 role-agnostic behavior cases (+ mkt-002) move verbatim to `golden/policy_behaviors.json` (`pol-001..011`); the runner expands the policy suite against every discovered role (`policy@<role>`). Role files now carry domain cases only, so analyst sessions author facts + red lines.
- Docs: README (policy semantics, compare workflow), runbook ("where does my case go?").

All tests tokenless/mocked.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

Phase 3 — validation + numbers-dependent docs (after PR-1 & PR-2 merge to dev)​

Task 14: v2 re-baseline runs​

  • Step 1: Dispatch 3 identical full runs (all roles + policy suite, K=5):
cd /home/alexdel/Projects/humanwork
git fetch origin dev -q
for i in 1 2 3; do gh workflow run domain-eval.yml --ref dev -f judge_votes=5; done
sleep 8 && gh run list --workflow=domain-eval.yml --limit 3 --json databaseId,status
  • Step 2: Background-watch (never block-watch):
# run_in_background:
for id in <ID1> <ID2> <ID3>; do gh run watch $id --exit-status >/dev/null 2>&1; \
  echo "run $id: $(gh run view $id --json conclusion -q .conclusion)"; done; echo ALL_DONE
  • Step 3: Download artifacts + compute spread
SCRATCH=/tmp/claude-1000/-home-alexdel-Projects-humanwork/*/scratchpad/v2-baseline
mkdir -p $SCRATCH
for id in <ID1> <ID2> <ID3>; do gh run download $id -n domain-eval-report -D $SCRATCH/$id; done
for id in <ID1> <ID2> <ID3>; do
  jq -r --arg id "$id" '.roles | to_entries[] | "\($id) \(.key): mean=\(.value.summary.mean_correctness) pass=\(.value.summary.passed)/\(.value.summary.cases)"' $SCRATCH/$id/report.json
done

Success criterion: per-group run-level mean_correctness spread ≤ 0.031 (Phase-0 v1 K-vote band) and zero quorum errors. If spread is larger, report honestly and investigate before docs (do not paper over).

Task 15: Docs finalization PR (v2 baseline)​

Files (fresh worktree off origin/dev, branch docs/domain-eval-v2-baseline):

  • Modify: docs/testing/domain-eval-noise-characterization.md

  • Step 1: Append a section with the measured numbers:

## Protocol v2 baseline (post item-level judge — <date>)

Judge v2 (item-level gates + median alignment KPI, reply-extraction) changed score
semantics; the tables above are the **holistic-judge era** and are **not comparable**
with numbers below.

3 × full runs (`judge_votes=5`, all roles + policy suite) on dev @ <sha>:

| group | mean_correctness (3 runs) | spread | pass |
|---|---|---|---|
| ecommerce_ops | <a> / <b> / <c> | <s> | <p> |
| marketing_content | … | … | … |
| policy@ecommerce_ops | … | … | … |
| policy@marketing_content | … | … | … |

Verdict: <within / outside> the v1 K-vote band (0.031). These are the reference
numbers for compare.py baselines until the next protocol change.

(also add a one-line pointer under the doc's intro: "v2 protocol baseline: see final section.")

  • Step 2: Mark the old summary table header with "(v1 holistic-judge protocol — superseded, see v2 baseline below)", commit, push, open a docs PR to dev, title docs(domain-eval): protocol v2 baseline.

Self-review checklist (done at write time)​

  • Spec coverage: 0→Task 1-2; A→Task 3 + 8; C→Task 4-5; workflow input→Task 6; PR-1 docs→7; B→10; D→11-12; PR-2 docs→13; V→14; docs finalization→15. ✓
  • No placeholders: Phase-3 <ID/N/date/sha> tokens are run-time measured values by design, with explicit instructions to fill from artifacts. ✓
  • Type consistency: plan_runs returns (report_key, prompt_role, case) everywhere; CaseResult.items/violations match Task 5 and the items-detail test; compare.Pair.delta/CompareResult.verdict names match tests. ✓
Last updated on by Degen