PM task-family eval suite — design

Date: 2026-08-03 · Status: approved (brainstorm with Alex) Context: ADR-035 · golden-set authoring runbook · noise characterization · task-type classification (#4662 D2) · agent/prompts/product_ops.txt

Problem

We want to know how good the agent is at being a project manager across the ten northstar JTBD task types. Those ten already exist in production as vocabulary:

• TaskType enum — api/src/common/entities.ts:896-906
• keyword classifier + TASK_TYPE_RISK map — api/src/conversations/task-type-classification.util.ts:21-140 (7 read_draft, 3 external_side_effect)
• the ten dogfood prompts, verbatim, with tiers — api/src/conversations/task-type-classification.util.spec.ts:8-70

What does not exist is any measurement of whether the agent does those jobs well. product_ops — the PM persona (agent/prompts/product_ops.txt, registered in PROMPT_TEMPLATE_KEYS) — has zero golden cases. Every PM-shaped claim about the product is therefore unmeasured, and the ① Define precondition in docs/architecture/specialist-expert-level.en.md:164 ("5-10 task types + eval cases") is unmet for this family.

ADR-035 §2 is explicit — reuse, don't fork. This suite is a new golden file under the existing runner, not a new instrument.

Scope

What	Where	Code change
20 PM cases (10 task types × 2)	agent/evals/domain/golden/product_ops.json	none — run.py::_load_goldens globs golden/*.json
4 synthetic KB fixture docs	agent/evals/domain/golden/kb/product_ops/	none
Per-case generation cap	harness.py — new optional case field max_output_tokens + DOMAIN_EVAL_GEN_MAX_TOKENS (default 600, unchanged)	small
Judge output cap	harness.py — DOMAIN_EVAL_JUDGE_MAX_TOKENS (default 800, was a hard-coded 400)	small
Free behavior coverage	policy@product_ops — the 11 existing policy cases auto-expand onto the new role (run.py:80-92)	none

Non-goals (deferred, with reasons):

• Tool-calling / multi-turn cases. The harness is one completion from two messages (harness.py:198-224); the runbook forbids authoring tool-dependent cases (docs/runbooks/golden-set-authoring.md:63); ADR-035:70 lists tool-mock injection and a multi-turn driver as unbuilt execution paths. GitHub is not even a runtime capability — no entry in AGENT_TOOL_REGISTRY, not in NANGO_CONNECT_PROVIDERS, and no write-class tool exists anywhere.
• DB mirror into eval_golden_scenarios. eval-gate.processor.ts:142-146 runs every enabled scenario on every org config publish. Adding 20 PM cases before this group's noise band is measured is blast radius for no signal. Revisit after characterization.
• A second cross-role suite. The suite mechanism is hard-coded to the literal "policy" in run.py:39,67-68,89-91; the PM red lines are domain-specific and belong in the role file anyway.
• Scoreboard A. It does not exist as an instrument — only as prose in docs/plans/2026-07-27-week-increment-plan.md:33 (blind side-by-side, win-or-tie ≥ 70%). This suite does not replace that human ritual.
• Human baseline. Cannot be manufactured; see Open prerequisites.

Design

1. Role and placement

role: "product_ops" — the production PM persona, not an eval-only fiction. Measuring anything else would measure a persona no Specialist is ever assigned. Dropping the file in also creates a policy@product_ops group for free, so clarify-vs-proceed / no-fabrication / escalation get covered for this persona at zero authoring cost, per the runbook's routing rule (:13): behavior in the policy suite, facts + red lines in the role file.

2. Case matrix — 10 task types × 2

Two cases per type: an -a task-success case (TEMPLATE archetype 1/3/4) and a -b case whose question tempts the failure (archetype 2). Case ids are permanent — compare.py pairs on (report_key, id), so a rename shows up as an unpaired case and silently weakens every future back-test.

ids	task_type	tier	-a task success	-b tempted failure
po-001 / po-002	status_rollup	read_draft	workstream dump → blockers + this-week asks, owner per blocker	dump is silent on DNS → invents a DNS status instead of naming the gap
po-003 / po-004	activity_summary	read_draft	Slack digest + commit list → one line per person with channel/commit refs	a listed person has no activity → fabricates work; states archive chatter as confirmed fact instead of attributing to author and date
po-005 / po-006	executive_report	read_draft	3 projects → 3 sentences each, material changes only	an explicitly unconfirmed figure → reported as settled fact
po-007 / po-008	meeting_notes_to_issues	external_side_effect	sync notes → numbered issue titles, explicitly awaiting approval	claims the issues were created + invents an item absent from the notes
po-009 / po-010	contract_review	read_draft	contract text → 3 risks including payment terms	a definitive legal opinion that it is safe to sign (persona's legal escalation contract)
po-011 / po-012	bug_from_screenshot	external_side_effect	UX report → repro steps, expected vs actual, severity	claims the issue was filed + invents browser/version/env not in the report
po-013 / po-014	stale_work_sweep	read_draft	PR/issue dump → fixed vs remaining counts, arithmetic on given state	KB has the PR-handling convention → invents a handling policy instead of citing the standard
po-015 / po-016	definition_of_done	read_draft	KB launch-readiness standard → DoD covering DNS / email-DWD per specialist / onboarding	asserts DNS records or DWD scopes absent from the KB
po-017 / po-018	demo_plan	read_draft	KB surface inventory → ordered end-to-end demo flow	demos a surface the inventory marks unshipped
po-019 / po-020	task_triage	external_side_effect	board dump + roster → suggested owner + rationale per item	claims to have assigned + assigns someone absent from the roster or marked unavailable

The three external_side_effect types carry the suite's most business-legible gate: propose, never claim to have acted. That is exactly what the user's own prompts ask for ("titles first for review, then create them after I approve"), it is the D1 risk-tier boundary, and — being a must_not_include string — it is a hard rubric gate rather than a graded opinion (ADR-035 §4: fabrication-class gates are hard, graded scores are advisory).

Question phrasing stays close to EVAL_PROMPTS so the production keyword classifier still fires on it; paraphrasing away from those substrings would make production task_type null and decouple the offline case from its Scoreboard-B slice.

3. Fact plumbing

Per the runbook's rule of thumb (:65) — doc-shaped fact → kb/ fixture; real-time figure → context:

• context injection (14 cases): Slack digests, commit lists, PR/issue dumps, meeting notes, contract text, board state, UX verbatims. Runs in the cheap domain-eval.yml with no pgvector. Tests faithful use of given state.
• Seeded KB, real retrieval (6 cases: po-013…po-020 minus the two pure-context ones): kb_dir: "kb/product_ops" with team-roster, launch-readiness-standard, pr-handling-convention, product-surface-inventory. The production retrieval pipeline must surface the doc, and kb_gold_docs makes the retrieval-miss vs hallucination split computable (ADR-035 §4 — "must be distinguished, or the metric lies"). Requires the run-domain-eval-kb workflow.

All fixtures are synthetic and PII-free (ADR-035 §1, ADR-020): invented company, invented people, invented figures. The shape mirrors our real situation — a domain needing DNS records, Google DWD per specialist, onboarding completion, gitflow PR handling — so the competence measured is identical, while no real org state or teammate name enters a committed fixture.

4. Harness knobs — the 600-token cap is an instrument bug for this family

1. Generation. generate_answer hard-caps max_tokens=600 (harness.py:221). A definition-of-done document, a per-person activity table, or a 3-project exec report exceeds that, so the judge would grade truncated text. Fix: optional per-case max_output_tokens, plus DOMAIN_EVAL_GEN_MAX_TOKENS. The default stays 600 so existing ecommerce / marketing / policy baselines do not move — only cases that opt in get more room.
2. Judge. judge caps max_tokens=400 (harness.py:265). Long PM rubric strings can truncate the judge's JSON; that vote then lacks alignment, is dropped (harness.py:430), and at K=1 the case errors on quorum instead of scoring. Fix: DOMAIN_EVAL_JUDGE_MAX_TOKENS, default 800. Raising an output ceiling cannot change a response that was not truncating, so this is strictly safe.

Neither knob is mirrored into eval_golden_scenarios — consistent with the no-DB-mirror decision above.

5. What this measures, and what it does not

Measures: PM judgment and synthesis quality given the relevant state — does the reply name blockers and owners, do the arithmetic on the given dump, cover the stated done-criteria, flag the payment-terms risk, attribute claims to a source, propose instead of claiming to have acted, and refrain from inventing facts the state does not contain.

Does not measure: whether the agent can fetch that state (no tool channel), multi-turn clarify-then-produce, actual issue creation (no write tool exists), or reading an attached PDF/screenshot as a file (EvalScenario has no artifact field — contract text and UX reports are inlined as text).

Honesty rails, inherited: correctness is the judge's ordinal opinion, never a quality percentage (README "Interpretation limits"); the sandbox advises and never gates (ADR-035 limitation 1); 20 cases is an anchor slice below the 50-100/role doctrine (runbook:143), so absolute means are not quoted outside the team; report DOMAIN_EVAL_JUDGE_VOTES=5 over ≥3 runs and trust within-batch deltas only. A PM correctness figure is never a Scoreboard-B cell — docs/architecture/measurement-answer-key.md:106 keeps client-judged task success (a human judgment) separate from the internal LLM judge, on purpose.

6. Verification

1. Tokenless: pytest evals/test_domain_harness.py evals/test_domain_seeded_kb.py — the well-formedness validator (test_domain_harness.py:20-38: role + cases present, ids unique, rubric lists are lists) and the seeded-KB invariant (test_domain_seeded_kb.py:81-98: every kb_gold_docs id exists in kb_dir, and seeded cases only under a role declaring kb_dir), plus new tests for the two token knobs.
2. Live, context cases: python -m evals.domain.run --role product_ops with an OpenRouter key. Confirms end-to-end scoring and gives a first read on rubric ambiguity (key_mismatches must be empty — that field is instrument health, not quality).
3. Live, seeded cases: same command against a throwaway pgvector container with DOMAIN_EVAL_KB_PG_CONN_STR set, so the six retrieval cases actually run and report retrieval_hit / failure_class.

Per-case verdicts from those runs are the deliverable evidence — not a claim that the suite "works".

7. Open prerequisites (logged, not hidden)

• Human baseline for the ten task types — what a competent PM produces. Needs Percy / Alex O; specialist-expert-level.en.md:164 makes it a precondition of the family, and week-increment-plan.md:90 already says Scoreboard A "needs formal setup".
• Judge human spot-check (~10-15 cases, ADR-035 §5) — until it is done, every number here means "the judge's opinion", backed only by its own measured consistency.
• Noise band for the product_ops and policy@product_ops groups — 3 identical runs at judge_votes=5, appended to docs/testing/domain-eval-noise-characterization.md, before any level is quoted.
• Growth path — toward 50-100 cases/role, seeded from real high-editRatio PM asks (as inspiration only; fixtures stay synthetic).