Skip to main content

PRD โ€” AI Specialist Value Output ("Why clients choose us")

Status: Draft for review ยท Owner: Franky (Product) ยท Date: 2026-07-06 Strategic context: PRODUCT_STRATEGY.md โ€” this PRD is pillar โ‘ก of the current two-week focus ("AI Specialist value output"), alongside pillar โ‘  pipeline reliability. Architecture alignment: ADR-033 (layered Specialist architecture โ€” status: Draft) ยท ADR-034 (template scoping) ยท ADR-035 (eval & golden sets โ€” status: Proposed, not accepted) ยท ADR-037 (runtime read-only + human-approved promotion โ€” closes OQ-205) ยท ADR-008 (KB K1/K2) ยท ADR-020 (isolation) ไธญๆ–‡็‰ˆ: prd-specialist-value-output.zh.md


1. Problem statementโ€‹

Our differentiation is that a client "hires a professional digital colleague," not software. That professionalism currently rests on: (a) one hand-written systemPrompt blob per Specialist, (b) 4 hard-coded role templates, (c) SOUL.md / SKILL.md files checked into the code repo โ€” invisible to and unconfigurable by the AM/operator, and (d) a good retrieval chain. Meanwhile the richest quality signal we receive (Expert edits to drafts) is discarded.

Consequences:

  • Onboarding a new client/domain requires engineering work (repo files, prompt surgery) โ€” the "industry red-line engine" and Specialist competence cannot be reused as product assets.
  • A long persona can silently replace the whole role template (prompt-loader override heuristic, flagged in ADR-033) โ€” the #1 anti-pattern.
  • We cannot prove the Specialist is professional โ€” no domain-accuracy measurement, so config changes are un-testable and value claims are unverifiable.

2. Goalsโ€‹

  1. A Specialist's professional identity is defined by five operator-configurable assets โ€” Soul (who I am), Skills (how I work), KB (what I know), Tools (what I can do), and a Feedback loop (how I improve) โ€” managed in the Ops backend, not in the code repo.
  2. A new Specialist for a known domain can be configured, previewed, and validated without an engineer.
  3. Every configuration change is versioned, audited, and eval-gated (no silent quality regressions).
  4. The feedback loop turns Expert/client corrections into measurable weekly improvement of draft quality.

Each goal maps to a ยง6 metric (time-to-configure, golden-set pass rate, acceptance rate). Goals are outcomes; absolute targets are set after the measurement baseline lands (ยง6 note).

Non-goals (this PRD)โ€‹

  • Agent write-back of its own skills (OQ-205 โ€” closed by ADR-037: runtime read-only; the loop proposes, humans approve).
  • Fully autonomous tool execution: R6 wires tools to the chat path, but write-class actions always require Expert approval โ€” removing that gate is out of scope.
  • Bespoke per-client tool integrations beyond the registry/Nango/MCP configuration surface (custom integrations remain engineering work).
  • Marketing-vertical-specific tooling (#3699/#3701 โ€” gated on the ICP-expansion decision).
  • Full 11-stage ADR-033 pipeline; this PRD implements its L0/L1/L6 layers, the runtime context-assembly spine that applies them per turn (R5, the minimum of L2/L3), plus the minimum L7 needed to gate config changes.

3. Current state (grounded, as of origin/dev 2026-07-06; rows corrected against the 2026-07-07 code audit โ€” see the master plan ยง2.1)โ€‹

AssetWhere it lives todayConfigurable?
Personaspecialists.systemPrompt (one textarea in /ops/specialists); SOUL.md files in repo (orgs/<org>/...); merge fixed by PR #244Partially (blob only)
Role/domain templateagent/prompts/<key>.txt, closed union of 4 keys via specialists.prompt_template_key (ADR-034)Selectable, not editable
Skillsorgs/<org>/skills/**/SKILL.md in repo, read-only bind-mount (#226); M2 plan moves to R2 โ€” the M2 runtime-config HTTP channel already exists (agent startup GET /v1/internal/org/{slug}/runtime-config delivers soul_md + config_yaml + skills[])No (repo files; delivery channel built)
KBK1/K2 per ADR-008; client upload UI shipped (#297); hybrid retrieval + rerank (ADR-032); kb_retrieval_event telemetry joinable to messages (#3189)Yes (docs); no golden answers / structured rules
FeedbackCorrection entity + POST /learning/corrections; edit-diff is not discarded โ€” releaseDraft/respond capture wasEdited + editRatio + correctionCategory to Message.metadata (#3323 shipped); correction_embeddings is read for few-shot injection (#3406, flag correction_fewshot_enabled default OFF); correctionโ†’KB processor is complete but auto-ingested without approval โ€” now gated OFF by correction_auto_ingest_enabled per ADR-037Partially (signal captured; consumption flag-gated, human approval pending R4.4)
EvalADR-035 Proposed (eval sandbox + golden sets โ€” not yet accepted); agent/evals/domain/ golden-set + LLM-judge + seeded-KB harness exists (#3324; judge not human-validated)Not wired to config changes

4. Requirementsโ€‹

User storiesโ€‹

  • As an AM, I want to configure a new Specialist's Soul, Skills, and KB from a client brief without engineering help, so a new client can go live in under a day.
  • As a SuperAdmin, I want catalog-level archetypes I can materialize and override per org, so domain competence is a reusable asset across clients.
  • As an Expert, I want each draft to show which skill and KB sources produced it and any red-line violations, so I can review quickly and trust what I send.
  • As an Expert, I want my edits captured automatically as improvement signals, so serving clients well is training the Specialist.
  • As a client user, I want the same persona giving professionally correct answers on every channel, so it feels like one competent colleague โ€” not a bot with settings.
  • As an AM, I want to grant a Specialist scoped tools (e.g. order lookup, CRM read) so it answers with live data instead of just talking about the domain.

Permissions & publish flowโ€‹

Edit โ‰  publish. Drafting config changes is free; publishing is a separate action that runs the eval gate (R4.6) and is always audited.

AssetEditPublish (eval-gated)Notes
Catalog Soul / Skill (archetype)SuperAdminSuperAdminCD-19: SA owns Specialists
Org-instance Soul / Skill overrideAM + SuperAdminAM + SuperAdminOrg ร— Specialist scoped (ADR-020)
K2 KB (Specialist-global)AM + SuperAdminAM + SuperAdmin
K1 KB (client documents)Client admin (existing UI)n/a โ€” retrieval-time asset, no publish step
Tool bindings (per Specialist instance)AM + SuperAdminAM + SuperAdmingrants data access; read tools allow-listed, write tools Expert-approved per call (R6.4)

Non-functional requirementsโ€‹

#Constraint
NFR1Latency: assembly + retrieval cascade adds โ‰ค 300ms p95 over the current draft path; each cascade stage is budget-skippable (R5.3).
NFR2Async publish: the eval gate never blocks live chat traffic; publishing enters a "publishing (eval running)" state and completes asynchronously.
NFR3Truncation order under token pressure: few-shot โ†’ history โ†’ supplementary KB โ†’ skill templates. Red lines and Soul identity are never truncated.
NFR4Trace data: R5.4 trace rows are Org ร— Specialist-scoped (ADR-020 row named in the entity docstring), default retention 90 days.
NFR5Config content safety: all operator-entered Soul/Skill/template content is validated against the AI-output discipline rules (#3054/#3500 โ€” no infrastructure details or server paths embeddable).
NFR6Tool outputs are untrusted input (R6): never interpreted as instructions, injection-guarded, and scrubbed per #3054/#3500 before entering any client-visible draft.

R1 โ€” Soul: independent persona, backend-configurable ใ€P0ใ€‘โ€‹

The Specialist has a distinct, persistent personality that survives across channels and conversations, defined as structured data, not a prompt blob.

#Requirement
R1.1Structured Soul schema per Specialist: identity (name, role, bio), voice & tone rules, boundaries (what it never does/says), languages, sign-off conventions. Stored platform-side (DB; large assets in R2), not in the code repo.
R1.2Ops backend editor (/ops/specialists): form-based Soul editing with live prompt-assembly preview; replaces the raw systemPrompt textarea. The raw textarea survives only as an "advanced" escape hatch with an explicit deprecation criterion: removed once all managed Specialists are migrated AND it has been unused for 4 consecutive weeks (it bypasses structured red lines and must not live forever).
R1.3Layered prompt assembly on the agent: Soul NEVER replaces the role template โ€” fix the persona-override heuristic (ADR-033 design rule: layers stay separable).
R1.4Versioning + audit: every Soul change records who/when/what; one-click rollback to a prior version.
R1.5Migration: existing repo SOUL.md content imported into the new store; repo files retired for platform-managed Specialists. Runtime remains read-only toward persona content (ADR-037).

Acceptance criteria (R1):

  • Creating/editing a Soul produces a versioned record (author, timestamp); rollback restores the exact prior assembled output.
  • The assembled-prompt preview is byte-identical to what the agent receives for the same config version.
  • A Soul exceeding its token budget is rejected at save time with a field-level error โ€” never silently truncated at runtime.
  • Given an oversized persona, the role-template sections still appear in the assembled prompt (override-heuristic regression test).

R2 โ€” Skills: domain-scenario capabilities, backend-configurable ใ€P0ใ€‘โ€‹

A Skill = a scenario-scoped instruction pack (trigger + procedure + templates + red lines) that makes the Specialist competent at a concrete task (e.g. "Launchpad campaign thread", "sanctions-check escalation").

#Requirement
R2.1Skill data model: platform-side skill registry; each skill has trigger conditions (scenario/topic), instructions, response templates, escalation rules, a structured red-line section (hard rules the draft must obey), and a reserved tools field declaring which tool capabilities the skill depends on (consumed by R6 โ€” reserved now to avoid Phase-6 rework). Scoped Org ร— Specialist per ADR-020; catalog-archetype vs materialized-instance distinction respected (#3447).
R2.2Ops backend CRUD: create/edit/clone skills; bind skills to a Specialist instance; import existing repo SKILL.md files.
R2.3Agent runtime loads skills from the platform store (R2/DB) instead of repo bind-mounts; read-only contract preserved (#226).
R2.4Per-turn skill selection: extend the ADR-034 resolution chain so the active skill(s) for a turn are chosen by trigger match, with the role template as fallback โ€” never a silent cross-domain default (the #1203 bleed class).
R2.5Red-line enforcement hook: the structured red lines from active skills are (a) injected as constraints at draft time and (b) exposed to a post-draft check that flags violations to the Expert. (Full guardrail engine is a follow-on; this creates its data substrate.)
R2.6Archetype โ†’ instance inheritance semantics: materializing a catalog skill for an org is copy-on-materialize; catalog updates never propagate silently โ€” instances see an explicit "upstream changed, pull update?" affordance, and pulling re-runs the eval gate. (Silent propagation would bypass R4.6.)

Acceptance criteria (R2):

  • A message matching a skill trigger activates it, and the assembly trace lists the skill id + version.
  • A message matching no skill falls back to role template + abstain posture; it never activates another domain's skill (#1203 regression test).
  • Editing a skill creates a new version; in-flight conversations keep the version active at turn start.
  • Skill content violating output-discipline rules (NFR5) is rejected at save time.
  • A catalog-skill update does not change any materialized instance until explicitly pulled and re-evaled.

R3 โ€” KB: domain knowledge, backend-configurable ใ€P0ใ€‘โ€‹

Mostly built โ€” this requirement completes it rather than rebuilds it.

#Requirement
R3.1Golden answers as a first-class KB type: curated Qโ†’A pairs per Specialist domain, retrieved with priority and used verbatim-with-adaptation (ADR-033 L1).
R3.2Structured domain rules: policy/eligibility/process facts as structured entries (effective_date / audience / jurisdiction metadata already supported by retrieval โ€” expose in authoring UI).
R3.3AM/Ops KB curation surface for K2 (Specialist-global) content, complementing the client-facing K1 upload UI.
R3.4KB coverage report: which client questions produced no/weak KB hits (from kb_retrieval_event joined to messages) โ€” the authoring to-do list for AMs.
R3.5KB ingestion connectors: scheduled sync from external sources (Notion / Drive / websites, via MCP or Nango) into K1/K2. Synced content enters the same curation / versioning / eval path as uploads โ€” live external data never bypasses the KB store into retrieval (that would break versioning, eval, and isolation; live data belongs to Tools, R6).

Acceptance criteria (R3):

  • A question with a matching golden answer retrieves it ranked above hybrid results.
  • Golden answers / structured rules carry effective_date / audience / jurisdiction metadata and are filtered by it at retrieval.
  • The coverage report lists no-hit/weak-hit turns linked to their source conversations.
  • Connector-synced documents are versioned and re-embedded on change; a failing connector degrades to stale-but-served KB, never blocks retrieval.

R4 โ€” Feedback loop: measurable continuous improvement ใ€P1ใ€‘โ€‹

Every Expert edit and client reaction becomes a signal; signals become KB/skill/calibration improvements; improvement is visible on a dashboard.

#Requirement
R4.1Edit-diff capture in releaseDraft (#3323): persist wasEdited, diff, edit-distance, edit category. Pull-forward candidate: this is the measurement baseline for everything else โ€” recommend landing it inside the two-week window even though R4 is P1 overall.
R4.2Draft quality metrics: acceptance rate (sent-unedited %), edit distance trend, per Specialist ร— Org ร— skill; joinable to KB retrieval events.
R4.3One-click correction taxonomy at release time (factual / tone / missing-step / wrong-action) โ€” completes the existing Correction capture UI gap.
R4.4Correction โ†’ KB refinement consumer (completes #303): corrections generate proposed KB/golden-answer edits; human (AM/Expert) approves; nothing self-applies (ADR-037). Retrofit, not greenfield: the processor already classifies, dedupes, and wrote KB directly without approval โ€” ADR-037 gates that behavior OFF (correction_auto_ingest_enabled); R4.4 replaces it with the proposeโ†’approve flow.
R4.5Few-shot reuse: retrieve relevant past corrections (correction_embeddings, currently write-only) into the drafting context.
R4.6Eval gate on config change (ADR-035): per-Specialist golden set; editing Soul/Skill/KB triggers an eval run in the sandbox; regressions block publish (override with audit). Without this, R1โ€“R3 configurability is regression roulette.
R4.7Client feedback capture: lightweight per-reply reaction (๐Ÿ‘/๐Ÿ‘Ž + optional comment) in the client portal, feeding the same signal store.

Acceptance criteria (R4):

  • 100% of released drafts persist wasEdited + diff + category (sampled audit against send log).
  • Publishing a config change whose golden-set score drops below baseline โˆ’ threshold is blocked; override requires a written reason and is audit-logged.
  • A correction-driven KB proposal is never applied without explicit human approval.

R5 โ€” Runtime application: context assembly & per-turn execution ใ€P0ใ€‘โ€‹

R1โ€“R3 (and R6's tool bindings) define the configurable assets; R5 is their consumer โ€” without it they are just config tables. R5 specifies how one inbound client message is turned into one professional draft using Soul + Skills + KB, deterministically and traceably.

Reference turn flow (Kaito example โ€” client asks "How do I participate in the XYZ launchpad?"):

1. Route        org=kaito, specialist=Amy โ†’ load her config
2. Trigger message matches skill "Launchpad FAQ triage"
โ†’ instructions + reply template + red lines (no price/allocation/vesting)
3. Retrieve golden answers first (participation steps + official links) โ†’ hybrid K1/K2 if thin
4. Reuse few-shot: how Experts edited similar replies before
5. Assemble role template (base) โ† Soul (identity/tone) โ† active skill(s)
โ† KB citations โ† history โ† few-shot
precedence: red lines > crisis > active directives > skill instructions > Soul > template; per-layer token budgets
6. Draft โ†’ { draft, confidence, risk }
7. Verify post-draft red-line check; violations flagged to the Expert
8. Abstain no skill match + weak KB โ†’ escalate posture, never bluff
#Requirement
R5.1Deterministic context assembler: fixed layer order, explicit precedence (red lines > crisis directive > active directives > skill instructions > Soul > role template), per-layer token budgets, KB content always cited. Absorbs the agent-side half of R1.3. Crisis directive and active_directives (L0.5, #3692) are already-shipped layers the assembler must preserve, not new work.
R5.2Trigger evaluation: multi-skill match merge/tie-break rules; explicit no-match fallback to role template + abstain posture (implements R2.4's runtime half; the #1203 bleed class stays dead).
R5.3Retrieval orchestration cascade: golden answers โ†’ hybrid K1/K2 retrieval โ†’ correction few-shot, in that order, each stage skippable on budget (consumes R3.1, R4.5).
R5.4Per-turn assembly trace: persist which Soul version, which skills, which KB chunks and budgets produced each draft โ€” joinable to messages, edit-diffs, and eval runs. Serves debugging, eval attribution, and future client-facing quality receipts. Trace rows are Org ร— Specialist-scoped with 90-day default retention (NFR4).
R5.5Abstain / out-of-scope rerouting: standardized "I'll get back to you" + Expert escalation when confidence in assembly inputs is low (ADR-033 L5) โ€” the Specialist never bluffs outside its configured competence.

Acceptance criteria (R5):

  • Same message + same config versions + same KB snapshot โ†’ identical assembled context (determinism test).
  • Red-line sections appear in full in every assembled prompt regardless of budget pressure (truncation-order test, NFR3).
  • Every draft has a trace row (Soul version, skill ids + versions, KB chunk ids, budgets) โ€” drives the ยง6 trace-coverage metric.
  • When assembly inputs fall below the confidence floor, the draft is the standard abstain reply and the queue item is flagged accordingly.

R6 โ€” Tools: capabilities the Specialist can execute, backend-configurable ใ€P1ใ€‘โ€‹

KB answers "what the Specialist knows"; Tools answer "what it can do". Without tools the Specialist talks about KYC/orders/data; with them it fetches real facts and performs real (approved) actions โ€” this is the value ceiling and the pricing anchor. Builds on existing infrastructure: static AGENT_TOOL_REGISTRY (bespoke | Nango), ADR-029 execution runtime, ActionPolicyService policy gate (#689 tracks the MCP direction). Note: tool_specialist_bindings (the ADR-020 row) is to-be-created โ€” it does not exist today; current state is a specialists.tools[] string array + the static registry (expand-contract migration required).

Why Tools is not part of KB: knowledge is static, curated, versioned, and eval-able; tools are live, executable, and side-effectful. Mixing live tool data into KB retrieval breaks versioning (golden sets can't pin live data), isolation (external sources have no Org ร— Specialist boundary), and the layer-separation rule (ADR-033).

#Requirement
R6.1Tool sources: the registry supports bespoke, Nango, and MCP servers as a third tool-source kind; MCP server config (endpoint, auth, allowed tool subset) is operator-managed integration data under ADR-030 credential isolation.
R6.2Per-Specialist tool bindings configurable in the Ops backend (creates tool_specialist_bindings โ€” to-be-created, migrating off specialists.tools[]); catalog archetypes declare default toolsets, instances override โ€” R2.6 inheritance semantics apply.
R6.3Chat-path wiring: the assembler (R5) exposes to the draft engine exactly active skills' declared tools โˆฉ Specialist bindings โ€” no active skill, no tool.
R6.4Policy gate on every call (ActionPolicyService): read-class tools allow-listed per binding; write-class tools create an approval item in the Expert queue โ€” tool approval is part of the review loop, never a bypass of it.
R6.5Tool outputs are untrusted input (NFR6); every call audited with osa_id (existing ToolCall audit); tool errors fail open to "draft without the tool + flag to Expert", never a broken turn.

Acceptance criteria (R6):

  • A tool that is not both bound to the Specialist and declared by an active skill is never exposed to the draft engine (deny-by-default test).
  • A write-class tool call creates an Expert approval item; nothing executes before approval.
  • MCP/tool credentials are isolation-scoped and never appear in prompts, traces, or client-visible output.
  • Tool output containing adversarial instructions does not alter Specialist behavior (injection regression test).
  • Tool failure during drafting yields a tool-less draft flagged for the Expert โ€” never an error surfaced to the client.

5. Design principlesโ€‹

  1. Layers stay separable (ADR-033): Soul (L0), Skills/rules (L1/L3), KB (L1), verification (L4), feedback (L6) are distinct assets โ€” never re-collapsed into one prompt.
  2. Config is data, not prose: structured fields with validation, not bigger textareas.
  3. Measure before/while configuring: no configurability ships without the eval gate + edit-signal baseline (R4.1, R4.6).
  4. Human-approved improvement: the loop proposes; operators approve (OQ-205 read-only default, now ADR-037).
  5. Isolation by construction: every new table carries the Org ร— Specialist scope per ADR-020; new entities name their isolation-matrix row in the docstring.
  6. Every turn is traceable: any draft can be explained by exactly which asset versions, skills, and KB chunks produced it (R5.4).

6. Success metricsโ€‹

MetricDefinitionDirection
Draft acceptance rate% of drafts sent without Expert editโ†‘ weekly
Mean edit distancenormalized diff size on edited draftsโ†“
Time-to-configurehours from "new client brief" to "validated Specialist config" (no engineer)< 1 day
KB no-hit rate% of client turns with no/weak retrieval hitโ†“
Golden-set pass rateper-Specialist eval score, run on every config changeโ‰ฅ baseline, never silently โ†“
Signal capture rate% of released drafts with edit-diff + category recordedโ†’ 100%
Assembly trace coverage% of drafts with a full per-turn asset trace (R5.4)โ†’ 100%

Baseline & targets: absolute targets are deliberately unset โ€” they are fixed two weeks after T0.1 lands (first edit-signal baseline). Measurement sources: acceptance/edit metrics from the T0.1 release-log capture; no-hit rate from kb_retrieval_event; golden pass rate from ADR-035 eval runs; trace coverage from R5.4 rows joined to sent drafts.

7. Task breakdownโ€‹

Legend: [BE] NestJS ยท [FE] frontend ยท [AG] agent ยท [DS] design/decision. Ordering within a phase = suggested sequence. Sizes are relative (S/M/L).

Phase 0 โ€” Foundations (do first, this window)โ€‹

IDTaskSizeDepends on
T0.1[BE] Edit-diff capture in releaseDraft (R4.1, #3323): persist wasEdited/diff/distance/category on release; backfill-safe (done โ€” landed 2026-07-07, feat/specialist-value-output-phase0)Sโ€”
T0.2[BE] Draft-quality metrics endpoints + minimal /ops dashboard card (R4.2) (done โ€” landed 2026-07-07)ST0.1
T0.3[DS] Soul & Skill schema design doc (field-level), incl. red-line section, versioning model, R2-vs-DB storage split, and archetypeโ†’instance inheritance semantics (R2.6: copy-on-materialize + explicit pull); review against ADR-033 L0/L1 (design doc drafted 2026-07-07 โ€” unified config asset model, pending review)Mโ€”
T0.4[DS] Confirm OQ-205 default (runtime read-only; loop proposes, human approves) โ€” promote to ADR (done โ€” ADR-037, 2026-07-07)Sโ€”

Phase 1 โ€” Configurable Soul (R1)โ€‹

IDTaskSizeDepends on
T1.1[BE] Soul entity + migration (versioned, audited, Orgร—Specialist-scoped where org-overridden; catalog-level for archetypes)MT0.3
T1.2[BE] Soul CRUD API + validation; import script from repo SOUL.md + existing systemPromptMT1.1
T1.3[FE] /ops/specialists Soul editor: structured form + assembled-prompt preview + version history/rollbackMT1.2
T1.4[AG] Layered prompt assembly: consume structured Soul via agent-api; remove persona-override heuristic; Soul can never displace the role templateMT1.2
T1.5[BE/AG] Retire repo SOUL.md path for platform-managed Specialists; runtime read-onlyST1.4

Phase 2 โ€” Configurable Skills (R2)โ€‹

IDTaskSizeDepends on
T2.1[BE] Skill entity + bindings (new binding table โ€” none exists today; ADR-020 row) + migration; archetype/instance split per #3447MT0.3
T2.2[BE] Skill CRUD API + repo SKILL.md import (seed from Kaito/Acme skill files)MT2.1
T2.3[FE] Skill management UI: catalog, editor (trigger/instructions/templates/red-lines sections), bind-to-SpecialistLT2.2
T2.4[AG] Platform-loaded skills replace repo bind-mount; per-turn trigger-based skill selection extending ADR-034 chain; explicit no-match fallback (no silent cross-domain default)LT2.2
T2.5[AG/BE] Red-line injection at draft time + post-draft violation flag surfaced in the Expert workspaceMT2.4

Phase 3 โ€” KB completion (R3)โ€‹

IDTaskSizeDepends on
T3.1[BE] Golden-answer KB type + priority retrieval pathMโ€”
T3.2[FE] K2 curation surface for AM/Ops (structured rules, metadata: effective_date/audience/jurisdiction)MT3.1
T3.3[BE/FE] KB coverage report from kb_retrieval_event ร— messages ("questions we couldn't answer")Mโ€”

Phase 4 โ€” Feedback loop close (R4)โ€‹

IDTaskSizeDepends on
T4.1[FE] One-click correction category at release (R4.3)ST0.1
T4.2[BE] Correction โ†’ proposed KB/golden-answer edits, human-approved queue (R4.4, completes #303)MT3.1, T4.1
T4.3[AG/BE] Few-shot retrieval of past corrections into drafting context (R4.5, activates correction_embeddings)MT0.1
T4.4[BE/AG] Eval gate: per-Specialist golden set store + sandbox run on Soul/Skill/KB publish; block-on-regression with audited override (R4.6, ADR-035)LT1.x, T2.x, T3.1
T4.5[FE/BE] Client ๐Ÿ‘/๐Ÿ‘Ž per reply + signal ingestion (R4.7)Sโ€”
T4.6[BE/FE] Improvement dashboard: acceptance rate / edit distance / no-hit rate / golden pass rate over time (ยง6 metrics)MT0.2, T4.4

Phase 5 โ€” Runtime application (R5)โ€‹

T1.4 and T2.4's agent-side work is implemented inside this module: T5.1 owns the assembler (absorbing T1.4), T5.2 owns trigger selection (absorbing T2.4's runtime half). They stay listed above for traceability but share one spec and owner here.

IDTaskSizeDepends on
T5.1[AG] Deterministic context assembler: layer order, precedence (red lines > crisis > active directives > skill > Soul > template, per R5.1), per-layer token budgets, mandatory KB citations (R5.1)LT0.3, T1.2, T2.2
T5.2[AG] Trigger evaluation engine: multi-skill merge/tie-break, explicit no-match โ†’ abstain posture (R5.2, R5.5)MT2.2
T5.3[AG/BE] Retrieval orchestration cascade: golden โ†’ hybrid โ†’ correction few-shot (R5.3)MT3.1
T5.4[AG/BE] Per-turn assembly trace, persisted and joinable to messages/edit-diffs/eval runs (R5.4). Minimal version (current assembly path: template key, prompt hash, KB chunks) lands in the two-week window (minimal version landed 2026-07-07); full version follows T5.1Mโ€” (minimal) / T5.1 (full)

Phase 6 โ€” Configurable Tools (R6)โ€‹

IDTaskSizeDepends on
T6.1[BE] MCP tool-source kind in the registry + MCP server integration config (ADR-030 isolation)MT0.3
T6.2[BE/FE] Per-Specialist tool-binding config UI (catalog defaults + instance overrides)MT6.1, T2.2
T6.3[AG] Chat-path tool exposure via the assembler (skills โˆฉ bindings) + execution through the policy gateLT5.1, T6.2
T6.4[BE/FE] Write-action approval items in the Expert queue + audit wiringMT6.3
T6.5[BE] KB ingestion connectors (R3.5): scheduled external-source sync into K1/K2Mโ€”

Dependency spineโ€‹

T0.3 โ”€โ”€โ–บ T1.x (Soul) โ”€โ”€โ”
T0.3 โ”€โ”€โ–บ T2.x (Skills) โ”€โ”ผโ”€โ”€โ–บ T5.1/T5.2 (assembler) โ”€โ”€โ–บ T4.4 (eval gate) โ”€โ”€โ–บ T4.6 (dashboard)
T3.1 (golden) โ”€โ”ดโ”€โ”€โ–บ T5.3 (retrieval cascade)
T0.1 (edit signal) โ”€โ”€โ–บ T0.2 โ”€โ”€โ–บ T4.1/T4.2/T4.3
T5.4 trace: minimal now โ”€โ”€โ–บ full with T5.1
T2.2 + T5.1 โ”€โ”€โ–บ T6.x (tools: bindings โ†’ chat-path โ†’ approvals)

Two-week window cut (aligned with the team announcement): T0.1โ€“T0.4 + T5.4 (minimal trace) + start T1.1/T1.2 and T2.1/T2.2. Everything else sequences behind.

Capacity note: this window runs concurrently with pillar โ‘  (pipeline reliability), which draws on the same backend engineers. T0.x tasks are deliberately sized S/M so the two pillars fit side by side; if contention bites, pillar โ‘  wins and Phase 1/2 starts slip โ€” the T0.x measurement tasks do not.

8. Rollout & migrationโ€‹

The riskiest moment of this PRD is switching a live paying client (Kaito/Amy) from the legacy prompt path to the new assembler. Rules:

  1. Shadow mode first. The new assembler runs alongside the legacy path: both produce drafts, both are logged and eval-compared; the client-facing draft stays legacy until cutover. Shadow deltas feed the golden set.
  2. Per-Specialist cutover flag (existing feature-flags module), with one-click rollback to the legacy path. No global switch.
  3. Cutover order: dogfood first (Eleanora / Acme) โ†’ non-critical Specialists โ†’ Kaito (Amy) last, and only after N days of shadow parity (eval delta โ‰ฅ 0, no new red-line violations).
  4. Content migration is dry-runnable: the SOUL.md / SKILL.md / systemPrompt import scripts (T1.2, T2.2) support dry-run + diff; imported config must pass the eval gate before its first publish.
  5. Legacy retirement: repo files and the raw systemPrompt path are removed only after all managed Specialists are cut over AND the escape hatch has been unused for 4 consecutive weeks (R1.2 deprecation criterion).

9. Open questionsโ€‹

#QuestionOwnerBlocking?
Q1OQ-205 formal close: confirm read-only runtime + human-approved promotion as the ADR'd default (resolved โ€” ADR-037, 2026-07-07)Product / EBlocking โ€” resolved
Q2Soul/Skill storage split: what lives in PG vs R2 (size, versioning, eval-sandbox access)Eng (T0.3)Blocking โ€” gates T0.3
Q3Does prompt_template_key's closed union stay, or do skills eventually subsume role templates? (keep union short-term; revisit after Phase 2)Eng/ProductNon-blocking
Q4Client-visible feedback (R4.7) copy & placement โ€” must not invite clients to rate "the AI" in a way that undercuts the personaDesignNon-blocking

Review cadence: re-scope after Phase 1 ships; success metrics reviewed weekly against ยง6.