PRD โ AI Specialist Value Output ("Why clients choose us")
Status: Draft for review ยท Owner: Franky (Product) ยท Date: 2026-07-06 Strategic context: PRODUCT_STRATEGY.md โ this PRD is pillar โก of the current two-week focus ("AI Specialist value output"), alongside pillar โ pipeline reliability. Architecture alignment: ADR-033 (layered Specialist architecture โ status: Draft) ยท ADR-034 (template scoping) ยท ADR-035 (eval & golden sets โ status: Proposed, not accepted) ยท ADR-037 (runtime read-only + human-approved promotion โ closes OQ-205) ยท ADR-008 (KB K1/K2) ยท ADR-020 (isolation) ไธญๆ็: prd-specialist-value-output.zh.md
1. Problem statementโ
Our differentiation is that a client "hires a professional digital colleague," not software. That professionalism currently rests on: (a) one hand-written systemPrompt blob per Specialist, (b) 4 hard-coded role templates, (c) SOUL.md / SKILL.md files checked into the code repo โ invisible to and unconfigurable by the AM/operator, and (d) a good retrieval chain. Meanwhile the richest quality signal we receive (Expert edits to drafts) is discarded.
Consequences:
- Onboarding a new client/domain requires engineering work (repo files, prompt surgery) โ the "industry red-line engine" and Specialist competence cannot be reused as product assets.
- A long persona can silently replace the whole role template (prompt-loader override heuristic, flagged in ADR-033) โ the #1 anti-pattern.
- We cannot prove the Specialist is professional โ no domain-accuracy measurement, so config changes are un-testable and value claims are unverifiable.
2. Goalsโ
- A Specialist's professional identity is defined by five operator-configurable assets โ Soul (who I am), Skills (how I work), KB (what I know), Tools (what I can do), and a Feedback loop (how I improve) โ managed in the Ops backend, not in the code repo.
- A new Specialist for a known domain can be configured, previewed, and validated without an engineer.
- Every configuration change is versioned, audited, and eval-gated (no silent quality regressions).
- The feedback loop turns Expert/client corrections into measurable weekly improvement of draft quality.
Each goal maps to a ยง6 metric (time-to-configure, golden-set pass rate, acceptance rate). Goals are outcomes; absolute targets are set after the measurement baseline lands (ยง6 note).
Non-goals (this PRD)โ
- Agent write-back of its own skills (OQ-205 โ closed by ADR-037: runtime read-only; the loop proposes, humans approve).
- Fully autonomous tool execution: R6 wires tools to the chat path, but write-class actions always require Expert approval โ removing that gate is out of scope.
- Bespoke per-client tool integrations beyond the registry/Nango/MCP configuration surface (custom integrations remain engineering work).
- Marketing-vertical-specific tooling (#3699/#3701 โ gated on the ICP-expansion decision).
- Full 11-stage ADR-033 pipeline; this PRD implements its L0/L1/L6 layers, the runtime context-assembly spine that applies them per turn (R5, the minimum of L2/L3), plus the minimum L7 needed to gate config changes.
3. Current state (grounded, as of origin/dev 2026-07-06; rows corrected against the 2026-07-07 code audit โ see the master plan ยง2.1)โ
| Asset | Where it lives today | Configurable? |
|---|---|---|
| Persona | specialists.systemPrompt (one textarea in /ops/specialists); SOUL.md files in repo (orgs/<org>/...); merge fixed by PR #244 | Partially (blob only) |
| Role/domain template | agent/prompts/<key>.txt, closed union of 4 keys via specialists.prompt_template_key (ADR-034) | Selectable, not editable |
| Skills | orgs/<org>/skills/**/SKILL.md in repo, read-only bind-mount (#226); M2 plan moves to R2 โ the M2 runtime-config HTTP channel already exists (agent startup GET /v1/internal/org/{slug}/runtime-config delivers soul_md + config_yaml + skills[]) | No (repo files; delivery channel built) |
| KB | K1/K2 per ADR-008; client upload UI shipped (#297); hybrid retrieval + rerank (ADR-032); kb_retrieval_event telemetry joinable to messages (#3189) | Yes (docs); no golden answers / structured rules |
| Feedback | Correction entity + POST /learning/corrections; edit-diff is not discarded โ releaseDraft/respond capture wasEdited + editRatio + correctionCategory to Message.metadata (#3323 shipped); correction_embeddings is read for few-shot injection (#3406, flag correction_fewshot_enabled default OFF); correctionโKB processor is complete but auto-ingested without approval โ now gated OFF by correction_auto_ingest_enabled per ADR-037 | Partially (signal captured; consumption flag-gated, human approval pending R4.4) |
| Eval | ADR-035 Proposed (eval sandbox + golden sets โ not yet accepted); agent/evals/domain/ golden-set + LLM-judge + seeded-KB harness exists (#3324; judge not human-validated) | Not wired to config changes |
4. Requirementsโ
User storiesโ
- As an AM, I want to configure a new Specialist's Soul, Skills, and KB from a client brief without engineering help, so a new client can go live in under a day.
- As a SuperAdmin, I want catalog-level archetypes I can materialize and override per org, so domain competence is a reusable asset across clients.
- As an Expert, I want each draft to show which skill and KB sources produced it and any red-line violations, so I can review quickly and trust what I send.
- As an Expert, I want my edits captured automatically as improvement signals, so serving clients well is training the Specialist.
- As a client user, I want the same persona giving professionally correct answers on every channel, so it feels like one competent colleague โ not a bot with settings.
- As an AM, I want to grant a Specialist scoped tools (e.g. order lookup, CRM read) so it answers with live data instead of just talking about the domain.
Permissions & publish flowโ
Edit โ publish. Drafting config changes is free; publishing is a separate action that runs the eval gate (R4.6) and is always audited.
| Asset | Edit | Publish (eval-gated) | Notes |
|---|---|---|---|
| Catalog Soul / Skill (archetype) | SuperAdmin | SuperAdmin | CD-19: SA owns Specialists |
| Org-instance Soul / Skill override | AM + SuperAdmin | AM + SuperAdmin | Org ร Specialist scoped (ADR-020) |
| K2 KB (Specialist-global) | AM + SuperAdmin | AM + SuperAdmin | |
| K1 KB (client documents) | Client admin (existing UI) | n/a โ retrieval-time asset, no publish step | |
| Tool bindings (per Specialist instance) | AM + SuperAdmin | AM + SuperAdmin | grants data access; read tools allow-listed, write tools Expert-approved per call (R6.4) |
Non-functional requirementsโ
| # | Constraint |
|---|---|
| NFR1 | Latency: assembly + retrieval cascade adds โค 300ms p95 over the current draft path; each cascade stage is budget-skippable (R5.3). |
| NFR2 | Async publish: the eval gate never blocks live chat traffic; publishing enters a "publishing (eval running)" state and completes asynchronously. |
| NFR3 | Truncation order under token pressure: few-shot โ history โ supplementary KB โ skill templates. Red lines and Soul identity are never truncated. |
| NFR4 | Trace data: R5.4 trace rows are Org ร Specialist-scoped (ADR-020 row named in the entity docstring), default retention 90 days. |
| NFR5 | Config content safety: all operator-entered Soul/Skill/template content is validated against the AI-output discipline rules (#3054/#3500 โ no infrastructure details or server paths embeddable). |
| NFR6 | Tool outputs are untrusted input (R6): never interpreted as instructions, injection-guarded, and scrubbed per #3054/#3500 before entering any client-visible draft. |
R1 โ Soul: independent persona, backend-configurable ใP0ใโ
The Specialist has a distinct, persistent personality that survives across channels and conversations, defined as structured data, not a prompt blob.
| # | Requirement |
|---|---|
| R1.1 | Structured Soul schema per Specialist: identity (name, role, bio), voice & tone rules, boundaries (what it never does/says), languages, sign-off conventions. Stored platform-side (DB; large assets in R2), not in the code repo. |
| R1.2 | Ops backend editor (/ops/specialists): form-based Soul editing with live prompt-assembly preview; replaces the raw systemPrompt textarea. The raw textarea survives only as an "advanced" escape hatch with an explicit deprecation criterion: removed once all managed Specialists are migrated AND it has been unused for 4 consecutive weeks (it bypasses structured red lines and must not live forever). |
| R1.3 | Layered prompt assembly on the agent: Soul NEVER replaces the role template โ fix the persona-override heuristic (ADR-033 design rule: layers stay separable). |
| R1.4 | Versioning + audit: every Soul change records who/when/what; one-click rollback to a prior version. |
| R1.5 | Migration: existing repo SOUL.md content imported into the new store; repo files retired for platform-managed Specialists. Runtime remains read-only toward persona content (ADR-037). |
Acceptance criteria (R1):
- Creating/editing a Soul produces a versioned record (author, timestamp); rollback restores the exact prior assembled output.
- The assembled-prompt preview is byte-identical to what the agent receives for the same config version.
- A Soul exceeding its token budget is rejected at save time with a field-level error โ never silently truncated at runtime.
- Given an oversized persona, the role-template sections still appear in the assembled prompt (override-heuristic regression test).
R2 โ Skills: domain-scenario capabilities, backend-configurable ใP0ใโ
A Skill = a scenario-scoped instruction pack (trigger + procedure + templates + red lines) that makes the Specialist competent at a concrete task (e.g. "Launchpad campaign thread", "sanctions-check escalation").
| # | Requirement |
|---|---|
| R2.1 | Skill data model: platform-side skill registry; each skill has trigger conditions (scenario/topic), instructions, response templates, escalation rules, a structured red-line section (hard rules the draft must obey), and a reserved tools field declaring which tool capabilities the skill depends on (consumed by R6 โ reserved now to avoid Phase-6 rework). Scoped Org ร Specialist per ADR-020; catalog-archetype vs materialized-instance distinction respected (#3447). |
| R2.2 | Ops backend CRUD: create/edit/clone skills; bind skills to a Specialist instance; import existing repo SKILL.md files. |
| R2.3 | Agent runtime loads skills from the platform store (R2/DB) instead of repo bind-mounts; read-only contract preserved (#226). |
| R2.4 | Per-turn skill selection: extend the ADR-034 resolution chain so the active skill(s) for a turn are chosen by trigger match, with the role template as fallback โ never a silent cross-domain default (the #1203 bleed class). |
| R2.5 | Red-line enforcement hook: the structured red lines from active skills are (a) injected as constraints at draft time and (b) exposed to a post-draft check that flags violations to the Expert. (Full guardrail engine is a follow-on; this creates its data substrate.) |
| R2.6 | Archetype โ instance inheritance semantics: materializing a catalog skill for an org is copy-on-materialize; catalog updates never propagate silently โ instances see an explicit "upstream changed, pull update?" affordance, and pulling re-runs the eval gate. (Silent propagation would bypass R4.6.) |
Acceptance criteria (R2):
- A message matching a skill trigger activates it, and the assembly trace lists the skill id + version.
- A message matching no skill falls back to role template + abstain posture; it never activates another domain's skill (#1203 regression test).
- Editing a skill creates a new version; in-flight conversations keep the version active at turn start.
- Skill content violating output-discipline rules (NFR5) is rejected at save time.
- A catalog-skill update does not change any materialized instance until explicitly pulled and re-evaled.
R3 โ KB: domain knowledge, backend-configurable ใP0ใโ
Mostly built โ this requirement completes it rather than rebuilds it.
| # | Requirement |
|---|---|
| R3.1 | Golden answers as a first-class KB type: curated QโA pairs per Specialist domain, retrieved with priority and used verbatim-with-adaptation (ADR-033 L1). |
| R3.2 | Structured domain rules: policy/eligibility/process facts as structured entries (effective_date / audience / jurisdiction metadata already supported by retrieval โ expose in authoring UI). |
| R3.3 | AM/Ops KB curation surface for K2 (Specialist-global) content, complementing the client-facing K1 upload UI. |
| R3.4 | KB coverage report: which client questions produced no/weak KB hits (from kb_retrieval_event joined to messages) โ the authoring to-do list for AMs. |
| R3.5 | KB ingestion connectors: scheduled sync from external sources (Notion / Drive / websites, via MCP or Nango) into K1/K2. Synced content enters the same curation / versioning / eval path as uploads โ live external data never bypasses the KB store into retrieval (that would break versioning, eval, and isolation; live data belongs to Tools, R6). |
Acceptance criteria (R3):
- A question with a matching golden answer retrieves it ranked above hybrid results.
- Golden answers / structured rules carry effective_date / audience / jurisdiction metadata and are filtered by it at retrieval.
- The coverage report lists no-hit/weak-hit turns linked to their source conversations.
- Connector-synced documents are versioned and re-embedded on change; a failing connector degrades to stale-but-served KB, never blocks retrieval.
R4 โ Feedback loop: measurable continuous improvement ใP1ใโ
Every Expert edit and client reaction becomes a signal; signals become KB/skill/calibration improvements; improvement is visible on a dashboard.
| # | Requirement |
|---|---|
| R4.1 | Edit-diff capture in releaseDraft (#3323): persist wasEdited, diff, edit-distance, edit category. Pull-forward candidate: this is the measurement baseline for everything else โ recommend landing it inside the two-week window even though R4 is P1 overall. |
| R4.2 | Draft quality metrics: acceptance rate (sent-unedited %), edit distance trend, per Specialist ร Org ร skill; joinable to KB retrieval events. |
| R4.3 | One-click correction taxonomy at release time (factual / tone / missing-step / wrong-action) โ completes the existing Correction capture UI gap. |
| R4.4 | Correction โ KB refinement consumer (completes #303): corrections generate proposed KB/golden-answer edits; human (AM/Expert) approves; nothing self-applies (ADR-037). Retrofit, not greenfield: the processor already classifies, dedupes, and wrote KB directly without approval โ ADR-037 gates that behavior OFF (correction_auto_ingest_enabled); R4.4 replaces it with the proposeโapprove flow. |
| R4.5 | Few-shot reuse: retrieve relevant past corrections (correction_embeddings, currently write-only) into the drafting context. |
| R4.6 | Eval gate on config change (ADR-035): per-Specialist golden set; editing Soul/Skill/KB triggers an eval run in the sandbox; regressions block publish (override with audit). Without this, R1โR3 configurability is regression roulette. |
| R4.7 | Client feedback capture: lightweight per-reply reaction (๐/๐ + optional comment) in the client portal, feeding the same signal store. |
Acceptance criteria (R4):
- 100% of released drafts persist
wasEdited+ diff + category (sampled audit against send log). - Publishing a config change whose golden-set score drops below baseline โ threshold is blocked; override requires a written reason and is audit-logged.
- A correction-driven KB proposal is never applied without explicit human approval.
R5 โ Runtime application: context assembly & per-turn execution ใP0ใโ
R1โR3 (and R6's tool bindings) define the configurable assets; R5 is their consumer โ without it they are just config tables. R5 specifies how one inbound client message is turned into one professional draft using Soul + Skills + KB, deterministically and traceably.
Reference turn flow (Kaito example โ client asks "How do I participate in the XYZ launchpad?"):
1. Route org=kaito, specialist=Amy โ load her config
2. Trigger message matches skill "Launchpad FAQ triage"
โ instructions + reply template + red lines (no price/allocation/vesting)
3. Retrieve golden answers first (participation steps + official links) โ hybrid K1/K2 if thin
4. Reuse few-shot: how Experts edited similar replies before
5. Assemble role template (base) โ Soul (identity/tone) โ active skill(s)
โ KB citations โ history โ few-shot
precedence: red lines > crisis > active directives > skill instructions > Soul > template; per-layer token budgets
6. Draft โ { draft, confidence, risk }
7. Verify post-draft red-line check; violations flagged to the Expert
8. Abstain no skill match + weak KB โ escalate posture, never bluff
| # | Requirement |
|---|---|
| R5.1 | Deterministic context assembler: fixed layer order, explicit precedence (red lines > crisis directive > active directives > skill instructions > Soul > role template), per-layer token budgets, KB content always cited. Absorbs the agent-side half of R1.3. Crisis directive and active_directives (L0.5, #3692) are already-shipped layers the assembler must preserve, not new work. |
| R5.2 | Trigger evaluation: multi-skill match merge/tie-break rules; explicit no-match fallback to role template + abstain posture (implements R2.4's runtime half; the #1203 bleed class stays dead). |
| R5.3 | Retrieval orchestration cascade: golden answers โ hybrid K1/K2 retrieval โ correction few-shot, in that order, each stage skippable on budget (consumes R3.1, R4.5). |
| R5.4 | Per-turn assembly trace: persist which Soul version, which skills, which KB chunks and budgets produced each draft โ joinable to messages, edit-diffs, and eval runs. Serves debugging, eval attribution, and future client-facing quality receipts. Trace rows are Org ร Specialist-scoped with 90-day default retention (NFR4). |
| R5.5 | Abstain / out-of-scope rerouting: standardized "I'll get back to you" + Expert escalation when confidence in assembly inputs is low (ADR-033 L5) โ the Specialist never bluffs outside its configured competence. |
Acceptance criteria (R5):
- Same message + same config versions + same KB snapshot โ identical assembled context (determinism test).
- Red-line sections appear in full in every assembled prompt regardless of budget pressure (truncation-order test, NFR3).
- Every draft has a trace row (Soul version, skill ids + versions, KB chunk ids, budgets) โ drives the ยง6 trace-coverage metric.
- When assembly inputs fall below the confidence floor, the draft is the standard abstain reply and the queue item is flagged accordingly.
R6 โ Tools: capabilities the Specialist can execute, backend-configurable ใP1ใโ
KB answers "what the Specialist knows"; Tools answer "what it can do". Without tools the Specialist talks about KYC/orders/data; with them it fetches real facts and performs real (approved) actions โ this is the value ceiling and the pricing anchor. Builds on existing infrastructure: static AGENT_TOOL_REGISTRY (bespoke | Nango), ADR-029 execution runtime, ActionPolicyService policy gate (#689 tracks the MCP direction). Note: tool_specialist_bindings (the ADR-020 row) is to-be-created โ it does not exist today; current state is a specialists.tools[] string array + the static registry (expand-contract migration required).
Why Tools is not part of KB: knowledge is static, curated, versioned, and eval-able; tools are live, executable, and side-effectful. Mixing live tool data into KB retrieval breaks versioning (golden sets can't pin live data), isolation (external sources have no Org ร Specialist boundary), and the layer-separation rule (ADR-033).
| # | Requirement |
|---|---|
| R6.1 | Tool sources: the registry supports bespoke, Nango, and MCP servers as a third tool-source kind; MCP server config (endpoint, auth, allowed tool subset) is operator-managed integration data under ADR-030 credential isolation. |
| R6.2 | Per-Specialist tool bindings configurable in the Ops backend (creates tool_specialist_bindings โ to-be-created, migrating off specialists.tools[]); catalog archetypes declare default toolsets, instances override โ R2.6 inheritance semantics apply. |
| R6.3 | Chat-path wiring: the assembler (R5) exposes to the draft engine exactly active skills' declared tools โฉ Specialist bindings โ no active skill, no tool. |
| R6.4 | Policy gate on every call (ActionPolicyService): read-class tools allow-listed per binding; write-class tools create an approval item in the Expert queue โ tool approval is part of the review loop, never a bypass of it. |
| R6.5 | Tool outputs are untrusted input (NFR6); every call audited with osa_id (existing ToolCall audit); tool errors fail open to "draft without the tool + flag to Expert", never a broken turn. |
Acceptance criteria (R6):
- A tool that is not both bound to the Specialist and declared by an active skill is never exposed to the draft engine (deny-by-default test).
- A write-class tool call creates an Expert approval item; nothing executes before approval.
- MCP/tool credentials are isolation-scoped and never appear in prompts, traces, or client-visible output.
- Tool output containing adversarial instructions does not alter Specialist behavior (injection regression test).
- Tool failure during drafting yields a tool-less draft flagged for the Expert โ never an error surfaced to the client.
5. Design principlesโ
- Layers stay separable (ADR-033): Soul (L0), Skills/rules (L1/L3), KB (L1), verification (L4), feedback (L6) are distinct assets โ never re-collapsed into one prompt.
- Config is data, not prose: structured fields with validation, not bigger textareas.
- Measure before/while configuring: no configurability ships without the eval gate + edit-signal baseline (R4.1, R4.6).
- Human-approved improvement: the loop proposes; operators approve (OQ-205 read-only default, now ADR-037).
- Isolation by construction: every new table carries the Org ร Specialist scope per ADR-020; new entities name their isolation-matrix row in the docstring.
- Every turn is traceable: any draft can be explained by exactly which asset versions, skills, and KB chunks produced it (R5.4).
6. Success metricsโ
| Metric | Definition | Direction |
|---|---|---|
| Draft acceptance rate | % of drafts sent without Expert edit | โ weekly |
| Mean edit distance | normalized diff size on edited drafts | โ |
| Time-to-configure | hours from "new client brief" to "validated Specialist config" (no engineer) | < 1 day |
| KB no-hit rate | % of client turns with no/weak retrieval hit | โ |
| Golden-set pass rate | per-Specialist eval score, run on every config change | โฅ baseline, never silently โ |
| Signal capture rate | % of released drafts with edit-diff + category recorded | โ 100% |
| Assembly trace coverage | % of drafts with a full per-turn asset trace (R5.4) | โ 100% |
Baseline & targets: absolute targets are deliberately unset โ they are fixed two weeks after T0.1 lands (first edit-signal baseline). Measurement sources: acceptance/edit metrics from the T0.1 release-log capture; no-hit rate from
kb_retrieval_event; golden pass rate from ADR-035 eval runs; trace coverage from R5.4 rows joined to sent drafts.
7. Task breakdownโ
Legend: [BE] NestJS ยท [FE] frontend ยท [AG] agent ยท [DS] design/decision. Ordering within a phase = suggested sequence. Sizes are relative (S/M/L).
Phase 0 โ Foundations (do first, this window)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T0.1 | [BE] Edit-diff capture in releaseDraft (R4.1, #3323): persist wasEdited/diff/distance/category on release; backfill-safe (done โ landed 2026-07-07, feat/specialist-value-output-phase0) | S | โ |
| T0.2 | [BE] Draft-quality metrics endpoints + minimal /ops dashboard card (R4.2) (done โ landed 2026-07-07) | S | T0.1 |
| T0.3 | [DS] Soul & Skill schema design doc (field-level), incl. red-line section, versioning model, R2-vs-DB storage split, and archetypeโinstance inheritance semantics (R2.6: copy-on-materialize + explicit pull); review against ADR-033 L0/L1 (design doc drafted 2026-07-07 โ unified config asset model, pending review) | M | โ |
| T0.4 | [DS] Confirm OQ-205 default (runtime read-only; loop proposes, human approves) โ promote to ADR (done โ ADR-037, 2026-07-07) | S | โ |
Phase 1 โ Configurable Soul (R1)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T1.1 | [BE] Soul entity + migration (versioned, audited, OrgรSpecialist-scoped where org-overridden; catalog-level for archetypes) | M | T0.3 |
| T1.2 | [BE] Soul CRUD API + validation; import script from repo SOUL.md + existing systemPrompt | M | T1.1 |
| T1.3 | [FE] /ops/specialists Soul editor: structured form + assembled-prompt preview + version history/rollback | M | T1.2 |
| T1.4 | [AG] Layered prompt assembly: consume structured Soul via agent-api; remove persona-override heuristic; Soul can never displace the role template | M | T1.2 |
| T1.5 | [BE/AG] Retire repo SOUL.md path for platform-managed Specialists; runtime read-only | S | T1.4 |
Phase 2 โ Configurable Skills (R2)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T2.1 | [BE] Skill entity + bindings (new binding table โ none exists today; ADR-020 row) + migration; archetype/instance split per #3447 | M | T0.3 |
| T2.2 | [BE] Skill CRUD API + repo SKILL.md import (seed from Kaito/Acme skill files) | M | T2.1 |
| T2.3 | [FE] Skill management UI: catalog, editor (trigger/instructions/templates/red-lines sections), bind-to-Specialist | L | T2.2 |
| T2.4 | [AG] Platform-loaded skills replace repo bind-mount; per-turn trigger-based skill selection extending ADR-034 chain; explicit no-match fallback (no silent cross-domain default) | L | T2.2 |
| T2.5 | [AG/BE] Red-line injection at draft time + post-draft violation flag surfaced in the Expert workspace | M | T2.4 |
Phase 3 โ KB completion (R3)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T3.1 | [BE] Golden-answer KB type + priority retrieval path | M | โ |
| T3.2 | [FE] K2 curation surface for AM/Ops (structured rules, metadata: effective_date/audience/jurisdiction) | M | T3.1 |
| T3.3 | [BE/FE] KB coverage report from kb_retrieval_event ร messages ("questions we couldn't answer") | M | โ |
Phase 4 โ Feedback loop close (R4)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T4.1 | [FE] One-click correction category at release (R4.3) | S | T0.1 |
| T4.2 | [BE] Correction โ proposed KB/golden-answer edits, human-approved queue (R4.4, completes #303) | M | T3.1, T4.1 |
| T4.3 | [AG/BE] Few-shot retrieval of past corrections into drafting context (R4.5, activates correction_embeddings) | M | T0.1 |
| T4.4 | [BE/AG] Eval gate: per-Specialist golden set store + sandbox run on Soul/Skill/KB publish; block-on-regression with audited override (R4.6, ADR-035) | L | T1.x, T2.x, T3.1 |
| T4.5 | [FE/BE] Client ๐/๐ per reply + signal ingestion (R4.7) | S | โ |
| T4.6 | [BE/FE] Improvement dashboard: acceptance rate / edit distance / no-hit rate / golden pass rate over time (ยง6 metrics) | M | T0.2, T4.4 |
Phase 5 โ Runtime application (R5)โ
T1.4 and T2.4's agent-side work is implemented inside this module: T5.1 owns the assembler (absorbing T1.4), T5.2 owns trigger selection (absorbing T2.4's runtime half). They stay listed above for traceability but share one spec and owner here.
| ID | Task | Size | Depends on |
|---|---|---|---|
| T5.1 | [AG] Deterministic context assembler: layer order, precedence (red lines > crisis > active directives > skill > Soul > template, per R5.1), per-layer token budgets, mandatory KB citations (R5.1) | L | T0.3, T1.2, T2.2 |
| T5.2 | [AG] Trigger evaluation engine: multi-skill merge/tie-break, explicit no-match โ abstain posture (R5.2, R5.5) | M | T2.2 |
| T5.3 | [AG/BE] Retrieval orchestration cascade: golden โ hybrid โ correction few-shot (R5.3) | M | T3.1 |
| T5.4 | [AG/BE] Per-turn assembly trace, persisted and joinable to messages/edit-diffs/eval runs (R5.4). Minimal version (current assembly path: template key, prompt hash, KB chunks) lands in the two-week window (minimal version landed 2026-07-07); full version follows T5.1 | M | โ (minimal) / T5.1 (full) |
Phase 6 โ Configurable Tools (R6)โ
| ID | Task | Size | Depends on |
|---|---|---|---|
| T6.1 | [BE] MCP tool-source kind in the registry + MCP server integration config (ADR-030 isolation) | M | T0.3 |
| T6.2 | [BE/FE] Per-Specialist tool-binding config UI (catalog defaults + instance overrides) | M | T6.1, T2.2 |
| T6.3 | [AG] Chat-path tool exposure via the assembler (skills โฉ bindings) + execution through the policy gate | L | T5.1, T6.2 |
| T6.4 | [BE/FE] Write-action approval items in the Expert queue + audit wiring | M | T6.3 |
| T6.5 | [BE] KB ingestion connectors (R3.5): scheduled external-source sync into K1/K2 | M | โ |
Dependency spineโ
T0.3 โโโบ T1.x (Soul) โโโ
T0.3 โโโบ T2.x (Skills) โโผโโโบ T5.1/T5.2 (assembler) โโโบ T4.4 (eval gate) โโโบ T4.6 (dashboard)
T3.1 (golden) โโดโโโบ T5.3 (retrieval cascade)
T0.1 (edit signal) โโโบ T0.2 โโโบ T4.1/T4.2/T4.3
T5.4 trace: minimal now โโโบ full with T5.1
T2.2 + T5.1 โโโบ T6.x (tools: bindings โ chat-path โ approvals)
Two-week window cut (aligned with the team announcement): T0.1โT0.4 + T5.4 (minimal trace) + start T1.1/T1.2 and T2.1/T2.2. Everything else sequences behind.
Capacity note: this window runs concurrently with pillar โ (pipeline reliability), which draws on the same backend engineers. T0.x tasks are deliberately sized S/M so the two pillars fit side by side; if contention bites, pillar โ wins and Phase 1/2 starts slip โ the T0.x measurement tasks do not.
8. Rollout & migrationโ
The riskiest moment of this PRD is switching a live paying client (Kaito/Amy) from the legacy prompt path to the new assembler. Rules:
- Shadow mode first. The new assembler runs alongside the legacy path: both produce drafts, both are logged and eval-compared; the client-facing draft stays legacy until cutover. Shadow deltas feed the golden set.
- Per-Specialist cutover flag (existing feature-flags module), with one-click rollback to the legacy path. No global switch.
- Cutover order: dogfood first (Eleanora / Acme) โ non-critical Specialists โ Kaito (Amy) last, and only after N days of shadow parity (eval delta โฅ 0, no new red-line violations).
- Content migration is dry-runnable: the SOUL.md / SKILL.md / systemPrompt import scripts (T1.2, T2.2) support dry-run + diff; imported config must pass the eval gate before its first publish.
- Legacy retirement: repo files and the raw
systemPromptpath are removed only after all managed Specialists are cut over AND the escape hatch has been unused for 4 consecutive weeks (R1.2 deprecation criterion).
9. Open questionsโ
| # | Question | Owner | Blocking? |
|---|---|---|---|
| Q1 | OQ-205 formal close: confirm read-only runtime + human-approved promotion as the ADR'd default (resolved โ ADR-037, 2026-07-07) | Product / E | |
| Q2 | Soul/Skill storage split: what lives in PG vs R2 (size, versioning, eval-sandbox access) | Eng (T0.3) | Blocking โ gates T0.3 |
| Q3 | Does prompt_template_key's closed union stay, or do skills eventually subsume role templates? (keep union short-term; revisit after Phase 2) | Eng/Product | Non-blocking |
| Q4 | Client-visible feedback (R4.7) copy & placement โ must not invite clients to rate "the AI" in a way that undercuts the persona | Design | Non-blocking |
Review cadence: re-scope after Phase 1 ships; success metrics reviewed weekly against ยง6.