P3.4 — Full Assembly Trace Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (- [ ]) syntax.

Goal: Surface the assembler's layer_manifest (per-layer id + charCount + sourceRefs) into the persisted draft metadata.assembly trace on the assembler path, and add the soul layer's published versionId as a sourceRef — so a draft records exactly which layers (KB chunk ids, skill asset:version, soul version) composed its prompt, for auditing and the future hit-rate/quality analytics.

Architecture: P3.4 extends the P0.6 metadata.assembly trace (six scalar fields today) with a layers array derived from the assembler's layer_manifest (which already carries charCount + sourceRefs for kb/skill — P3.1/P3.2). NestJS owns the manifest (it produced it), so the trace is stamped NestJS-side from assembledPrompt.layer_manifest directly — NOT round-tripped through the agent (the agent lossily reduces the manifest to layer-ids and NestJS doesn't read even that back). messages.metadata is jsonb → extending metadata.assembly is free, no migration (D-4: extend field, not a separate table). The trace is write-only today (no SQL/code consumer yet — the persona-override hit-rate query is documented intent, unimplemented), so additive fields break nothing.

Tech Stack: NestJS 11 (AssemblerService, ConversationsService trace stamp, ConfigAssetsService soul reader), Jest.

Scope note: Long-lived branch feat/specialist-value-output-p2plus. Depends on P3.1 (manifest) + P3.2 (skill sourceRefs) + P3.3 (correction/budget). Commit per sub-task.

Verified reality shaping this plan (Explore):

• Manifest scope asymmetry: on regenerateDraft, assembledPrompt is at method-body scope and reachable at the metadata-stamp site. On the main dispatch, assembledPrompt is trapped in the inner agent-dispatch try (declared ~:3262, dies at the } catch ~:3375) while the persist site is :3410-3441 — the manifest must be folded into the method-scope assemblyTrace (:3359, where assembledPrompt is still live) which already reaches the persist site. Per-site handling, not one uniform edit.
• soul version_id is must-BUILD: getPublishedSoulMd returns a bare string, discarding the publishedVersionId it already fetched. No {content, versionId} variant (unlike getPublishedSkills). Task 2 adds one.
• per-layer tokenBudget is a dead field (always null — the budget is a single total-body budget, not per-layer). P3.4 does NOT surface tokenBudget (no real value); it surfaces charCount (meaningful) + sourceRefs.
• trace is write-only today — additive layers field has zero existing consumers to break.

Boundary (P3.4 does NOT do):

• Does NOT surface per-layer tokenBudget (dead field).
• Does NOT change the flag-OFF path (no assembler → no manifest → the P0.6 six-field trace is unchanged when OFF; degraded/stub drafts still carry no assembly key).
• Does NOT round-trip the manifest through the agent (stamped NestJS-side).
• Does NOT build a trace consumer / analytics query (write-only; that's a later analytics task).
• Does NOT add a migration (metadata is jsonb).

Pre-existing baseline: verify against branch base 490517fb. No agent-side change → agent evals unaffected, but re-run the assembler/wiring specs.

---

File Structure

Modify:

• api/src/conversations/assembler.service.ts — add soulSourceRefs?: string[] to AssembleInput and attach to the soul spec entry (mirrors skillSourceRefs/kbSourceRefs), so soul's version id rides the manifest. Add a small pure helper manifestToTrace(layer_manifest): AssemblyLayerTrace[] (exported) mapping each Layer to {id, charCount, sourceRefs?} for the metadata trace (drops priority/neverTruncate/tokenBudget — the trace needs id + size + refs).
• api/src/config-assets/config-assets.service.ts — add getPublishedSoul({orgId, specialistId}): Promise<{ soulMd: string; versionId: string; versionNo: number } | null> (sibling of getPublishedSoulMd, surfaces the version id it already fetches; same fail-open). Keep getPublishedSoulMd (existing callers unchanged) — implement it as a thin wrapper over the new method OR leave both; do not break the string-returning contract.
• api/src/conversations/conversations.service.ts:
  • resolveSpecialistPersona — return soulVersionId: string | null alongside systemPrompt/promptTemplateKey (from the new getPublishedSoul; null when persona comes from the systemPrompt column, not a published soul).
  • Main dispatch: thread soulSourceRefs: soulVersionId ? ["soul:" + soulVersionId] : undefined into assemble({...}); fold manifestToTrace(assembledPrompt.layer_manifest) into assemblyTrace at ~:3359 (where assembledPrompt is live).
  • Regenerate: same soul thread; fold the manifest into the regenerate trace object (assembledPrompt is in scope there).
• api/src/conversations/CLAUDE.md — document the extended trace.

---

Task 1: manifest → metadata.assembly.layers (both sites, no soul version yet)

Files:

• Modify: api/src/conversations/assembler.service.ts (add manifestToTrace)

• Modify: api/src/conversations/conversations.service.ts (fold into both trace stamps)

• Test: api/src/conversations/assembler.service.spec.ts (manifestToTrace unit) + prompt-assembler-wiring.spec.ts (trace has layers)

• Step 1: Read assembler.service.ts:35-42 (Layer shape) + :133-140 (manifest builder). conversations.service.ts:2681 (let assemblyTrace), :3358-3367 (main trace construct), :3419-3441 (main persist stamp), the regenerate trace construct + stamp (grep assembly: — the second occurrence, ~:6678), and confirm assembledPrompt scope at each (main: inner try; regenerate: method body).

• Step 2: Write the failing test (assembler unit for the pure helper):

import { AssemblerService, manifestToTrace } from "./assembler.service";

test("manifestToTrace maps layers to {id, charCount, sourceRefs?}", () => {
  const { layer_manifest } = svc.assemble({
    ...baseInput,
    kb: "kb body", kbSourceRefs: ["d1:c1"],
    skill: "skill body", skillSourceRefs: ["sk-1:2"],
  });
  const trace = manifestToTrace(layer_manifest);
  const kb = trace.find((l) => l.id === "kb")!;
  expect(kb.charCount).toBeGreaterThan(0);
  expect(kb.sourceRefs).toEqual(["d1:c1"]);
  const skill = trace.find((l) => l.id === "skill")!;
  expect(skill.sourceRefs).toEqual(["sk-1:2"]);
  // layers without refs carry no sourceRefs key
  const soul = trace.find((l) => l.id === "soul")!;
  expect(soul.sourceRefs).toBeUndefined();
  expect(soul.charCount).toBeGreaterThan(0);
  // priority / neverTruncate / tokenBudget are NOT in the trace
  expect((kb as any).tokenBudget).toBeUndefined();
  expect((kb as any).priority).toBeUndefined();
});

And a wiring assertion (flag ON → trace carries layers):

it("flag ON: metadata.assembly carries the layer trace (charCount + sourceRefs)", async () => {
  featureFlags.resolve.mockImplementation(flagResolver(true));
  await driveSendMessage();
  // find the persisted agent draft's metadata.assembly (however the harness captures the saved draft)
  const saved = msgRepo.save.mock.calls.map((c) => c[0]).find((m) => m.role === "agent" && m.metadata?.assembly);
  expect(saved.metadata.assembly.layers).toBeDefined();
  const kbLayer = saved.metadata.assembly.layers.find((l: any) => l.id === "kb");
  expect(kbLayer.charCount).toBeGreaterThan(0);
});

Adapt the "capture the saved draft" line to how the wiring spec already inspects saved messages — read the spec; msgRepo.save is mocked, so its call args hold the persisted metadata. If the existing tests assert on a different capture, mirror it.

Run → FAIL (manifestToTrace absent; assembly.layers not stamped).

• Step 3: Implement manifestToTrace in assembler.service.ts:

export interface AssemblyLayerTrace {
  id: LayerId;
  charCount: number;
  sourceRefs?: string[];
}

/**
 * Reduce a layer_manifest to the persisted trace shape (P3.4): id + charCount
 * + sourceRefs only. Drops priority/neverTruncate/tokenBudget — the persisted
 * `metadata.assembly.layers` records WHAT composed the prompt and how big each
 * layer was, not the (dead) per-layer budget mechanics. Pure.
 */
export function manifestToTrace(manifest: Layer[]): AssemblyLayerTrace[] {
  return manifest.map((l) => ({
    id: l.id,
    charCount: l.charCount,
    ...(l.sourceRefs ? { sourceRefs: l.sourceRefs } : {}),
  }));
}

• Step 4: Fold into the trace at both sites.
  • Main dispatch (~:3359, inside the if (agentResp.promptVersion) block where assembledPrompt is still live): after building the six scalar fields, add:

    ...(assembledPrompt ? { layers: manifestToTrace(assembledPrompt.layer_manifest) } : {}),

• Regenerate: same additive spread in the regenerate trace object (assembledPrompt is in method scope there).

Import manifestToTrace from ./assembler.service. The ...(assembledPrompt ? ...) guard means OFF path (no assembledPrompt) → no layers key → byte-identical P0.6 trace. Degraded drafts (no promptVersion) already carry no assembly key at all.

• Step 5: Run tests — new assembler unit + wiring trace test pass; all existing assembler/wiring/conversations tests green; flag-OFF trace unchanged. npx tsc --noEmit clean.

• Step 6: Commit

git add api/src/conversations/assembler.service.ts api/src/conversations/conversations.service.ts api/src/conversations/assembler.service.spec.ts api/src/conversations/prompt-assembler-wiring.spec.ts
git commit -m "feat(conversations): surface layer_manifest (charCount + sourceRefs) into metadata.assembly.layers (P3.4)"

---

Task 2: soul version_id as a soul-layer sourceRef

Files:

• Modify: api/src/config-assets/config-assets.service.ts (add getPublishedSoul)

• Modify: api/src/conversations/assembler.service.ts (soulSourceRefs input)

• Modify: api/src/conversations/conversations.service.ts (resolveSpecialistPersona returns soulVersionId; thread to both assemble sites)

• Test: config-assets spec (getPublishedSoul) + assembler spec (soulSourceRefs) + wiring (soul ref in trace)

• Step 1: Read getPublishedSoulMd (config-assets.service.ts:420-452) — note it fetches asset.publishedVersionId (:436) + version (:437) then discards the id. resolveSpecialistPersona (conversations.service.ts:591-622) returns {systemPrompt, promptTemplateKey}. getPublishedSkills (:477+) is the precedent for returning versionId/versionNo.

• Step 2: Write the failing test (config-assets):

describe("getPublishedSoul", () => {
  it("returns {soulMd, versionId, versionNo} for a published soul", async () => {
    // seed a published soul via the harness (mirror the getPublishedSoulMd test setup)
    const r = await service.getPublishedSoul({ orgId, specialistId });
    expect(r).toMatchObject({ versionId: expect.any(String), versionNo: expect.any(Number) });
    expect(typeof r!.soulMd).toBe("string");
  });
  it("returns null (never throws) when no published soul / missing id / parse fail / repo error", async () => {
    // mirror getPublishedSoulMd's fail-open cases
  });
});

Run → FAIL (method absent).

• Step 3: Implement getPublishedSoul (surface the id getPublishedSoulMd already has):

async getPublishedSoul(params: {
  orgId: string;
  specialistId: string;
}): Promise<{ soulMd: string; versionId: string; versionNo: number } | null> {
  if (!params.orgId || !params.specialistId) return null;
  try {
    const asset = await this.assetRepo.findOne({
      where: { assetType: "soul", scope: "org_instance", orgId: params.orgId,
               specialistId: params.specialistId, slug: SOUL_ASSET_SLUG, archivedAt: IsNull() },
    });
    if (!asset?.publishedVersionId) return null;
    const version = await this.versionRepo.findOneBy({ id: asset.publishedVersionId });
    if (!version) return null;
    const parsed = SoulContent.safeParse(version.content);
    if (!parsed.success) return null;
    return { soulMd: renderSoulMd(parsed.data), versionId: version.id, versionNo: version.versionNo };
  } catch (err) {
    this.logger.warn(`published-soul lookup failed for org=${params.orgId} specialist=${params.specialistId}: ${err instanceof Error ? err.message : String(err)}`);
    return null;
  }
}

Optionally refactor getPublishedSoulMd to return (await this.getPublishedSoul(params))?.soulMd ?? null (DRY — keeps the string contract for existing callers). Confirm the existing getPublishedSoulMd tests still pass after the refactor.

• Step 4: Thread the version id. In resolveSpecialistPersona, add soulVersionId: string | null to the return type; call getPublishedSoul instead of getPublishedSoulMd, set systemPrompt from .soulMd and soulVersionId from .versionId (null when falling back to the column). At both assemble sites, pass:

      soulSourceRefs: soulVersionId ? [`soul:${soulVersionId}`] : undefined,

(capture soulVersionId from the persona/regenPersona result).

• Step 5: Add soulSourceRefs to the assembler — in AssembleInput (after soul?): soulSourceRefs?: string[];; in the spec array soul entry: { id: "soul", text: input.soul, neverTruncate: true, sourceRefs: input.soulSourceRefs }. Assembler unit test: soulSourceRefs: ["soul:v-9"] → soul layer manifest has sourceRefs: ["soul:v-9"].

• Step 6: Run tests — config-assets getPublishedSoul + getPublishedSoulMd (unchanged contract) green; assembler soulSourceRefs green; wiring trace shows soul layer with sourceRefs when a published soul is used; full conversations + config-assets suites green. npx tsc --noEmit clean.

• Step 7: Commit

git add api/src/config-assets/config-assets.service.ts api/src/config-assets/*.spec.ts api/src/conversations/assembler.service.ts api/src/conversations/conversations.service.ts api/src/conversations/*.spec.ts
git commit -m "feat(config-assets): getPublishedSoul surfaces version id → soul-layer sourceRef in assembly trace (P3.4)"

---

Task 3: docs + full verification

Files:

• Modify: api/src/conversations/CLAUDE.md, api/src/config-assets/CLAUDE.md

• Step 1: Docs

  • conversations/CLAUDE.md — extend the P0.6 assembly-trace bullet: on the assembler path metadata.assembly additionally carries layers: [{id, charCount, sourceRefs?}] derived NestJS-side from assembledPrompt.layer_manifest (via manifestToTrace, NOT round-tripped through the agent). sourceRefs: kb=document_id:chunk_id, skill=assetId:versionNo, soul=soul:versionId. tokenBudget deliberately NOT surfaced (dead per-layer field; budget is single total-body). Flag OFF → no layers key (byte-identical P0.6 trace). Write-only today (no consumer yet).
  • config-assets/CLAUDE.md — add getPublishedSoul ({soulMd, versionId, versionNo}) next to getPublishedSoulMd (which is now a thin .soulMd wrapper if refactored); same dual-scope + fail-open contract.

• Step 2: Self-check no machine paths — grep -rnE '/Users/|/home/|/private/tmp' api/src/conversations/CLAUDE.md api/src/config-assets/CLAUDE.md; echo exit=$? → 1.

• Step 3: Full verification

  • cd api && npx tsc --noEmit && npm run lint → clean (hard lint budget).
  • npx jest src/conversations/ src/config-assets/ src/feature-flags/ --runInBand → all pass; flag-OFF trace unchanged.
  • cd agent && <venv>/bin/python -m pytest evals/test_prompt_echo_guard.py evals/test_chat_helpers.py evals/test_prompt_assembly_meta.py -q → pass (P3.4 is NestJS-side only — no agent change — but confirm nothing shifted).

• Step 4: Commit docs

git add api/src/conversations/CLAUDE.md api/src/config-assets/CLAUDE.md
git commit -m "docs(P3.4): metadata.assembly.layers trace + getPublishedSoul version-id reader"

• Step 5: No further commit — proceed to Phase-3-task-level review. Dispatch code-reviewer: verify flag-OFF trace byte-equality (no layers key when OFF), the trace is stamped from NestJS-side manifest (not agent round-trip), getPublishedSoul/getPublishedSoulMd contract preserved (existing callers unaffected), dual-scope + fail-open on the new reader, and the main-dispatch scope-fold correctness (assembledPrompt live where the trace is built).

---

Self-Review Checklist

Spec coverage (design §6.5 P3.4): layer_manifest sourceRefs (kb chunk ids ✅ P3.2, skill ids+versions ✅ P3.2, soul version_id ✅ Task 2) + per-layer charCount → metadata.assembly.layers ✅ (Task 1). D-4 extend-field-not-table ✅ (jsonb, no migration). Per-layer budget deliberately omitted (dead field — documented).

Type consistency: manifestToTrace(Layer[]) → AssemblyLayerTrace[] (Task 1) consumed at both trace stamps. getPublishedSoul return {soulMd, versionId, versionNo} (Task 2) → resolveSpecialistPersona soulVersionId → soulSourceRefs: ["soul:"+id] → AssembleInput.soulSourceRefs → soul layer manifest sourceRefs → back into layers trace via manifestToTrace. Full loop closes.

Safety net: flag OFF → no assembledPrompt → ...(assembledPrompt ? {layers} : {}) yields no layers key → P0.6 trace byte-identical. getPublishedSoul fail-open (null) mirrors getPublishedSoulMd; soulVersionId null → no soul sourceRef (graceful). Degraded drafts carry no assembly key (unchanged).

Boundary: no tokenBudget surfacing; no OFF-path change; no agent round-trip; no consumer/analytics; no migration.