PRD: Omnichannel Specialist Continuity — Shared Memory / KB / Skills (Text × Tavus Video)

• Status: Reviewed — O1–O5 decided by Franky on 2026-07-14 (see §9 decision record); revised 2026-07-15 after a code walkthrough (highlights: long-term text-side memory specialist_memories already exists → D4 revised to extend that table; new R1.5 Active-Directives-to-video and R2.7 assembly ordering; multiple §5 baseline updates — every change carries a "review-corrected" annotation); Epic breakdown pending
• Authors: Franky (PM) + Claude
• Date: 2026-07-14 (decisions same day) · corrections 2026-07-15
• Language: this is the English mirror of omnichannel-specialist-continuity-prd.zh.md (Chinese is the working original; keep both in sync)
• Related: docs/specs/tavus-integration.md · ADR-008 (KB K1/K2) · ADR-006 (removal of the legacy memory module; note: specialist_memories postdates it, see §5) · #646 (Tavus account-level KB cross-tenant risk) · #1387 (one-shot persona provisioning) · #2058 (client on-demand video calls) · #2059 (WeeklyCallSchedulerService scheduled calls; adjacent epic in §11) · #1197/#3131 (specialist_memories, the existing cross-session memory layer) · #3692 (Active Directives) · #4399 (chat-path tool wiring, W3 prerequisite)

---

1. Problem statement

Product goal: whichever channel a client uses to talk to their Specialist (webchat, Slack, email, video call), they experience the same human expert — one who remembers prior conversations, knows the documents the client uploaded, and behaves the same way.

Gap between current state and that goal (2026-07 code investigation; details in the §5 baseline table):

1. The video AI does not read the KB. The text Specialist retrieves tenant KB per turn (K1/K2, api/src/kb/kb-retrieval.service.ts); the Tavus video AI's "knowledge" is only the conversational_context string injected per call (brief snapshot / generic onboarding script / 7-day conversation summary). Documents a client uploaded for their Specialist: in text he has "read" them, on video he has "never seen" them.
2. Cross-channel memory: the text side already has a shallow implementation; the video side has none (review-corrected 2026-07-15 — the original claim "the text side has none either" was wrong). Text has two layers: per-conversation agent_context (history_summary/facts/preferences, JSONB + Redis 1h; breaks on thread/channel switch) and the org × specialist specialist_memories (#1197/#3131: long-term cross-session memory, injected on all three chat paths — draft/stream/regenerate — with a 20-entry / 4,000-char budget and an Expert-workspace view/delete UI). But its extraction end is toy-grade — only 4 hardcoded English regex phrasings, triggered only on conversation resolve; actual accumulation ≈ 0. Video calls have no cross-call memory at all; every call is a one-shot snapshot.
3. Skills/tools are two unaligned stacks. The text draft path carries no business tools (tools exist only on the agentic/AgentRun path, bound per org via tool_permissions); the Tavus persona carries a fixed tool union identical for all personas (tavus_function_definitions + 3 hardcoded in-call tools), unrelated to the Specialist.
4. Deliberate architectural decoupling: the video AI is driven by Tavus's own LLM; the h.work Python agent is not in the loop (spec §6.2); persona prompts are baked offline and "no longer share templates with the Python agent" (#1390). The only shared element is the persona identity (name/role/capabilities derived from the same catalog row) — and that sharing drifts: the text side now reads the published Soul from platform-side config-assets (effective next turn after publish), while the Tavus persona is still baked offline from the catalog row; publish a Soul without re-provisioning and the two channels' personalities fork (see §5).
5. Active Directives reach the text channel only (compliance-grade break). #3692 is live: Expert-set situational directives (red-line-class content that outranks the KB) are injected into text prompts; the video path is entirely unaware. In a crisis ("a hack occurred — do not discuss security measures"), a client video call = red-line bypass.

(2) is the root-cause item: the goal is not "align video to text", nor to build a shared layer from scratch, but to upgrade the existing specialist_memories into a channel-agnostic shared layer (LLM-ify extraction, add the video writer, add governance fields), then plug every channel — including text itself — into it. (5)'s red-line-class consistency ships ahead of facts memory.

2. Goals (measurable)

#	Goal	Metric
G1	Video AI can recall tenant KB	For questions whose answers exist in the KB, video-call recall ≥ 80% of the text side's recall on the same questions (sampled eval)
G2	Cross-channel memory continuity	Facts stated by the client in channel A are referenceable in the next session on channel B; eval-set pass rate ≥ 90%
G3	Video memory write-back	Within ≤ 5 min of call end, the transcript summary is written to the shared memory layer (BullMQ job success ≥ 99%)
G4	Zero cross-tenant leakage	All KB/memory read-write paths carry org × specialist filters; 100% of new paths have spec assertions
G5	Perceived continuity	"Feels like the same expert" score in client satisfaction surveys (baseline established by the first survey)

3. Non-goals

• No use of Tavus platform documents/knowledge_base APIs (#646 account-level sharing → cross-tenant leakage; the existing three-layer hard ban stays).
• No vector/embedding retrieval upgrade (current KB scoring is lexical; retrieval-quality upgrades are a separate initiative).
• No Expert-facing (human) memory tooling (improving what the Expert sees is a separate initiative).
• No change to the HITL default (autoRespondThreshold=101) or the draft review flow.
• v1 does not add tool injection to the text draft path (see W3 — the text side's "tool story" must be settled first).

4. User stories

• As a client employee, I told my Specialist on Slack last week that our settlement cycle moved to monthly; on this week's video call he proactively discusses in monthly terms instead of asking again.
• As a client employee, I ask on video about "company X's terms in the supplier list you compiled for us", and the Specialist cites that KB document on the spot.
• As an Expert, when reviewing text drafts I can see the client's shared memory (facts), so I don't release replies that contradict established facts.
• As an AM, I can view/correct the accumulated facts of an (org × specialist); wrong memories can be deleted and take effect across all channels immediately.
• As a SuperAdmin, I am confident no memory/KB path can carry client A's information to client B.

5. Current-state baseline (code evidence)

Dimension	Text Specialist (/v1/chat)	Tavus video AI	Aligned?
KB	Per-turn HTTP to /kb/retrieval, filtered specialist_id = $1 AND org_id IN (org, HP_GLOBAL) (K1/K2); injected into system prompt	No KB retrieval; Tavus documents API hard-banned at three layers (#646); TavusContextBuilderService.buildFromClientKb is registered dead code with no callers	❌
Memory	last-12-turn history + per-conversation agent_context (Redis 1h TTL) + specialist_memories (org × specialist long-term memory, already injected into prompts; but extraction is 4 English regexes on resolve — accumulation ≈ 0)	No cross-call memory; per-call snapshots (reporting = brief JSON, on-demand = 7-day conversation summary, onboarding = generic 9-bucket script)	⚠️ Text has a shallow implementation; video has none
Situational directives (Active Directives)	Injected into prompts, outranking KB (#3692)	Unaware	❌ (compliance-grade)
Skills/tools	Draft path has no business tools (#4399 tracks wiring toolsManifest); file-generation toolsets are enabled (#3944, HW_ATTACHMENT outbound); agentic path binds per org (tool_permissions → AgentRun.toolsManifest)	Fixed tool union identical across personas; in-call tools read the DB directly (get_client_history/get_current_goals/escalate_to_chat)	❌
System prompt / identity	Specialist.system_prompt (fallback) + templates + published Soul from config-assets (getPublishedSoulMd, single renderer, effective next turn; SOUL.md files retired to fallback), layered assembly on the Python side	buildSystemPrompt(catalogRow) derived from the same catalog row, baked offline	⚠️ Same identity source but drifts: Soul publish affects text only; the persona updates only on re-provision
Runtime	h.work Python agent (Hermes)	Tavus's own LLM; h.work agent not in the loop	❌

Key files: api/src/kb/kb-retrieval.service.ts · agent/kb_client.py · agent/prompts_loader.py · api/src/tavus/tavus.client.ts · api/src/tavus/video-call-tools.service.ts (corrected: originally miswritten as the video-calls/ directory) · api/src/tavus/video-call-tool-definitions.ts · api/src/meetings/conversation-starter.service.ts · api/src/specialists/specialist-memory.service.ts · api/src/specialists/specialist-memory-extraction.job.ts · api/src/conversations/agent-context-cache.service.ts · api/scripts/provision-tavus-personas.ts. Re-verify line numbers against HEAD at implementation time.

6. Approach: three workstreams

W1 — Video access to tenant KB (P0, fastest win)

Mechanism: a function-call tool, not pre-injection. Add a search_knowledge_base tool to the Tavus persona; the Tavus LLM issues a function-call during the conversation → existing …/tavus/webhooks/function-call callback → a new handler calls KbRetrievalService.retrieve(orgId, specialistId, query, topK) (K1/K2 filtering reused as-is) → result returned to Tavus.

• Why a tool: fully bypasses Tavus account-level KB (the #646 red line is never touched); the in-call tool path is already proven by 3 existing tools; retrieval is real-time and unconstrained by context size.
• Auxiliary (P1): at call creation, splice top-N KB document titles+snippets into conversational_context as a zero-latency fallback (mind the per-call context size budget).
• Requirements:
  • R1.1 New search_knowledge_base tool definition (into the tavus_function_definitions library) + webhook handler; org/specialist resolved from our own video_call/meeting rows — never trust tenant parameters echoed by Tavus.
  • R1.2 Handler reuses KbRetrievalService (including kb_retrieval_enabled flag semantics); add an independent switch for the video path, video_kb_enabled (org × OSA granularity).
  • R1.3 Retrieval calls are audited (orgId, specialistId, query, docIds, meetingId/videoCallId).
  • R1.4 Persona toolset updates go through the re-provision/PATCH script (personas are baked offline).
  • R1.5 (added in the 2026-07-15 review, compliance-grade) Active Directives reach video: at call creation, inject the (org × specialist)'s active directives at the top of conversational_context (reusing conversation-starter's existing top-directive-block precedent), framed as "highest priority, overrides all other context, never to be quoted to the client"; cross-channel consistency of red-line-class content ships ahead of facts memory. Optional: an in-call refresh tool (directive changes during long calls) goes to v1.1.

W2 — Cross-channel shared memory layer (P0, architectural core)

Storage: extend the existing specialist_memories by default (#1197/#3131; D4 revised in the 2026-07-15 review — the original text misjudged "no long-term memory on the text side"; that table exists, is wired into chat injection and the Expert governance UI): add source_channel, source_user_id, confidence, status (active/retracted) fields and rolling_summary storage via expand-contract; create a new specialist_client_memory table only if engineering shows extension is infeasible. Granularity: org × specialist (team-shared) (per D1, naturally matching the existing table; known privacy risk below). Multi-tenancy matrix row: Org × Specialist (specialist_id NOT NULL, RLS dual GUC + service-layer dual filter).

⚠️ D1 known privacy trade-off (deliberately accepted by product; implementers and ops must know): memory is team-shared per org × specialist — within the same client org, what employee A tells the Specialist (once distilled into facts/summaries) may be referenced in employee B's sessions with the same Specialist on any channel. v1 accepts this to deliver the "one human expert" continuity; there is no per-employee private partition yet. Mitigations: facts carry source_user_id for provenance (R2.3 correction/deletion locates by it); for sensitive orgs the AM evaluates before enabling shared_memory_enabled. If "private per-user memos" are needed later, v2 adds a visibility: team|private field (expand-compatible; no table rebuild).

• Structure (draft): facts[] (with source channel, source user source_user_id, timestamp, confidence, status active/retracted), preferences[], rolling_summary (per-channel + aggregate), updated_at. Schema work follows expand-contract.
• Writers:
  • R2.1 Text: merge the two existing write lines — per-conversation agent_context distillation and the specialist-memory-extraction job (today: 4 English regex phrasings on resolve) — upgraded to LLM-based extraction (reuse LlmService, keep the key whitelist and the org-level-facts-only guard) writing to the shared layer; agent_context remains as in-session working memory, dual-write during transition.
  • R2.2 Video: call-end callback triggers a BullMQ job (stable jobId, idempotent) → transcript summarization/fact extraction → write to the shared layer, source video. ⚠️ Shared-pipeline requirement (§11): post-call transcript processing must be one pipeline with multiple outputs — a single LLM extraction simultaneously produces ① shared-memory facts/summary (this PRD) and ② action items → client_briefs.data.follow_ups (the adjacent epic's post-call auto-follow-up). The two initiatives must not each build a transcript pipeline.
  • R2.3 Memory governance: facts are viewable/correctable/deletable by AM/Expert (retract takes effect across all channels immediately); write governance per D2 (auto-write + low-confidence flag; strong facts go through the Expert queue). Correction/deletion locate by source_user_id + source channel. Note: the Expert workspace already has a memory view/delete UI (workspace.service) — add correction/retract and confidence display on top of it; do not build a new page.
• Readers:
  • R2.4 Text /v1/chat: injection already exists (appendSpecialistMemoryContext, on draft/stream/regenerate) — the work here is budget review (currently 20 entries / 4,000 chars), format alignment with the new fields (confidence/status), and clarifying the layering semantics vs agent_context.
  • R2.5 Video: at call creation, splice the shared-memory summary into conversational_context; add a recall_memory in-call tool for deep lookups. ⚠️ Shared injection point (§11): call-creation context assembly must converge on one composable builder (memory summary + daily digest/agenda + KB snapshot as separate segments); the adjacent epic's "digest injection + proactive opening" (reworking buildOpeningPhrase's cold start) reuses the same builder — no second assembly stack. The opening line changes from "how can I help?" to proactively reporting on the injected content; both initiatives benefit.
  • R2.6 Slack/email paths reuse the text read/write ends (they all converge on conversations → /v1/chat and benefit automatically).
  • R2.7 (added in the 2026-07-15 review) Assembly ordering: shared memory ranks below red lines / Active Directives / golden answers and above general conversation history in prompt assembly — aligned with prd-specialist-value-output R5.1's precedence chain; memory content is injected wrapped as untrusted data (per the #3190/#3406 precedent) so "memory" cannot become an instruction-injection surface.
• Consistency: memory writes are best-effort (never block a business reply); readers tolerate minute-level lag.

W3 — Skill/tool consistency (P2, v2)

• Goal: derive the Tavus persona toolset per Specialist (same source as the text side), replacing the fixed union; Specialist tool changes → persona PATCH sync.
• Prerequisite: the text side's own "tool story" must land first — that convergence is already ticketed: #4399 (chat-path toolsManifest + tool callbacks); today the draft path has no business tools and agentic binding is per-org, not per-specialist. Aligning video before #4399 lands is castle-in-the-air work, so this is v2; produce a design doc first (possibly an ADR).
• Added scope: the structured red-line section of config-assets skills (consumed by the Phase-3 assembler) is red-line-class content and belongs in the video alignment scope at the same priority as R1.5 (directives) — red lines effective in text but bypassed on video is the same class of compliance break.

7. Security & isolation (hard constraints)

1. The Tavus documents/knowledge_base API stays hard-banned at three layers; every knowledge injection in this PRD goes through our own retrieval + callbacks.
2. The memory store (extension of the existing specialist_memories, or the fallback new table) is "Customer PII" isolation class (register/review in the ADR-020 matrix): RLS dual GUC + service-layer org×specialist dual filter; new/changed entities must carry the matrix docstring (repo rule).
3. Webhook handlers always resolve tenant context from our own video_calls/meetings rows; Tavus-echoed parameters are used for validation only.
4. Memory content is client data: export/deletion joins the (future) data-deletion flow; facts are traceable via source channel + source_user_id.
   • D1 privacy annotation: memory is org × specialist team-shared (see the risk box in §6 W2). This is a deliberate v1 trade-off, not an implementation oversight; any client-facing material describing the memory feature must state truthfully that "your team's communication with this Specialist is remembered collectively".
5. Tavus webhook auth keeps the shared-token mechanism (do NOT revert to HMAC — Tavus does not sign; repo security rule #6).

8. Dependencies & phases

Phase	Content	Depends on
v1 (P0)	W1 (search_knowledge_base tool + R1.5 directives-to-video) + W2 skeleton (memory storage extension + video write-back + text read alignment)	O1/O2 decisions (decided, see §9 D1/D2); ADR-020 matrix registration
v1.1 (P1)	Pre-call KB/memory snapshot injection; memory governance UI (AM/Expert view & correct); Expert review page shows facts	v1
v2 (P2)	W3 tool consistency (ADR first); memory quality evals and recall-strategy tuning	Text-side tool story converged (#4399)

9. Decision record (2026-07-14, decided by Franky; D4 revised 2026-07-15)

Formerly open questions O1–O5, now decided (D1–D5). Implementation and breakdown follow these:

• D1 Memory identity key → org × specialist (team-shared). v1 adopts team-shared granularity to deliver the "one human expert" continuity. Known and accepted privacy issue: information spreads between employees of the same org via shared memory (what A said may be referenced in B's sessions) — see the §6 W2 risk box and §7.4; facts record source_user_id for provenance and correction; if per-user private memos are needed later, v2 extends with a visibility field.
• D2 Memory write review → v1 auto-write + governance. Auto-distillation + low-confidence flags + AM/Expert view/correct/delete (retract effective across channels immediately); strong facts (amounts, dates, commitments) take effect only after confirmation through the Expert review queue.
• D3 Video KB recall form → v1 tool-only, v1.1 adds pre-injection. v1 ships only the search_knowledge_base in-call tool; v1.1 adds pre-call top-N snapshot injection into conversational_context as a zero-latency fallback.
• D4 W2 data-model home → extend the existing specialist_memories by default (revised in the 2026-07-15 review). The original recommendation to create specialist_client_memory was based on the misjudgment that "the text side has no long-term memory"; the review found specialist_memories (#1197/#3131) already exists, wired into chat injection (appendSpecialistMemoryContext) and the Expert governance UI, with the same org × specialist granularity. Revised: extend that table via expand-contract by default (source_channel/source_user_id/confidence/status + rolling_summary); a new table is the fallback, only if engineering shows extension infeasible. The original rationale that "mixing into client_briefs would rot its semantics" still stands. Product semantics (granularity, fields, governance) unchanged.
• D5 Retention & quotas → conservative v1 defaults, aligned with compliance. Initial suggestion: facts ≤ 200 per key (evict by confidence + age beyond the cap), rolling_summary ≤ 4KB, retention initially matching conversation-data cycles; confirm with compliance before launch, parameters as config not hardcode. Align with existing mechanisms: specialist_memories already has expires_at/enabled and a delete API — expiry and disable use the existing fields, no parallel switches; injection budget starts from the current 20 entries / 4,000 chars and is tuned by evals.

10. Success measurement & acceptance

• Eval method for G1–G4: build seed datasets of (org, specialist, KB docs, existing facts); run the same Q&A on text and video; isolation assertions go into unit tests (video handler org filtering, cross-org negative cases).
• Launch switches: video_kb_enabled, shared_memory_enabled, gated per org × OSA, default off; enable first in the internal demo org.

11. Adjacent work integration: daily digest + configurable scheduled calls (Terence feedback, 2026-07)

Terence's product feedback (regular reporting calls + "the AI enters the call with no context and waits to be asked") partially converges with this PRD. Verified: scheduled-call infrastructure exists (WeeklyCallSchedulerService, #2059: BullMQ fan-out per primary Specialist assignment, creates scheduled calls, emails links), but the cadence is hardcoded (Mondays 10am), there is no dedup, no Expert one-off scheduling UI/API, and scheduled calls have no agenda field; and all calls (including scheduled weeklies) are cold-start rooms — the persona only does the buildOpeningPhrase-style "Hi, how can I help?".

Initiative boundary: the digest page (/dashboard/daily + client_daily_digests) and Expert-configurable scheduling (cadence/agenda templates/dedup) belong to an independent adjacent epic (filed under the Tavus/video-calls area), not merged into this PRD. But the two initiatives must share the following components, with this division:

Shared component	This PRD owns	Adjacent epic owns	Constraint
Call-creation context assembly	Composable builder skeleton + memory-summary segment + KB-snapshot segment (R2.5 / R1 series)	Daily-digest segment + scheduled-agenda segment	One builder, pluggable segments; two context-assembly stacks are forbidden
Proactive opening (buildOpeningPhrase cold-start rework)	Opening references shared memory ("last time we discussed…")	Opening reports the daily digest/agenda ("since yesterday: X, Y, Z…")	Same opening mechanism; content generated from injected segments
Post-call transcript processing	facts/summary → shared memory layer (R2.2)	Action items → client_briefs.data.follow_ups	One BullMQ pipeline, one LLM extraction, multiple outputs
Daily digest data sources	The shared memory layer can be one digest input (fact changes, open follow-ups)	client_daily_digests generation job	The digest job reads the memory layer through the official service API, never raw tables

Sequencing suggestion (merging both perspectives): digest job + storage → digest page + call button → shared context builder (this PRD's W1/W2 read side + digest injection land together) → proactive opening → Expert-configurable cadence → shared transcript pipeline (this PRD's R2.2 + follow-ups land together). People: Alex K (scheduler), Alex D (frontend), Emanuele (infra if the digest job needs a new worker).

Impact on this PRD: W1/W2 scope and priority unchanged; R2.2 and R2.5 gain the "shared pipeline / shared injection point" constraints (see those items). When the epics are created, sub-issues on both sides must cross-reference; each shared component gets one clearly-owned issue to avoid parallel wheel-building.