Skip to main content

ADR-033 β€” Professional Specialist Architecture (production-grade, eval-driven)

Status: Draft (pre-acceptance) Date: 2026-06-26 Epic: #3322 Authors: @stoneton Related ADRs: ADR-008 (KB K1/K2), ADR-020 (isolation), ADR-032 (retrieval engine), ADR-007 (expert access)

Thesis: a professional Specialist is not a prompt β€” it is a layered, measured, self-improving, human-in-the-loop pipeline. The 2026 industry consensus is to separate instruction / permission / retrieval / verification / output into independent layers (collapsing them into one prompt is the #1 anti-pattern), and to run quality as a live system with the same engineering rigor as the agent itself, not as a one-time prompt-tuning exercise.

Context​

h.work delivers a managed "AI draft β†’ Expert review β†’ send as Specialist" service. The professionalism of a Specialist reply is the product. Today that professionalism rests almost entirely on (a) one hand-written systemPrompt per Specialist over 8 shared role templates, and (b) a genuinely good retrieval chain. We have deep-dived the full chain (knowledge/persona, retrieval, tools, risk/HITL, learning, eval) against current code.

Strong foundation (keep / extend)​

  • Retrieval (L2): hybrid BM25 + pgvector + RRF + cohere/rerank-v3.5, citations, effective_date/audience/jurisdiction filters (rag/pipelines/retrieval/pipeline_wrapper.py; ADR-032, #3166). Offline RAG eval baseline exists (hybrid MRR 0.99 vs keyword 0.47, #3091). Per-retrieval telemetry kb_retrieval_event is written and joinable to Message/Correction by message_id (#3189).
  • HITL (L5): atomic F3 release (expert-queue.service.ts), always-HITL default (autoRespondThreshold=101).
  • Policy gate (L3): ActionPolicyService + ApprovalRequest sit between plan and tool execution β€” this is exactly the industry "ToolSafe" pattern, and h.work already has it.
  • Enterprise invariants: OrgΓ—Specialist isolation (ADR-020), 3-layer outbound protection, BullMQ, graceful shutdown, audit.

Gaps that cap professionalism (priority order)​

  1. No domain-accuracy measurement. agent/evals and rag/evals test plumbing / format / retrieval ranking β€” never "is the answer professionally correct". EvalResult has no expected_reply; the L2 pipeline eval mocks the LLM. We are flying blind.
  2. Expert-edit signal discarded. releaseDraft (expert-queue.service.ts:1374) never diffs draft.content vs finalText, never marks wasEdited. The richest, zero-extra-effort quality signal is thrown away β€” cascading into manual learning intake, uncalibrated confidence, and no quality KPI.
  3. No grounding/faithfulness verification (L4 absent). A draft is never checked against the retrieved KB before it reaches the Expert; KB hits are stored as metadata only.
  4. Tools not wired to the conversation path. Live /v1/chat runs Hermes terminal,delegation; business tools (KYC/orders/finance) are prompt-described but have no executor on the chat path. The Specialist talks about expert work but can't fetch real data.
  5. Confidence uncalibrated (model self-reported; the estimate_confidence calibrator is dead on the live path); correction_embeddings write-only (never retrieved for few-shot).

Decision​

Adopt a layered reference architecture and an explicit 11-stage processing pipeline for every client turn, and treat quality (eval + learning) as a first-class online subsystem. Build the missing layers in a measurement-first order.

Layered reference architecture​

L7 Eval & Observability  golden-set Β· LLM-as-judge Β· trace-eval Β· online faithfulness   (cross-cutting)
L6 Learning Flywheel     edit-diff capture β†’ corrections β†’ KB refine + few-shot + calibration
L5 Governance / HITL     risk grading Β· abstain / out-of-scope reroute Β· atomic Expert release
L4 Verification β˜…NEW     NLI grounding check Β· claim extract & verify Β· calibrated confidence
L3 Reasoning & Action    LLM reasoning + tool use via policy gate β†’ tool-gateway
L2 Retrieval (Agentic RAG) adaptive routing Β· hybrid+RRF+rerank Β· learned top-k Β· multi-pass
L1 Knowledge             K1 tenant / K2 global Β· structured domain rules/policy Β· golden answers
L0 Identity & Persona    persona Β· structured domain capability Β· layered prompt assembly
   cross-cutting: OrgΓ—Specialist isolation Β· reliability (CB/retry/DLQ/BullMQ) Β· cost & latency budgets Β· audit

Design rule: these layers must stay separable β€” never re-collapse instruction + permission + retrieval + verification + output into a single prompt. (The current prompts_loader persona-override heuristic, which can let a long persona silently replace the whole role template, violates this and must be fixed.)

The 11-stage turn pipeline (processing logic)​

Each stage is independently observable (one OTel span), evaluable (trace-eval target), and fail-safe (fail-open to "queue for Expert", never a 500 or a silent send).

#StageIndustry patternCurrent β†’ target
0Ingest / Dedupβ€”βœ… Redis NX + DB unique index
1Triage / Route2-stage LLM triage β†’ NO_ACTION / DRAFT_FOR_EXPERT; emit specialist_id❌ keyword regex β†’ LLM triage (start by wiring the already-built SlackSemanticMatcherService)
2Context Assembly (Agentic RAG)adaptive routing; agent judges context sufficiency, may re-retrieveβ—‘ good but static β†’ add adaptive routing + query rewrite + multi-pass
3Plan & Actplanner-executor; policy validation between plan and execute (ToolSafe)❌ tools not wired to chat β†’ bridge Hermes ↔ tool-gateway (policy gate already exists)
4Draft Generationstructured outputβœ… exists
5Verification β˜…RAG-triad as a gate: claim extract β†’ NLI entailed/neutral/contradicted β†’ aggregate; HALT-RAG calibrated NLI + abstention; online 200ms–5s❌ absent β†’ the single largest professionalism jump
6Confidence + Risk calibrationcalibration curve drives thresholds; low-confidence/high-risk/ungrounded β†’ abstainβ—‘ confidence uncalibrated; risk = fragile regex β†’ calibrate + LLM risk classifier
7HITL Routingabstention + structured escalationβ—‘ HITL solid but draft un-annotated β†’ hand Expert a verified, three-color draft; reroute out-of-scope to the right Specialist
8Expert ReviewHITLβœ… atomic F3 release
9Send as Specialistβ€”βœ… 3-layer protection
10Learn β˜…edit = preference signal; online logs feed golden-set❌ diff discarded β†’ capture in releaseDraft: unlocks the whole flywheel
11Eval / Observetrace-based eval + online eval feeding golden-set❌ no domain-accuracy eval β†’ build it

Adaptive routing (cost/quality): stage 1 classifies complexity β†’ trivial greetings: no-retrieval + small model; ordinary: hybrid+rerank+mid model; high-risk/complex (KYC, finance, compliance): multi-pass agentic RAG + tools + strong model (opus tier) + full verification. Keep cost low for simple queries; deploy the full arsenal only where it's needed.

Three highest-leverage layers (what makes it "professional" vs "looks professional")​

  1. L4 Verification β€” grounding as a gate, not a metric. In the serving path: extract atomic claims β†’ NLI-check each against retrieved KB / tool outputs β†’ on contradicted/neutral claims, lower confidence + annotate the offending sentence in the draft + abstain/escalate per severity. This frees the Expert from re-reading to "judging red flags" and makes professionalism provable, not asserted.
  2. L7 eval + L6 learning β€” the measure-and-recover loop. Capture the releaseDraft edit-diff (one change unlocks learning intake + confidence calibration + quality KPI); build a per-role domain golden-set with LLM-as-judge (50–100 anchor cases, β‰₯500 for trustworthy aggregates), run it on every PR; feed online failures back into the golden-set; calibrate confidence against actual edit ratios; wire correction_embeddings for few-shot.
  3. L3 action β€” let the Specialist actually do expert work. Bridge Hermes ↔ tool-gateway so KYC/order/finance tools fetch real data in a turn; keep and table-ize the existing ActionPolicyService/ApprovalRequest policy gate.

Maturity roadmap​

PhaseThemeKey actionsUnlocks
P0 ηœ‹εΎ—θ§ (2–3 wk)measure + recover signaledit-diff capture in releaseDraft; domain golden-set + LLM-as-judge in CI; OTel trace spansfirst time we know if it's professional
P1 ε­¦εΎ—δΌš (3–5 wk)close the flywheeleditβ†’correction auto-intake (feeds the live KB-refinement cron); correction_embeddings few-shot retrieval; confidence calibrationimproves with use
P2 说得准 (4–6 wk)L4 verificationNLI grounding + claim verification gate; three-color annotated draft for Experts; abstain / out-of-scope reroutefrom "looks professional" to "provably grounded"
P3 εšεΎ—εˆ° + adaptive (quarter)action + adaptive RAGHermes↔tool-gateway bridge + unified tool catalog; adaptive multi-pass agentic RAG; risk/difficulty model routing; structured K2 domain knowledge (decision rules, golden answers)data-backed expert-level action

First slice = P0 β‘  edit-diff capture + β‘‘ golden-set. No instrument, no improvement: the mechanism (RAG/HITL/KB flywheel) is already good; the two breaks are the output (no measurement) and the input (discarded edit signal). Fix those first.

Consequences​

  • Positive: professionalism becomes measurable and self-improving; Experts get triaged, annotated drafts (higher leverage); ungrounded claims are caught before a human; the Specialist can act on real data. Builds on existing strengths rather than replacing them.
  • Cost / latency: L4 online verification adds 200ms–5s per turn (use a lightweight NLI model online, reserve LLM-as-judge for high-risk; gate by adaptive routing). Golden-set eval adds CI time (opt-in workflow like rag-eval-baseline.yml).
  • Risk: L4 false-positives could over-flag; mitigate with calibration + always-HITL backstop (a wrong "ungrounded" flag only adds an Expert annotation, never blocks a real send since HITL is the gate anyway).
  • Non-goals: model fine-tuning/RLHF (the flywheel is retrieval/few-shot/KB based); per-Specialist separate Slack apps (that's ADR-030); replacing the human Expert (always-HITL stays).

References (industry)​

Last updated on by Degen