ADR-032: KB Retrieval — Quality Measurement, Event Collection & Engine Consolidation
Date: 2026-06-25 Status: Accepted (implemented across #3091, #3189, #3166) Issues: #3166 (consolidation, P1) Related PRs: #3091 (eval harness), #3189 (event collection), #3271 (consolidation) Authors: Alex (with Claude)
Numbering note: 024–031 are taken; this lands at 032, the next free number.
Context
The platform had no way to measure KB retrieval quality and, on inspection, was running two retrieval engines in parallel on every Specialist draft, with the weaker one privileged in the prompt. This ADR records the decisions made resolving that, in three stages. Full design detail lives in docs/superpowers/specs/2026-06-23-rag-retrieval-eval-harness-design.md, 2026-06-23-kb-retrieval-events-design.md, and 2026-06-25-kb-consolidate-hybrid-retrieval-design.md.
The two engines
| Engine | Path | Mechanism |
|---|---|---|
| Keyword (B) | KbRetrievalService → GET /kb/retrieval | metadata-only scoring (title×4 / desc×2 / filename×3); no chunk body, no embeddings, no rerank |
| Hybrid (A) | HaystackRetrievalService → rag service | BM25 + vector + RRF + cohere/rerank-v3.5 |
Decision 1 — Measure retrieval quality with an offline harness (#3091)
Build a reproducible harness (rag/evals/retrieval/) that scores each engine in isolation on a synthetic query set: Hit@k (retriever ceiling; ≡ Recall@k under single-gold) and MRR (reranker scorecard). Ground truth = LLM-generated questions from real chunks (single gold doc). NDCG/Precision deferred until graded relevance exists (needs an LLM-judge). Runs in CI (rag-eval-baseline workflow) against a pgvector service container.
Baseline (rag/evals/retrieval/BASELINE.md): hybrid MRR 0.988 / Hit@1 0.977 vs keyword MRR 0.474 / Hit@1 0.349. Keyword's misses cluster on docs whose distinctive facts live in the body — invisible to metadata-only scoring.
Decision 2 — Collect per-retrieval events in first-party Postgres (#3189)
Persist a kb_retrieval_event row per drafted reply (query, engine, per-doc {doc_id, chunk_id, rank, score}, correlation keys message_id/conversation_id/org/specialist), written async via BullMQ; plus low-cardinality Prometheus kb_retrieval_* metrics (total, empty, top_score, latency).
- Event store = our Postgres, NOT PostHog/Prometheus-as-bus. The valuable query is a join to
Message/Correction(which live in Postgres); volume is human-paced; and retrieval events carry customer query text that must stay inside the multi-tenant PII boundary. Prometheus is a metrics plane (can't hold per-event/high-cardinality records); external analytics is the wrong plane and a PII risk. ClickHouse is the documented scale-out path, not warranted now (would add stateful infra against the "PG/Redis/R2 only" deployment rule). - Enables KB-gap vs ranking-gap analysis: join retrieval → Expert
Correctionto ask "when the Expert fixed the draft, was the right doc even retrieved?" Retroactive from deploy onward (no backfill).
Decision 3 — Consolidate drafts onto the hybrid engine (#3166)
On the Specialist draft path, the agent self-fetched keyword KB and rendered it into the system prompt, while NestJS's hybrid result was only injected into history. Consolidate onto hybrid (the MRR-0.99 engine), keyword becomes an outage fallback.
Chosen approach (A′), shaped by #3190: #3190 had already moved the NestJS hybrid result into a security-hardened user-role history turn (untrusted KB must not arrive with system privilege). So consolidation = the agent stops self-fetching keyword when NestJS already provided KB, signaled by a kb_context_provided flag. We explicitly do not move KB into the system prompt (that would regress #3190). Rejected: original "pass content in a field rendered as a system-prompt block" (regresses #3190); agent calling the hybrid endpoint itself (redundant fetch + latency).
The flag's contract — gate on genuine completion (!paused && !degraded): kb_context_provided is true when the hybrid call genuinely completed (including a real 0-results hit — we trust it and don't resurrect keyword), and false only on an outage (paused, or circuit-open / failOpen). The wrinkle: search() fails open by returning { results: [], total: 0 } on circuit-open/error, otherwise indistinguishable from a genuine empty. So search() now tags fail-open returns with degraded: true (pause keeps paused: true); the caller sets kb_context_provided = ctx.paused !== true && ctx.degraded !== true. Two failed gates (both Greptile P1s on #3271): !paused alone wrongly skips fallback on degraded outages; results.length > 0 wrongly falls back on real empties. The flag is also forwarded on the legacy LEGACY_AGENT_API chat/stream payloads, not just v1 — else legacy deployments keep the duplicate keyword fetch.
Also: the hybrid search() on the draft path is now specialist-scoped (specialistId), matching the keyword path's ADR-008 K1/K2 scoping (it was previously org-only — a latent over-broad-visibility gap).
Consequences
- Drafts now retrieve KB once (hybrid), specialist-scoped; keyword is a documented outage fallback. Redundant double-retrieval removed.
KbRetrievalService//kb/retrievalretained (fallback only); the unused/v1/agent-api/contextendpoint left as-is.- Open / deferred: whether prompt-slot placement materially affects answer quality (#3166 hypothesis 3) is unvalidated — it needs an end-to-end answer-quality eval (faithfulness / answer-relevancy) that does not yet exist. Slot placement is an answer-quality concern, not a retrieval-relevance one; do not conflate. The eval harness measures retrieval, not generation.