KB Retrieval Event Collection — Design

Date: 2026-06-23 Status: Approved design, pre-implementation Author: Alex (with Claude) Related: retrieval eval harness (rag/evals/retrieval/, PR #3091); consolidation issue #3166

Problem

We can now measure retrieval quality on a synthetic corpus (the eval harness), but we have no production signal about how retrieval performs on real traffic, and no way to monitor it. The valuable signals are partly captured but disconnected:

• audit_log already records kb.retrieval.context_loaded (doc_ids, top_k, query_hash) — but no per-doc scores, and no link to the resulting draft or the Expert's outcome.
• Correction captures the Expert's edit (original vs corrected draft) — but is not linked to which docs were retrieved.
• Per-doc relevance scores are computed then dropped before anything is persisted.
• The Expert workspace KB search emits no telemetry.

The missing piece is the join: nothing connects "what retrieval returned" to "what the Expert did about the draft." That join is the production relevance signal.

Goals

Collect "valuable events" to (A) monitor retrieval health, (B) harvest real ground truth to feed the eval harness, and (C) enable a correction/feedback loop. All three are in scope; B is the north star (it makes the harness measure reality), and A falls out of the same plumbing.

Architecture decision — the "bus"

Events and metrics are different data planes; we use the right tool for each, and explicitly not an external analytics SaaS.

Plane	Tool	Why
Events (system of record for B/C)	Dedicated Postgres table (kb_retrieval_event), written async via BullMQ	The valuable query is a join to existing Postgres entities (Message/Correction/ExpertQueueItem); co-locating events makes it plain SQL, no ETL. Volume is human-paced (thousands–tens of thousands), well within Postgres. jsonb gives flexible schema.
Metrics (health, goal A)	Prometheus (existing /metrics)	Aggregate counters/histograms; low-cardinality labels only.
Product analytics	PostHog/Segment — rejected	External SaaS would export customer query content out of the strict multi-tenant PII boundary; it is sampled/lossy and not joinable to our relational entities.

Scale-out path (later, only if volume demands): tee the same events into ClickHouse for heavy OLAP. The event-table abstraction is forward-compatible; not now (would add stateful infra against the "PG/Redis/R2 only" deployment rule).

Prometheus is rejected as the event bus because it stores aggregates, not records — it physically cannot hold doc_id/query/message_id or support joins.

Event taxonomy

Event	When	Status
retrieval (core)	A draft's KB context is resolved	Built now
draft_outcome	Expert accepts / edits / rejects	Derived by join, not emitted — outcome already lives in Message/ExpertQueueItem/Correction
kb_search	Expert manual KB search in workspace	Deferred (frontend click stream)

Correlation keys

• message_id — the precise 1:1 link from a retrieval to its draft and therefore to its outcome (Message → ExpertQueueItem → Correction). The money join. Nullable-tolerant (a draft may fail to persist).
• conversation_id (thread) — denormalized rollup dimension, always present. Enables thread-level patterns (e.g. client rephrases after a weak answer → re-retrieval), avoids a message → conversation join hop, and survives the null-message_id edge case.
• org_id / specialist_id — tenancy + analysis dimensions (Org × Specialist per ADR-020).

Capture point

NestJS, at draft-save (conversations.service.ts), where message_id and the scored retrieval results coexist. The agent retrieves before the Message row exists, so it cannot know message_id; NestJS already attaches retrievedKb citation metadata to the draft (conversations.service.ts:2454) and holds the scored results (:2261), so this is an enrichment of an existing hook, not new capture infrastructure.

Post-#3166 (when the agent's KB context is consolidated onto the hybrid retriever), this same capture point records the consolidated retrieval — forward-compatible. engine is recorded per-event so the data is correct before and after that change.

Schema

CREATE TABLE kb_retrieval_event (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  -- tenancy & analysis dimensions (Org × Specialist per ADR-020)
  org_id          uuid NOT NULL,
  specialist_id   uuid NOT NULL,
  -- correlation keys
  conversation_id uuid NOT NULL,         -- thread rollup dimension (always present)
  message_id      uuid NULL,             -- precise 1:1 draft link; null if no draft persisted
  -- retrieval facts
  engine          text NOT NULL,         -- 'hybrid' | 'keyword' | ...
  query           text NOT NULL,         -- retrieval query (first-party, within tenancy)
  top_k           int  NOT NULL,
  result_count    int  NOT NULL,         -- 0 => empty-result signal (goal A)
  latency_ms      int  NULL,
  retrieved       jsonb NOT NULL DEFAULT '[]',  -- [{doc_id, chunk_id, rank, score}] — the scores we drop today
  created_at      timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX kb_retrieval_event_org_spec_time ON kb_retrieval_event (org_id, specialist_id, created_at DESC);
CREATE INDEX kb_retrieval_event_conversation  ON kb_retrieval_event (conversation_id);
CREATE UNIQUE INDEX kb_retrieval_event_message_uq ON kb_retrieval_event (message_id) WHERE message_id IS NOT NULL;

Notes:

• retrieved jsonb is the flexible part — captures per-doc {doc_id, chunk_id, rank, score} (dropped today); no migration when the shape evolves (e.g. add used_in_final).
  • Enriched in #3281 (no migration, as predicted): each candidate now also carries { bm25_score, vector_similarity, pre_rerank_rank, rerank_score, used_in_final }. The sub-scores are surfaced by the rag hybrid pipeline (_collect_sub_scores) and threaded through haystack.client.ts → ContextResult.subScores → the builder; used_in_final is the citation-extracted relevance label. A sibling kb_search_event table captures the Expert manual-search signal (kb.search.performed / kb.result.clicked / kb.citation.inserted). See docs/analytics/kb-retrieval-gap-analysis.md for the KB-gap-vs-ranking-gap proof query.
• query stored raw — it is the client's message text, already in Message; stays inside the tenancy boundary (no external export).
• result_count as a typed column (not only derivable from retrieved) so the empty-result health metric is a cheap indexed query.
• The TypeORM entity must carry the tenancy docstring (Org × Specialist) — a code-review gate in this repo.
• A child kb_retrieval_event_doc table was considered (relational per-doc rows) and rejected for v1: jsonb is simpler at this volume and the per-doc analysis is occasional, not hot-path.

Write path & failure semantics

draft Message persisted (business tx commits)
   ├─ (sync, in-process) Prometheus counters  — cheap, never fails the request
   └─ (async) BullMQ job → processor INSERT ... ON CONFLICT DO NOTHING

• Async via BullMQ, after commit — never blocks/rolls back a client reply. Matches "audit failure must not roll back business" and "no setImmediate — use BullMQ".
• Idempotent — INSERT … ON CONFLICT DO NOTHING on the partial unique index (message_id) WHERE message_id IS NOT NULL; BullMQ jobId = message_id for cross-pod dedup under active-active. (Null message_id may duplicate; rare.)
• Fail-open — enqueue failure logs + drops; the draft still sends. Eventual consistency, intentional.
• Prometheus labels low-cardinality — engine only; never org_id/doc_id. Per-org/per-doc analysis is the Postgres table's job.

Metrics emitted (Prometheus)

• kb_retrieval_total{engine} (counter)
• kb_retrieval_empty_total{engine} (counter; result_count == 0)
• kb_retrieval_top_score (histogram; best score per retrieval)
• kb_retrieval_latency_ms (histogram)

What it unlocks

Goal	Mechanism
A — health	empty-result rate, score distribution, latency (Prometheus + PG scans); alert on empty-result spikes
B — ground truth	kb_retrieval_event ⋈ Correction/Message: accepted-as-is drafts where doc X ranked top → positive (query → relevant-doc); heavily-edited → retrieval suspect. Feeds the eval harness via a new production QuerySource (the harness's CorpusProvider/query-source seam — no harness rewrite)
C — feedback	Correction ⋈ retrieved: distinguishes KB gap (right info not retrieved at all → fix KB/ingestion) from ranking gap (retrieved but ranked low → fix reranker)

Testing

• Unit: event-builder (Message + scored results → row); BullMQ processor idempotency (ON CONFLICT).
• Integration (Postgres): partial-unique-index dedup under duplicate dispatch.
• Assert Prometheus counters increment.
• No live LLM required — all deterministic.

Scope

In scope (v1): kb_retrieval_event table + entity; NestJS capture at draft-save (scores + correlation keys); BullMQ async write; Prometheus health metrics; the join queries for B/C as documented SQL (not yet a dashboard).

Deferred (seams noted, not built):

• Frontend kb_search click telemetry (Expert workspace).
• A production QuerySource adapter feeding the eval harness from harvested events (depends on this data existing first).
• A retrieval-quality dashboard.
• ClickHouse tee for OLAP at scale.
• An explicit draft_outcome event (derive by join until a need appears).

Open questions / risks

• query retention / PII. Stored raw inside our Postgres (same boundary as Message.content). If a retention/redaction policy is later required, add a TTL/redaction job — not v1.
• Capture accuracy pre-#3166. Until consolidation, the agent's actual KB context comes from the keyword engine while NestJS holds the hybrid results. v1 records the retrieval NestJS resolves at draft-save and stamps engine accordingly; after #3166 there is a single engine. Document this clearly so early data isn't misread.