Skip to main content

RAG Retrieval-Quality Eval Harness — Design

Date: 2026-06-23 Status: Approved design, pre-implementation Author: Alex (with Claude)

Problem​

h.work has no metric for retrieval quality. The only existing eval (rag/evals/ragas_smoke.py) measures answer quality (faithfulness, answer_relevancy) on 5 hardcoded fixture rows with a deterministic stub LLM and fake embeddings — it never asks "did retrieval fetch the right document?". Nothing in the codebase computes Recall, MRR, Precision, or any retrieval metric. So the question "is the search good?" is currently unanswerable.

We want to change that so we can eventually tune re-ranking with evidence instead of guesswork.

Two retrieval engines exist, tangled in production​

Investigation of the live inbound→draft path found that two competing retrieval implementations both fire on every drafted reply, independently, then both land in the same LLM prompt:

  • Engine A — Haystack hybrid + reranker (HaystackRetrievalService → rag service /retrieval/run): BM25 + vector + RRF fusion + cohere/rerank-v3.5. Called at api/src/conversations/conversations.service.ts:2149. Its results are folded into the prompt's conversation history as a system message (:2156-2159) — a weak prompt position.
  • Engine B — keyword-only (KbRetrievalService → NestJS /kb/retrieval, AgentTokenGuard): scores only document title/description/filename, no chunk-body search, no embeddings, no rerank. Called by the Python agent at agent/main.py:507 → agent/kb_client.py. Its results occupy the prompt's labeled "Relevant knowledge base context:" section — the strong position.

Net finding: every draft retrieves KB context twice from two different engines; the better (reranked) engine sits in the weaker prompt slot, while the keyword engine owns the labeled KB slot. Tuning the reranker today would tune the de-emphasized input. This is a likely architecture bug — but the decision to consolidate must be driven by numbers, not taste. Hence: measure first.

Goal​

Build an offline, reproducible evaluation harness that scores each retrieval engine in isolation on a synthetic query set and reports Hit@k, Recall@k, and MRR side-by-side.

Primary deliverable: the first retrieval-quality numbers this system has ever had, and a head-to-head Engine A vs Engine B comparison that informs (a) which engine to keep/consolidate onto and (b) reranker tuning.

Level of measurement​

This measures the retrieval component in isolation (Level 2), not the end-to-end user flow (Level 1, answer quality). Rationale: reranking is a retrieval-stage knob; an end-to-end reply score moves for too many reasons (LLM, prompt assembly, etc.) to attribute a change to retrieval. A retrieval-only score moves only when retrieval changes — which is exactly what tuning requires.

The two engines are scored separately (not mixed as in production) because the question "which retriever is better?" cannot be answered while their results are combined.

Ground truth approach​

Synthetic — generate questions from real KB chunks (approach A only, for v1). For each chunk in the corpus, an LLM writes the question that chunk answers; the source chunk becomes the labeled-relevant document. This yields a labeled query set at scale, for free, with exactly one known-relevant chunk per query.

That single-gold structure dictates the metric set: Hit@k, Recall@k, MRR are the natural fits ("did the needle come back, and how high?"). Graded NDCG and Precision@k are not built — they need multiple/graded relevant docs per query, which synthetic-single-gold doesn't provide.

Known caveat (documented, not fixed): synthetic questions can be too lexically similar to their source chunk, which can unfairly flatter the keyword engine (Engine B). Mitigation: prompt the generator to paraphrase, not quote. This is a measurement caveat to state in the report, not a defect to eliminate.

Corpus approach​

Start with a controlled seed corpus; design for real dev data later (approach C). v1 ingests a fixed, committed seed corpus (~15–30 docs of varied shape) into an ephemeral pgvector store + kb_documents rows, so both engines index the same corpus through their real code paths. The corpus source sits behind a CorpusProvider interface so a future DevKbCorpusProvider (real dev KB) is a drop-in, not a rewrite.

Location​

In-repo, under rag/evals/retrieval/ (alongside the existing ragas_smoke.py). Decisive reason: the dual-engine comparison must invoke the real retrieval code (the actual Haystack components and the real keyword scoring), which only an in-repo harness can import without HTTP indirection or copy-drift.

Comparison normalization​

The two engines return different granularities — Engine A returns chunks (from pgvector), Engine B returns documents (metadata-scored). The harness normalizes both to document granularity: gold = the gold chunk's parent document; an engine "hits" if that parent document appears in its top-k. This is the only fair common denominator.

Expected (and acceptable) result: Engine B may score poorly on questions written from chunk body text, because it only scores title/description/filename. That is a finding (evidence for consolidation), not a harness bug.

Components​

Each component is isolated and independently testable.

  1. CorpusProvider (adapter) — get_documents() -> [Doc]. v1: SeedCorpusProvider over committed seed docs. Future: DevKbCorpusProvider (C-seam).
  2. Ingestor — loads corpus into an ephemeral pgvector store + kb_documents rows (chunk+embed for Engine A; document rows for Engine B). Teardown in a finally.
  3. QuestionGenerator — per chunk, OpenRouter LLM (temperature 0) writes a paraphrased question. Output: (question, gold_doc_id, gold_chunk_id) pairs. Cached to a committed JSON file; --regenerate re-runs deliberately.
  4. Engine adapters — common interface retrieve(query, k) -> [ranked doc_ids]:
    • EngineA (Haystack hybrid+rerank): builds a pipeline from the real Haystack components pointed at the ephemeral store, accepting a config (reranker model, top_k, rerank on/off, fusion top_k) — production bakes these in at import, so the adapter constructs its own parameterized instance. This config seam is what makes a later reranker sweep a simple loop.
    • EngineB (keyword-only): invokes the real KbRetrievalService keyword scoring.
    • Both normalize to document-granularity ranked doc_id lists.
  5. Metrics — pure functions: Hit@k, Recall@k, MRR over k ∈ {1,3,5,10,20}, aggregated (mean) per engine.
  6. Reporter — results.md (engine × metric × k table) + per_query.jsonl (drill-down).
  7. Runner (CLI) — python -m rag.evals.retrieval.run orchestrates the full flow with deterministic seeds.

Data flow​

1. CorpusProvider.get_documents()        → fixed seed doc set (committed)
2. Ingestor.load(docs)                   → ephemeral pgvector + kb_documents rows
3. QuestionGenerator                     → per chunk, LLM writes paraphrased Q
   (cache hit? load committed JSON. miss? call OpenRouter, write cache)
                                         → labeled set: [(question, gold_doc_id)]
4. for engine in [EngineA, EngineB]:
       for (question, gold) in labeled_set:
           ranked_doc_ids = engine.retrieve(question, k_max=20)
           record(engine, question, gold, ranked_doc_ids)
5. Metrics.compute(records)              → Hit@k, Recall@k, MRR  (k ∈ 1,3,5,10,20)
6. Reporter                              → results.md + per_query.jsonl
7. Ingestor.teardown()                   → drop ephemeral store (finally)

The labeled set is generated once and committed, so steps 4–6 are pure replay: identical query set every run, only engine behaviour varies. That makes "did my reranker change help?" a clean before/after.

Reproducibility & determinism​

  • Seed corpus committed → fixed input.
  • Generated questions cached + committed as JSON → generator runs once; reruns replay the cache (no token spend, no drift, identical queries). --regenerate re-runs generation deliberately.
  • Generator at temperature 0; metrics are pure functions of (gold, ranking).
  • Result: two runs of the same code produce byte-identical metrics. A metric delta means the engine changed, never noise.

Error handling — fail-loud (opposite of production)​

Production retrieval is fail-open (return [], never break the user). The eval is the inverse — fail-loud, never silently degrade, because a swallowed error becomes a fake metric.

  • Engine error on a query → recorded as an explicit error outcome (counts as a miss) and surfaced in the report. Never silently skipped.
  • Question-generation failure → logged, excluded, excluded-count printed. No silent truncation; the report always states "N questions, M errored."
  • Ephemeral store teardown in finally so a mid-run crash doesn't leak a Postgres schema.

Testing the harness​

  • Unit tests for metrics.py (TDD) — hand-built rankings with known answers: gold at rank 1 → MRR 1.0; gold at rank 3 → MRR 0.333…; gold absent → Hit@k 0, etc. This is the part that must be correct.
  • Tiny fixture corpus (3–4 docs) for a fast self-test of full-pipeline wiring, separate from the real seed corpus.
  • Existing ragas_smoke.py left untouched.

File layout​

rag/evals/retrieval/
  corpus/
    seed_docs/                 # committed fixed corpus
    provider.py                # CorpusProvider interface + SeedCorpusProvider
                               #   (DevKbCorpusProvider = future C-seam)
  ingest.py                    # Ingestor: load → ephemeral store, teardown
  generate_questions.py        # QuestionGenerator + cache
  questions.cache.json         # committed labeled set
  engines/
    base.py                    # RetrievalEngine interface: retrieve(q, k) -> [doc_id]
    engine_a_haystack.py       # configurable hybrid+rerank adapter
    engine_b_keyword.py        # keyword adapter
  metrics.py                   # pure Hit@k / Recall@k / MRR
  report.py                    # results.md + per_query.jsonl
  run.py                       # CLI orchestrator
  tests/
    test_metrics.py
    fixtures/

Scope​

Delivers: first-ever retrieval-quality numbers; Engine A vs Engine B head-to-head on identical queries; a reranker-config seam so tuning becomes a loop; a reproducible, CI-able baseline.

Deliberately defers (YAGNI), each with a named seam so it's additive, not a rewrite:

  • LLM-as-judge relevance (approach B) — cross-check synthetic labels.
  • Human calibration anchor (~20 queries, approach C-labels).
  • Production-signal mining (approach D) — needs telemetry that doesn't exist yet (agent path logs only query_hash + doc_ids, no scores).
  • Graded NDCG / Precision@k (needs multi/graded gold).
  • Real-dev corpus source (DevKbCorpusProvider).
  • The actual reranker parameter sweep (the harness provides the seam; running the sweep is follow-on work).

Open questions / risks​

  • Ephemeral pgvector provisioning — implementation must decide how the eval stands up an isolated pgvector (local Docker, a throwaway schema in dev PG, or an in-process option). To be resolved in the implementation plan.
  • Importing real engine code — Engine A builds from Haystack components (Python, in-process). Engine B decision (resolved 2026-06-23): reimplement its keyword scoring faithfully in Python rather than call NestJS over HTTP. Rationale: Engine B is the path we expect to consolidate away, the harness mainly needs to demonstrate "metadata-only keyword retrieval is much worse," and a pure-Python harness avoids standing up NestJS. Accepted risk: the Python port is a copy of KbRetrievalService scoring (title×4, description×2, filename×3, BM25-ish over kb_documents metadata) and can drift if the NestJS logic changes. Mitigation: a code comment in engine_b_keyword.py pinning the source KbRetrievalService file:line it mirrors, and a fidelity note in the report. Revisit (switch to HTTP, option ii) only if Engine B turns out to be the engine we keep.
Last updated on by Degen