Skip to main content

Consolidate Specialist Draft Retrieval onto the Hybrid Engine (#3166) — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stop the agent self-fetching keyword-only KB on the draft path; NestJS's existing hybrid retrieval (already injected as a #3190 user-role turn) becomes the single KB source. Keyword stays as an outage fallback.

Architecture: NestJS signals kb_context_provided on the chat request when it attempted (non-paused) hybrid retrieval; the agent skips its _load_kb_context keyword fetch whenever that flag is true. No KB content moves into the system prompt (preserves #3190). The hybrid search is also specialist-scoped (ADR-008 K1/K2).

Tech Stack: NestJS 11 / TypeScript (api), FastAPI / Python 3.11 (agent), Jest, pytest.

Spec: docs/superpowers/specs/2026-06-25-kb-consolidate-hybrid-retrieval-design.md

Conventions​

  • NestJS commands run from api/; agent commands from agent/. Commit after each green step.
  • Commit messages: feat(3166): … / test(3166): ….
  • Only the runAgentPipeline draft path (conversations.service.ts:2055) injects hybrid KB; the stream (:4674) and regenerateDraft (:5044) call sites do not and are out of scope (they keep current behavior).

File structure​

api/src/common/agent.client.ts                         # + kbContextProvided on req + payload
api/src/conversations/conversations.service.ts          # specialistId in search; compute + pass flag
agent/models/chat.py                                    # + kb_context_provided on v1 ChatRequest
agent/routers/chat.py                                   # map flag into legacy request
agent/main.py                                           # legacy field + _maybe_load_kb_context guard
agent/tests/test_kb_context_gate.py                     # NEW: guard unit test
api/src/conversations/conversations.service.spec.ts      # extend: flag + specialistId assertions

Task 1: Agent client — carry kbContextProvided on the chat payload​

Files:

  • Modify: api/src/common/agent.client.ts

  • Step 1: Add the field to the AgentChatRequest interface

Find the AgentChatRequest interface (ends with throwOnError?: boolean; then }). Add before the closing brace:

  /** #3166: true when NestJS already performed (non-paused) hybrid KB retrieval
   * and injected it into history — tells the agent NOT to self-fetch keyword KB.
   * Omitted/false ⇒ agent falls back to its keyword self-fetch. */
  kbContextProvided?: boolean;
  • Step 2: Add the field to the AgentV1ChatPayload type

In the AgentV1ChatPayload type (the one with message_history, latest_message, agent_context), add:

  kb_context_provided?: boolean;
  • Step 3: Set it in buildV1ChatPayload

In buildV1ChatPayload, add to the returned object (e.g. after the agent_context: req.agentContext, line):

      kb_context_provided: req.kbContextProvided ?? false,
  • Step 4: Build

Run (from api/): npm run build Expected: no TypeScript errors.

  • Step 5: Commit
git add api/src/common/agent.client.ts
git commit -m "feat(3166): carry kb_context_provided on agent chat payload"

Task 2: NestJS draft path — specialist-scope search + compute/pass the flag​

Files:

  • Modify: api/src/conversations/conversations.service.ts (runAgentPipeline, ~:2089–:2440)

  • Test: api/src/conversations/conversations.service.spec.ts

  • Step 1: Write/extend the failing test

In conversations.service.spec.ts, the suite already constructs ConversationsService with a mocked HaystackRetrievalService ({ search: jest.fn() }) and a mocked agent client. Add a test that drives sendMessage (the path that calls runAgentPipeline) and asserts the agent chat mock received kbContextProvided: true and that haystackRetrieval.search was called with a specialistId. Use the existing test's setup/mature helpers; the new assertions:

it("#3166: passes kbContextProvided=true and specialist-scopes hybrid search", async () => {
  // haystack returns a non-paused result
  haystackRetrieval.search.mockResolvedValue({ results: [], paused: false });
  const chatSpy = jest.fn().mockResolvedValue({
    reply: "ok", confidence: 50, riskLevel: "low", flagForReview: true,
  });
  agentClient.forOrg.mockResolvedValue({ chat: chatSpy });

  await service.sendMessage(/* use the same args shape the existing sendMessage tests use */);

  expect(haystackRetrieval.search).toHaveBeenCalledWith(
    expect.any(String),
    expect.any(String),
    expect.objectContaining({ specialistId: expect.anything() }),
  );
  expect(chatSpy).toHaveBeenCalledWith(
    expect.objectContaining({ kbContextProvided: true }),
  );
});

Adapt the sendMessage(...) arguments and the agentClient/haystackRetrieval mock variable names to the existing spec's conventions (read the top of the file). If the existing suite already has a "drafts a reply" test, model this one on it.

  • Step 2: Run to confirm it fails

Run (from api/): npm test -- conversations.service Expected: FAIL — kbContextProvided is undefined and/or specialistId not passed.

  • Step 3: Specialist-scope the hybrid search

In runAgentPipeline, the search call (~:2300) currently reads:

        const ctx = await this.haystackRetrieval.search(retrievalOrgId, dto.content ?? "", {
          limit: Number(process.env.HAYSTACK_TOP_K ?? 5),
          source: "kb",
          failOpen: true,
        });

Change the options object to include the specialist:

        const ctx = await this.haystackRetrieval.search(retrievalOrgId, dto.content ?? "", {
          limit: Number(process.env.HAYSTACK_TOP_K ?? 5),
          source: "kb",
          failOpen: true,
          specialistId: conv.specialistId,
        });
  • Step 4: Track whether NestJS provided KB

Near the top of runAgentPipeline where retrievedKbContext is declared (:2089, let retrievedKbContext: ContextResult[] = [];), add right after it:

    // #3166: true once a non-paused hybrid retrieval runs — signals the agent
    // to skip its keyword self-fetch (the hybrid result is the single source).
    let kbContextProvided = false;

Then, inside the retrieval try block immediately after retrievedKbContext = ctx.results;, set it:

        retrievedKbContext = ctx.results;
        kbContextProvided = ctx.paused !== true;

(Leave the existing history.unshift({ role: "user", ... }) #3190 injection unchanged.)

  • Step 5: Pass the flag to the agent chat call

In the orgAgentClient.chat({ ... }) call within runAgentPipeline (~:2423), add the field (e.g. after inboundArtifacts,):

          kbContextProvided,
  • Step 6: Run the test to confirm it passes

Run (from api/): npm test -- conversations.service Expected: the new test passes; existing tests still pass.

  • Step 7: Commit
git add api/src/conversations/conversations.service.ts api/src/conversations/conversations.service.spec.ts
git commit -m "feat(3166): specialist-scope hybrid search + signal kb_context_provided"

Task 3: Agent v1 request — accept + map kb_context_provided​

Files:

  • Modify: agent/models/chat.py

  • Modify: agent/routers/chat.py

  • Step 1: Add the field to the v1 ChatRequest

In agent/models/chat.py, the ChatRequest(BaseModel) (the one with message_history, latest_message, inbound_artifacts). Add after the inbound_artifacts field:

    # #3166: when true, NestJS already provided hybrid KB context (in history);
    # the agent must NOT self-fetch keyword KB. Accepts the camelCase alias.
    kb_context_provided: bool = Field(
        default=False,
        validation_alias=AliasChoices("kb_context_provided", "kbContextProvided"),
    )

(AliasChoices and Field are already imported in this file.)

  • Step 2: Map it through _to_legacy_request

In agent/routers/chat.py, inside _to_legacy_request, add to the main.ChatRequest(...) constructor call (e.g. after run_id=req.run_id,):

        kb_context_provided=req.kb_context_provided,
  • Step 3: Verify import

Run (from agent/): python3 -c "import models.chat, routers.chat; print('ok')" Expected: ok (no import error). (Full behavior is covered in Task 4.)

  • Step 4: Commit
git add agent/models/chat.py agent/routers/chat.py
git commit -m "feat(3166): accept kb_context_provided on agent v1 request"

Task 4: Agent — gate the keyword self-fetch (TDD)​

Files:

  • Modify: agent/main.py

  • Test: agent/tests/test_kb_context_gate.py

  • Step 1: Add the field to the legacy ChatRequest

In agent/main.py, the class ChatRequest(BaseModel) (~:257) currently has kb_context: list[dict[str, Any]] = Field(default_factory=list). Add a sibling field:

    kb_context_provided: bool = False
  • Step 2: Write the failing test — agent/tests/test_kb_context_gate.py
import asyncio
from unittest.mock import AsyncMock, patch

import main


def _req(**kw):
    return main.ChatRequest(
        message="hi", role="support", org_id="o1", specialist_id="s1", **kw
    )


def test_skips_keyword_fetch_when_provided():
    req = _req(kb_context_provided=True)
    with patch.object(main, "_load_kb_context", new=AsyncMock()) as fetch:
        asyncio.run(main._maybe_load_kb_context(req, "res-1"))
        fetch.assert_not_awaited()


def test_fetches_keyword_when_not_provided():
    req = _req(kb_context_provided=False)
    with patch.object(main, "_load_kb_context", new=AsyncMock()) as fetch:
        asyncio.run(main._maybe_load_kb_context(req, "res-1"))
        fetch.assert_awaited_once_with(req, "res-1")
  • Step 3: Run to confirm it fails

Run (from agent/): python3 -m pytest tests/test_kb_context_gate.py -v Expected: FAIL — main._maybe_load_kb_context does not exist.

  • Step 4: Add the guard helper and use it

In agent/main.py, add the helper next to _load_kb_context (~:361):

async def _maybe_load_kb_context(req: "ChatRequest", resource_id: str) -> None:
    """#3166: only self-fetch keyword KB when NestJS did not already provide
    hybrid KB context (signaled by kb_context_provided). When provided, the
    hybrid result is already in the conversation history (#3190)."""
    if req.kb_context_provided:
        return
    await _load_kb_context(req, resource_id)

Then replace the two call sites that currently call _load_kb_context directly (/chat ~:521 and /chat/stream ~:847):

        await _maybe_load_kb_context(req, response_session_id)
  • Step 5: Run to confirm it passes

Run (from agent/): python3 -m pytest tests/test_kb_context_gate.py -v Expected: 2 passed.

  • Step 6: Run the agent suite for regressions

Run (from agent/): python3 -m pytest tests/ -q Expected: no new failures.

  • Step 7: Commit
git add agent/main.py agent/tests/test_kb_context_gate.py
git commit -m "feat(3166): gate keyword self-fetch on kb_context_provided (TDD)"

Task 5: NestJS — paused retrieval sends kbContextProvided=false (TDD)​

Files:

  • Test: api/src/conversations/conversations.service.spec.ts

This locks the outage-fallback contract. The implementation from Task 2 (kbContextProvided = ctx.paused !== true) already produces it; this task proves it.

  • Step 1: Add the test
it("#3166: paused hybrid retrieval sends kbContextProvided=false (keyword fallback)", async () => {
  haystackRetrieval.search.mockResolvedValue({ results: [], paused: true });
  const chatSpy = jest.fn().mockResolvedValue({
    reply: "ok", confidence: 50, riskLevel: "low", flagForReview: true,
  });
  agentClient.forOrg.mockResolvedValue({ chat: chatSpy });

  await service.sendMessage(/* same args as the Task 2 test */);

  expect(chatSpy).toHaveBeenCalledWith(
    expect.objectContaining({ kbContextProvided: false }),
  );
});
  • Step 2: Run

Run (from api/): npm test -- conversations.service Expected: passes (with the Task 2 implementation in place).

  • Step 3: Commit
git add api/src/conversations/conversations.service.spec.ts
git commit -m "test(3166): paused retrieval falls back to keyword (kbContextProvided=false)"

Task 6: Post-merge verification (AC#4) — not a code change​

  • After merge to dev, trigger the rag-eval-baseline GitHub Actions workflow (add the run-rag-eval label to the PR, or workflow_dispatch) and record the result. The served engine is now hybrid; the numbers should reflect Engine A. Note in the PR that the answer-quality eval (AC#5) remains a deferred follow-up (no end-to-end eval exists yet).

Self-Review​

Spec coverage:

  • Specialist-scope hybrid search (ADR-008) → Task 2 Step 3. ✓
  • Compute kb_context_provided (true unless paused) → Task 2 Step 4; paused→false proven Task 5. ✓
  • Pass flag through agent.client → Task 1; v1 request + map → Task 3; legacy field + guard → Task 4. ✓
  • Agent skips keyword self-fetch when provided; falls back when not → Task 4 (TDD). ✓
  • Keep #3190 user-role injection, do NOT touch system prompt → Task 2 explicitly leaves history.unshift unchanged; no chat_helpers/system-prompt edits anywhere. ✓
  • Re-run eval (AC#4) → Task 6. Answer-quality (AC#5) deferred → noted. ✓
  • Out of scope (stream/regenerate call sites, removing KbRetrievalService, /v1/agent-api/context) → untouched. ✓

Placeholder scan: The Task 2/5 tests say "adapt to the existing spec's sendMessage arg shape / mock names" — this is a real instruction to match existing conventions in a file the implementer reads, not a TODO. All code blocks are complete.

Type/name consistency: NestJS kbContextProvided (camel) on AgentChatRequest + spec; wire kb_context_provided (snake) on AgentV1ChatPayload and both Python models; helper _maybe_load_kb_context(req, resource_id) defined in Task 4 and called in Task 4 Steps. ctx.paused matches HaystackRetrievalService.search()'s ContextResponse & { paused?: boolean } return. Consistent.

Last updated on by Degen