ADR-008: Knowledge Base architecture — K1 + K2 (defer K3)

Date: 2026-05-25 Status: Accepted Issue: #541 (this ADR); reads from ADR-007 / #537; cascades into KB implementation issues (filed as #541a–d) Authors: Eusden, ethumanity

Note (2026-06-04): The K1/K2 ACL decision below stands. The retrieval backend has since moved from RAGflow to Haystack/Hayhooks per ADR-024; the ACL primitive ((org_id, specialist_id) against expert_access) is enforced in humanwork application code and is backend-agnostic. The 2026-05-28 RAGflow shared-tenant addendum below is historical — see ADR-024 and docs/runbooks/haystack-deploy.md for current dev provisioning.

Context

The Knowledge Base (KB) is the platform's durable document store for Specialist context: client-supplied policy documents, vendor contracts, HP-authored persona playbooks, eventual RAG retrieval inputs. Existing issues that touch KB (#296 onboarding business-context upload; future RAGFlow integration #291/#315) implicitly assume a per-Org KB. That assumption was made before ADR-007 landed.

ADR-007 (Expert access scope) moved the canonical visibility key for Expert data access to (Org × Specialist) — a dual-scope expert_access table that grants either a single (org × specialist) pair OR a whole org. Because KB documents are Expert-readable, the KB ACL must align with the same primitive. Anything else creates a second visibility model for the same data, and the two will drift.

We need to pick the dominant KB shape (what (scope_key) does a KB document attach to?) before any KB implementation lands.

The three candidate shapes

Shape	Data unit	Examples	Expert ACL primitive
K1. Per-(Org × Specialist)	"Acme's KYC compliance docs uploaded only into Acme × Carol-KYC"	Per-client per-Specialist data: vendor contracts, audit logs, KYC procedures, comp policies	expert_access(scope='specialist', org_id, specialist_id, expert_id) — ADR-007 already covers this. No new table.
K2. Per-Specialist global	HP-owned persona library for Bob the Finance Specialist (training playbook, persona voice guide, generic finance reference)	Persona authoring artifacts, HP-curated reference material	Derive from ADR-007: any Expert with a grant on (any_org × bob) can see Bob's global library. No new ACL primitive.
K3. Per-Org cross-Specialist	Acme's company handbook used by every Acme Specialist (Bob + Carol + …)	Brand voice, holiday calendar, company-wide HR policies	NOT covered by ADR-007 — would need either a new org_resource_access grant OR a "shared with all Specialists in this Org" flag on K1 docs

Decision

Adopt K1 + K2. Defer K3.

If a real cross-Specialist client document appears, model it as a K1 document with a share_across_specialists: true flag at upload time, which duplicates the doc-to-specialist association rows (not the bytes). We will not introduce a third ACL primitive until a concrete use case forces it.

Rationale

1. K1 matches the dominant shape. Almost every document a client uploads is meaningful only to one Specialist (KYC docs → KYC Specialist; sales scripts → Sales Specialist). The leak that motivated ADR-007 is exactly this case — KB inherits the same fix for free.
2. K2 has no ACL ambiguity. HP-owned. Any Expert with any grant on the Specialist can see it. There is no client data and therefore no cross-tenant risk.
3. K3 looks tempting but is cheaper to fake than to model. "Acme employee handbook" → upload once, flag as shared-across-specialists, association rows fan out at insert time. Storage cost: trivial (associations are tiny; bytes live once in object storage). Query cost: K1's existing access check works unchanged. No new ACL table, no new socket room, no new audit pattern.
4. Aligns with ADR-007. The expert_access table is the single visibility key for the platform. K3 would force a parallel org_resource_access table and a second ACL pathway — exactly the drift ADR-007 was written to prevent.

Rejected alternatives

K3 as a first-class ACL primitive

Rejected. Introduces a second visibility model alongside expert_access. Every future ACL check would have to consult both tables. The "Acme handbook" use case is solvable with a K1 association fan-out at far lower architectural cost.

Per-Org KB (the original implicit assumption)

Rejected. Pre-dates ADR-007. Would mean a Finance Expert with only a (Acme × Bob) grant can read Acme's KYC compliance docs the moment they're uploaded org-wide. That's the exact leak ADR-007 closed.

"Anything goes, K1/K2/K3 all coexist"

Rejected. Three parallel ACL models, three retrieval pipelines for RAG, three socket-room patterns. Optionality at this layer is technical debt up front.

Schema sketch (for #541a)

K1: per-(Org × Specialist) documents

@Entity('kb_documents')
export class KbDocument {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  // Scope key — K1 documents are tied to a specific (org, specialist) pair.
  @Column('uuid')
  @Index()
  orgId: string;

  @Column('uuid')
  @Index()
  specialistId: string;

  // For K3-as-K1-fanout (deferred): when true, on insert we create one
  // kb_document_association row per assigned (org, specialist) pair so the
  // single doc is visible across all Specialists on that Org.
  @Column({ type: 'boolean', default: false })
  shareAcrossSpecialists: boolean;

  @Column()
  title: string;

  @Column({ type: 'text', nullable: true })
  description: string | null;

  // Object storage reference (R2/S3). Bytes never live in Postgres.
  @Column()
  storageKey: string;

  @Column({ type: 'bigint' })
  sizeBytes: number;

  @Column()
  mimeType: string;

  @Column({ type: 'uuid', nullable: true })
  uploadedBy: string | null; // user id (AM, SuperAdmin, or client uploader)

  @CreateDateColumn()
  createdAt: Date;

  @UpdateDateColumn()
  updatedAt: Date;

  @DeleteDateColumn()
  deletedAt: Date | null; // soft delete; audit retention
}

Composite index: (org_id, specialist_id, deleted_at) — primary access pattern is "list KB docs for this (Org × Specialist)".

K2: per-Specialist global library

K2 documents live in the same table with a sentinel org_id:

// K2 documents: org_id = HP_GLOBAL_ORG_UUID, specialist_id = the Specialist.
// Visibility derived from ADR-007: any expert_access row on this Specialist
// (any org) grants read.

A single boolean is_global: true column on kb_documents is an alternative — both work. Recommend the sentinel-org approach because it keeps the query shape uniform: every doc has a real (org_id, specialist_id) pair; the access-check helper doesn't branch on doc shape.

Access check (the load-bearing query)

-- Can Expert E read KB document D?
-- E has access iff there exists an expert_access row that grants either:
--   (scope='specialist', org_id=D.org_id, specialist_id=D.specialist_id)
--   (scope='org',        org_id=D.org_id)
-- AND (for K2 specifically) we treat HP_GLOBAL_ORG_UUID as "any grant on D.specialist_id passes".

SELECT 1
FROM kb_documents d
JOIN expert_access ea ON
  (
    -- K1 path
    (ea.scope = 'specialist' AND ea.org_id = d.org_id AND ea.specialist_id = d.specialist_id)
    OR (ea.scope = 'org' AND ea.org_id = d.org_id)
  )
  OR
  (
    -- K2 path: HP-global docs, visibility derived from any Specialist grant
    d.org_id = $HP_GLOBAL_ORG_UUID
    AND ea.specialist_id = d.specialist_id
  )
WHERE d.id = $documentId AND ea.expert_id = $expertId AND d.deleted_at IS NULL
LIMIT 1;

Helper service: KbAccessService.canReadDocument(expertId, documentId) — mirrors the existing ExpertAccessService pattern from #537. SuperAdmin and AM-for-this-org bypass via the existing role guards.

Access pattern walkthrough

For GET /kb?orgId=acme&specialistId=carol-kyc&q=…:

Caller	Returns
Client user (no Expert role)	K1 docs scoped to their own Org × Specialist via the existing client portal access rules. K2 not exposed to clients.
Expert with expert_access(scope='specialist', org='acme', specialist='carol-kyc')	K1 (Acme × Carol) + K2 (Carol global).
Expert with expert_access(scope='specialist', org='acme', specialist='bob-finance')	K1 (Acme × Bob) + K2 (Bob global). Cannot see Carol's KB.
Expert with expert_access(scope='org', org='acme')	All K1 for Acme (every Specialist) + K2 for every Specialist assigned to Acme.
AM for Acme	Same as scope='org' on Acme (existing AM rules).
SuperAdmin	Everything.

The controller uses KbAccessService.listVisibleDocuments(expertId, { orgId?, specialistId? }) — no controller-level ACL branching.

RAG implications (the retrieval shape)

The KB ACL decision determines the retrieval-pipeline shape:

• K1 → per-(Org × Specialist) embedding collection. One collection per pair. The agent runtime for (Acme, Carol-KYC) queries exactly that collection. Clean isolation, no leak risk at the vector layer.
• K2 → per-Specialist global collection. One collection per Specialist, shared across all the Orgs that Specialist is assigned to. The agent runtime queries (per-pair) ∪ (global-for-specialist).
• K3 → would have required a third collection-per-Org retrieved via a third query path. Deferring K3 means the agent runtime config is exactly two collections per agent. This is the right minimum.

When K3-as-K1-fanout is needed: at upload, materialize association rows for every active Specialist in the Org. The retrieval pipeline doesn't change — the K1 collections already contain the doc per Specialist.

Review gating (2026-06-01). Documents in these collections are not all Specialist-visible. The KB Research & Synthesis agent (#1279) adds a review state to org_documents.status, and retrieval serves only approved (or legacy NULL) documents — pending_review / rejected / superseded agent-synthesized drafts are excluded from the retrieval shape described above until a human approves them.

Open questions for follow-up implementation

These are intentionally left for the sub-issues to resolve rather than baked into the ADR:

1. R2 / Cloudflare Storage vs. pgvector for embeddings storage — orthogonal to the ACL decision; deferred to the RAG integration thread (#291/#315).
2. Chunking strategy + embedding model choice — downstream of the KB shape decision.
3. Soft-delete vs. hard-delete retention policy — recommended default soft-delete with 90-day purge; finalized in #541a.
4. K2 authoring UX — who can write to a Specialist's global library? Recommend SuperAdmin only at launch. Finalized in #541b.
5. K3 as K1 fanout — exact UX and migration ergonomics if a Specialist is later added to an Org with existing "shared" docs. Finalized only when #541d unblocks.

Follow-up implementation issues

Filed concurrently with this ADR:

• #541a [P1][feat] KB schema (K1 + K2) — entity, migration, controller scaffolding. Depends on #537. Lands the table, migration, KbAccessService, and the bare GET /kb + POST /kb endpoints. No upload UX. No RAG wiring.
• #541b [P1][feat] KB upload UX in /ops/clients/:orgId/specialists/:specialistId. SuperAdmin/AM-driven upload, attaches to K1; SuperAdmin-only upload to K2.
• #541c [P1][feat] KB retrieval wiring into agent runtime. The agent loads (per-pair) ∪ (global-for-specialist) collections for context retrieval on every turn.
• #541d [P2][spike] K3 cross-Specialist KB. Parked until a real cross-Specialist client document appears. The K1-fanout path is documented above and should be the implementation if/when this unblocks.

Out of scope (this ADR)

• RAGFlow vs. pgvector vs. another vector backend. The ADR is storage-agnostic — any backend that can index by (org_id, specialist_id) works.
• Embedding model selection.
• Search ranking / hybrid retrieval.
• Client-portal KB browsing UX. The ADR enables it but doesn't design it.

Dependencies

• Reads from: ADR-007 (the expert_access primitive this consumes).
• Reads from: #537 (expert_access implementation — must land before #541a).
• Blocks: any future KB implementation issue. #541a–d are the children.
• Related: #296 (onboarding business-context upload — will use whatever this decides), #291 / #315 (legacy RAGFlow infrastructure spike issues — superseded by Haystack/ADR-024; ACL decision orthogonal to backend choice).

Acceptance

• Decision recorded: K1 + K2, defer K3.
• Rejected alternatives documented.
• Schema sketch (K1 + K2 in the same table, sentinel-org for K2) ready to lift into a real PR.
• Access pattern walkthrough for all five caller types.
• RAG retrieval shape implied by the ACL choice.
• Follow-up issues #541a–d filed with concrete acceptance criteria.
• Approved by Paul AND Eusden (recorded in #541 thread).

Addendum 2026-05-28 (HISTORICAL): RAGflow shared-tenant fallback — superseded by ADR-024

The original 2026-05-28 addendum described a RagflowClientFactory.getCredentialConfig env-var fallback (RAGFLOW_DEFAULT_BASE_URL, RAGFLOW_DEFAULT_API_KEY, etc.) for unblocking dev KB endpoints without per-org RAGflow provisioning.

This addendum is historical. Post-ADR-024 (Accepted 2026-06-02), the RAGflow stack and the RAGFLOW_DEFAULT_* env vars have been removed (#1448, #1451, #1452, #1456). Dev environments now use Haystack/Hayhooks per ADR-024 + ADR-025. See docs/runbooks/haystack-deploy.md for the current dev provisioning path.

The K1+K2 ACL decision above is unaffected: the access-check primitive is enforced in humanwork application code on kb_documents / expert_access, independent of the vector store.