P4.1 — Eval Gate Wiring: Field-Level Design
Design document. 2026-07-08. Used as the basis for writing the implementation plan. scope = minimal gate-able (user's call). Based on directly-verified facts:
config-publish.service.ts/config-asset-version.entity.ts/agent/evals/domain/harness.py/docs/decisions/035-eval-sandbox-and-golden-sets.md. All paths are repo-relative.
Three settled decisions (this design conforms to them)
- golden scenario storage = new standalone
EvalGoldenScenariotable (not stuffed into the config-asset foundation). The source of truth lives in the DB → a derived copy is synced to the Langfuse dataset (same pattern as P2.4 golden_answer→Haystack: "keep the source of truth in the foundation + sync a derived copy externally"). - scope = minimal gate-able: EvalRun table + EvalGoldenScenario storage + eval worker + T3 transition + hallucination single-axis hard gate (other axes advisory) + gate wiring + ADR-035 reconsideration + closing OD-13/D4.
- Cross-service coordination = reuse the agent harness: the API worker (consuming
publish_status='publishing') → triggers the agent side's existingharness.py(run_case/judge/aggregate) → collects scores → API decides the gate + writes EvalRun. The new agent endpoint goes through a paired PR.
Non-goals (not in this phase, explicitly deferred)
- Manual judge validation of 10-15 cases (ADR-035 open item, needs human/analyst annotation anchors) — the gate trusts the existing judge for now; manual validation tightens this later.
- Multi-axis noise-band regression rules (comparing scores across runs to judge regression; Langfuse has no built-in primitive for this, master plan §4 P4.1 boundary (a)) — this phase only does the hallucination single-axis absolute gate, not "compare against a baseline run to judge regression."
- Full Langfuse dashboard suite / experiments UI — this phase's Langfuse dataset sync is optional advisory (see DECISION POINT D-3); the first version of gate decisions uses harness scores directly and doesn't depend on Langfuse being online.
- Tightening advisory axes into hard gates — ADR-035 §5 "advise until confident, then tighten"; this phase only gates hallucination.
1. EvalGoldenScenario table (new)
Isolation ownership (ADR-020 matrix): platform-level (platform-global), org_id IS NULL. A golden scenario is a platform evaluation set that doesn't belong to any client — no org×specialist isolation applies. This is this table's row in the ADR-020 matrix; the entity docstring must explicitly declare it (otherwise it triggers the "new entity with no docstring naming its row" code-review block). Reads have no RLS specialist GUC dependency; writes are SuperAdmin-only (managing the evaluation set is a platform-ops function).
The existing 3 JSON datasets (agent/evals/domain/golden/{ecommerce_ops,marketing_content,policy_behaviors}.json) are organized role-file style: {role, description, cases[]}. The table design is one row per case (scenario = case), with role as a column, to make filtering by role and incrementally adding cases easy.
TABLE eval_golden_scenarios
id uuid PK default gen_random_uuid()
role text NOT NULL -- "ecommerce_ops" | "marketing_content" | "policy_behaviors" | ...
case_key text NOT NULL -- the original JSON's case.id, e.g. "mkt-001-supplement-promo-compliance"
question text NOT NULL
golden_answer text NOT NULL
must_include jsonb NOT NULL default '[]' -- string[]
must_not_include jsonb NOT NULL default '[]' -- string[]
context text NULL -- the provided-knowledge block for grounded cases (injection mode)
-- seeded-KB mode fields (#3873, eco-007+ goes through production retrieval):
kb_gold_docs jsonb NULL -- string[] (expected-hit docs)
kb_top_k int NULL
kb_dir text NULL -- KB fixture directory name (seed use, not an absolute path)
suite text NULL -- special-suite marker like "policy" (run.py's role-file vs suite distinction)
enabled boolean NOT NULL default true -- soft-disable a case without deleting it
created_at timestamptz NOT NULL default now()
updated_at timestamptz NOT NULL default now()
CONSTRAINT eval_golden_role_case_uq UNIQUE (role, case_key) -- idempotent seed / upsert
Migration (expand-contract, following 1806700000000-CreateSpecialistConfigAssets.ts): CREATE TABLE IF NOT EXISTS + CHECK constraint (role/case_key/question/golden_answer NOT NULL is already covered by column declarations; add CHECK char_length(role) between 1 and 64). No FK (platform-level, doesn't reference org/specialist).
Migrating the 3 datasets into the table = a one-time seed migration (not an importer). Rationale: golden data is platform-level, low-change-frequency, maintained by engineering/analysts; a single seed migration reads the 3 JSON files and does an idempotent INSERT ... ON CONFLICT (role, case_key) DO UPDATE load — no runtime importer UI needed. DECISION POINT D-2: if analysts need to self-serve add cases in the future, add an importer then (YAGNI for now). The JSON files remain as the fixture source for the agent harness + a seed source (after the source of truth moves to the DB, the harness receives scenarios from the API endpoint, see §4; JSON degrades to a seed/offline fixture).
2. EvalRun table (new — the target for eval_run_id)
config-asset-version.entity.ts:72-75's eval_run_id uuid nullable (a dangling column, no FK) points to this table.
TABLE eval_runs
id uuid PK default gen_random_uuid()
version_id uuid NOT NULL -- FK → specialist_config_versions.id
asset_id uuid NOT NULL -- redundant, for convenient per-asset lookup (= version.assetId)
org_id uuid NOT NULL -- the org of the published asset (publishing is an org-scoped action; golden data is itself platform-level, but "the eval run for this publish" belongs to that org for audit/billing purposes)
status text NOT NULL -- "running" | "passed" | "failed" | "error"
gate_verdict text NULL -- "pass" | "block" (result of the hallucination single-axis verdict)
scenario_snapshot jsonb NOT NULL -- snapshot of the scenario id set run this time (which cases, what version) — reproducible
scores jsonb NOT NULL default '{}' -- per-axis: {hallucination:{failed:int,cases:[...]}, correctness_avg:float, alignment_avg:float, advisory:{...}}
hallucination_failures jsonb NOT NULL default '[]' -- case_key[] that triggered the hard gate (the direct basis for the gate verdict, kept redundantly for query/audit convenience)
langfuse_experiment_run_id text NULL -- Langfuse run reference (advisory, nullable — see D-3)
error_detail text NULL -- reason when status=error
started_at timestamptz NOT NULL default now()
finished_at timestamptz NULL
CONSTRAINT eval_run_version_fk FOREIGN KEY (version_id)
REFERENCES specialist_config_versions(id) -- NOT VALID then VALIDATE (expand-contract)
ADR-020 ownership: eval_runs is owned by org_id (row-5 audit class: audit follows the org of the published asset). The entity docstring declares this row. The version_id FK completes the referential integrity for the eval_run_id dangling column (after T2 succeeds in the publish state machine, version.evalRunId = run.id is backfilled).
Where the gate verdict is stored: gate_verdict (pass/block) + hallucination_failures[] (the basis for the decision) + scores (all axes; advisory axes are recorded too but don't affect the verdict). Audit chain: the config.publish.blocked audit payload references evalRunId (replacing the current evalGate:"not_wired").
3. Wiring the gate into the publish state machine
Current state (config-publish.service.ts:101-172): T1 (draft→publishing atomic claim :118) + T2 (publishing→published :129) + pointer switch (:142) are synchronous pass-through in the same tx, with audit evalGate:"not_wired" (:158-163).
Change to an async gate flow:
requestPublish:
[tx1] T1: claim draft→publishing (atomic :118 semantics preserved)
+ sibling-publishing check (see below)
+ create eval_runs row status="running"
[tx1 commit]
[post-commit] enqueue eval-gate job (BullMQ, jobId = version_id, idempotent)
return { status: "publishing", evalRunId } -- caller sees publishing, no longer synchronously published
eval-gate worker (consumes publishing):
1. Load the role associated with this version → fetch EvalGoldenScenario (enabled cases for that role)
2. Call agent /eval/run (§4) → collect per-case CaseResult + aggregate
3. Decide the gate: any case with failure_class=="hallucination" → BLOCK; otherwise PASS
(other axes correctness/alignment/violations are recorded into scores as advisory, don't affect the verdict)
4. Write back to eval_runs: status, gate_verdict, scores, hallucination_failures, finished_at
5. [tx2] branch:
PASS → T2: publishing→published + pointer switch + version.evalRunId=run.id
audit config.publish { evalRunId, evalGate:"passed" }
BLOCK → T3: publishing→blocked + version.evalRunId=run.id
audit config.publish.blocked { evalRunId, evalGate:"blocked", hallucination_failures }
[tx2 commit]
[post-commit] keep the existing golden-answer-sync enqueue (PASS branch only, after successful publish)
Sibling-publishing check (docstring :47-51 closure point): under the synchronous pass-through, "at most one publishing version per asset" was vacuously satisfied (it never stayed in publishing). After going async, before the T1 claim there must be an explicit query SELECT 1 FROM versions WHERE asset_id=$1 AND publish_status='publishing' AND id<>$2; a hit → 409 (a sibling is already running eval). Backstop: a PG partial unique index CREATE UNIQUE INDEX ... ON specialist_config_versions (asset_id) WHERE publish_status='publishing' (required by design spec §5.2 :245, added this phase).
Eval doesn't block live chat (ADR-019): the old published pointer keeps serving while eval runs; only a successful T2 switches the pointer. A blocked version never becomes live.
T3 transition code (doesn't exist yet, built this phase): an atomic publishing→blocked update (mirroring T2's update({id, publishStatus:"publishing"}, {publishStatus:"blocked"})). The only exit from blocked remains the existing overridePublish (T4, requires publish_note; currently vacuous because blocked is unreachable — this phase makes it genuinely reachable).
evalGate audit values: "not_wired" → "passed" | "blocked" ("running" isn't recorded in audit, only in eval_runs.status).
4. Cross-service contract (paired agent PR)
New agent endpoint (agent/CLAUDE.md hard constraint → paired PR):
POST /eval/run
auth: X-Agent-Secret (AGENT_SERVICE_SECRET, reuses existing agent auth, see D-4)
request:
{
"role": "marketing_content",
"scenarios": [ -- API fetches from EvalGoldenScenario, passes to agent (source of truth is in the API, agent doesn't read the DB)
{ "id": "<eval_golden_scenario.id>",
"case_key": "mkt-001-...",
"question": "...",
"golden_answer": "...",
"must_include": [...],
"must_not_include": [...],
"context": null, -- injection mode
"kb_gold_docs": null, "kb_top_k": null, "kb_dir": null }
]
}
response:
{
"role": "marketing_content",
"cases": [ CaseResult.as_dict() ], -- reuses the existing dataclass: id/correctness/verdict/
-- missing_required/forbidden_present/items/violations/
-- failure_class/unsupported_claims/rationale/...
"aggregate": { ... }, -- reuses the existing aggregate() output
"langfuse_experiment_run_id": "..." -- non-null if the agent side ran run_experiment (D-3), else null
}
Reuse: the endpoint handler is a thin wrapper around the existing harness.run_case (per case) + harness.aggregate. No re-implementation of judge/aggregate. The source of truth doesn't move: the API fetches scenarios from the EvalGoldenScenario table and passes them to the agent (the agent has no DB access, consistent with the service boundary); the JSON fixtures degrade to offline/seed use.
API worker → EvalRun mapping: cases[].failure_class=="hallucination" is aggregated into hallucination_failures[]; aggregate plus each case's correctness/alignment go into scores; gate_verdict = pass if hallucination_failures is empty, otherwise block.
Where the Langfuse run_experiment runs: the Python SDK API → agent side (the harness is Python). But Langfuse isn't required for the minimal gate-able version (see D-3): the first version of the API worker decides the gate directly from the CaseResult returned by /eval/run; the Langfuse dataset sync + run_experiment are added later as an async advisory dashboard, and langfuse_experiment_run_id can be null.
5. Hallucination hard-gate decision
The harness already has built-in hallucination signals (harness.py CaseResult):
failure_class: "retrieval_miss" | "hallucination" | None(:191) — in seeded-KB mode,judge_groundedness(RAGAS-faithfulness style, :285) computesunsupported_claims, and the caller attributes it to hallucination vs retrieval_miss.- injection/
contextmode (no KB retrieval): hallucination shows up asviolations(amust_not_includehit; the judge outputs aviolationsmap :233) — e.g. goldenmkt-003's"a refund window other than 30 days"/"an invented policy detail not in the provided knowledge".
Hard-gate threshold (defined this phase): a case triggers the hallucination hard gate if and only if:
failure_class == "hallucination"(seeded-KB mode, groundedness verdict has unsupported_claims), or- in injection mode, a hallucination-class violation hits within
must_not_include(violations[k]==true, where k's semantics are "fabricated/invented/untrue").
DECISION POINT D-1: how does injection mode distinguish "hallucination-class must_not_include" vs "ordinary must_not_include" (e.g. mkt-001's "medical claim" is a compliance violation, not a hallucination)? This phase's pragmatic default = any violations hit goes into advisory scores.violations, but only failure_class=="hallucination" (the groundedness path) triggers BLOCK. That is: for seeded-KB grounded cases, groundedness is the sole authoritative signal for the hard gate; for injection cases, violations are all recorded as advisory and never block. Rationale: groundedness is the hallucination metric explicitly defined in ADR-035 §4, with unambiguous semantics; must_not_include mixes compliance/style/hallucination semantics, so hard-blocking on it would cause false positives. This is reversible by the user: if injection-case "fabrication" violations should also hard-block, must_not_include needs a semantic annotation added (which entries are hallucination-class) — that's a golden schema extension, deferred.
Other axes are advisory: correctness (judge alignment 0-1), must_include coverage (items), non-hallucination violations, retrieval_miss — all recorded in scores, none affect gate_verdict.
6. ADR-035 reconsideration diff
ADR-035 is still Proposed (docs/decisions/035-eval-sandbox-and-golden-sets.md:3). Convention: Proposed but not yet Accepted → change its own Status/Decision (not an ADR-024-style superseding ADR, which is used to replace an already-Accepted decision).
:3Status:Proposed (for discussion — not yet accepted)→Accepted (2026-07-08; P4.1 wires the minimal gate).:113Langfuse section:**LangFuse**: **deferred.**→ change to**LangFuse**: **adopted as sync-copy + runner/dashboard layer** (P4.1). Golden truth source stays first-party (EvalGoldenScenario table, ADR-020 platform-global); a derived copy syncs to a Langfuse dataset for run_experiment + trend dashboards. The PII/data-sovereignty objection is addressed by keeping the truth source in-DB — Langfuse holds only a synced copy. SDK v3 run_experiment now supports the CI-gate workflow (master plan §4 P4.1 note 2026-07-07).(retain the "gate the one unambiguous failure (hallucination)" hybrid rule :96, which this phase is exactly implementing).- Closing OD-13/D4:
docs/.../OPEN_DECISIONS.md:53-65(OD-13 golden-set contribution format) — fold the conclusion into ADR-035 (Option 2 / Option B is already the established format, as evidenced by the existing 3 datasets), delete the OPEN_DECISIONS OD-13 entry. Mark master plan:199D4 as closed.
7. DECISION POINT (reversible defaults made on the user's behalf, user can override)
- D-1 hallucination gate signal: only groundedness (
failure_class=="hallucination") triggers BLOCK; injection-casemust_not_includeviolations are all recorded as advisory and never block (to avoid false positives on compliance/style violations). Cost to reverse: add hallucination-class annotations to the golden schema's must_not_include.- ⚠️ Trigger-surface data (directly verified 2026-07-09; must-read before confirming D-1 while the user is away): of the existing 21 golden cases, only 3 are grounded/seeded-KB (able to go through groundedness → trigger the hard gate): ecommerce_ops 2 + marketing_content 1 + policy_behaviors 0. The remaining 18 are injection-only (must_not_include only, all advisory under D-1, never block). So this version of the hard gate has an actual trigger surface of 3/21 on the existing golden set. But after verifying the content of policy_behaviors' must_not_include, confirmed: these are behavioral/semantic descriptions (e.g. "guessing the topic and shipping a generic deck without asking", "asking unnecessary clarifying questions when they were already given"), not deterministic string blocklists — hard-blocking on substring matching would cause massive false positives, and pol-004/005's "shouldn't over-clarify" is the opposite direction from pol-001's "shouldn't under-clarify" — it's a quality judgment the judge makes in context. So D-1 (not hard-coding must_not_include as a gate) is technically correct: this batch of golden data was designed from the start for LLM-as-judge multi-axis advisory scoring; "advisory first, then tighten" is explicit design intent from ADR-035 §5, not a defect. The value of this P4.1 phase = wiring up the gate infrastructure (EvalRun/worker/T3/async split) so the gate exists and can be tightened; the real fix for the narrow hard-gate trigger surface is adding grounded golden cases (deferred), not hard-coding the semantic rubric. When the user returns, if a wider trigger surface is wanted → add grounded cases or annotate must_not_include with hallucination-class tags (both are golden-side extensions, no change to gate architecture).
- D-2 golden table-migration approach: a one-time seed migration (
INSERT ON CONFLICT), not a runtime importer (YAGNI; add one later if analysts need to self-serve add cases). JSON remains as the seed/offline fixture source. - D-3 whether Langfuse is required for the first version: not required. Under the minimal gate-able scope, the API worker decides the gate directly from
/eval/run's CaseResult; the Langfuse dataset sync + run_experiment are added later as an async advisory dashboard, andlangfuse_experiment_run_idcan be null. To reverse: if gate decisions should go through a Langfuse experiment, the agent side needs to bring in the Langfuse SDK and the API side needs to reference the experiment run (heavier). - D-4 agent /eval/run auth: reuse the existing
AGENT_SERVICE_SECRET(X-Agent-Secret), no new token minted. Consistent with existing /chat auth. - D-5 EvalRun.org_id ownership: an eval run is owned by the org of the published asset (audit/billing follows the publish action), even though the golden scenario itself is platform-level.
8. Implementation task breakdown (TDD granularity, marked API-only / paired agent PR / dependencies)
| # | Task | Type | Dependency |
|---|---|---|---|
| T1 | EvalGoldenScenario entity + migration (CREATE TABLE + CHECK + unique(role,case_key) + partial unique index on versions publishing) + ADR-020 docstring | API-only | — |
| T2 | Seed migration: 3 golden JSON files → eval_golden_scenarios (idempotent INSERT ON CONFLICT) | API-only | T1 |
| T3 | EvalRun entity + migration (FK version_id NOT VALID→VALIDATE) + ADR-020 docstring | API-only | — |
| T4 | Agent POST /eval/run endpoint (thin wrapper over run_case/aggregate + request/response schema) + evals regression tests | paired agent PR | — |
| T5 | API AgentClient.runEval() (calls /eval/run, X-Agent-Secret, fail-open) | API-only | T4 |
| T6 | eval-gate BullMQ worker (consumes publishing → runEval → decides gate → writes back EvalRun) + QueueMetricsCollector registration | API-only | T3,T5 |
| T7 | config-publish.service async split: T1 claim + sibling check + create running EvalRun + enqueue; remove the synchronous T2 pass-through | API-only | T6 |
| T8 | T2/T3 branching (in the worker: pass→published+pointer / block→blocked) + real evalGate audit values + version.evalRunId backfill | API-only | T7 |
| T9 | overridePublish reachability verification (blocked→published + publish_note, currently vacuous → genuinely reachable) | API-only | T8 |
| T10 | ADR-035 reconsideration diff (Status + Langfuse section) + closing OD-13/D4 (delete OPEN_DECISIONS entry) + config-assets/CLAUDE.md contract update (§5.3 from not_wired to a real gate) | Documentation | T8 |
| T11 | (Deferred / optional) Langfuse dataset sync + run_experiment advisory dashboard | paired | D-3 |
Critical ordering: T1/T3 start in parallel → T4 (agent PR) can run in parallel with T1-3 → T5→T6→T7→T8→T9→T10. T4 is the only paired agent PR (must merge first or coordinate wiring with the API side).