Skip to main content

P3.5 — Red-Line Post-Draft Check Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development. Steps use checkbox (- [ ]) syntax.

Goal: After the agent produces a draft, before NestJS persists it, check the drafted reply against the Specialist's published skill.redLines + soul boundaries (never/escalate); on a hit, force the draft to Expert review (bar auto-send) with escalationReason: "red_line_violation" and stamp the fired rules on the draft metadata for Expert-workspace rendering. Runs on ALL drafts (flag-independent of the assembler), behind its own default-ON flag.

Architecture: The "runtime half" of red-lines (the storage half — the redLines/boundaries schema fields — shipped in P2.2/P3.2/P3.4). Decision D-5: NestJS persist-side (redLines + boundaries are config-assets data, the queue flag is NestJS-local; the agent just returns the draft — no agent change). The check slots into the existing scrub-then-save chokepoint (same pattern as #3500 scrubAgentReplyPaths), OUTSIDE the if (assembling) block, so it protects every draft regardless of prompt_assembler_enabled (which defaults OFF — a red-line check gated on it would silently protect almost nobody).

⚠️ DECISION POINT (safe default during user absence — reversible): How does the check decide a draft "violates" a red-line? Red-line rule is free operator text (e.g. "never promise refunds"). Deciding true violation is a semantic problem (like P3.2 trigger), but the failure mode is INVERTED: a red-line is a safety boundary, so a silent miss (violated but not flagged) is far more dangerous than a false alarm. This plan takes conservative over-flag: if a red-line rule's keywords/topics appear in the draft, force Expert review — do NOT try to auto-adjudicate actual violation; leave that judgment to the Expert. Rationale: (1) it matches the platform's always-HITL default (threshold=101) — the red-line's value is "ensure a draft touching a sensitive boundary always gets human review", not "auto-decide guilt"; (2) unlike P2.4/P3.2 where a keyword MISS means lost value (a feature silently failing), here a miss means DANGER, so conservative over-flag is the correct direction — the opposite trade-off; (3) pure NestJS, deterministic, fast, fail-open — no LLM latency/cost/non-determinism on the pre-persist critical path. If the user wants semantic/LLM violation judgment, the matcher is isolated behind a pure function (same seam idea as P3.2) and can be upgraded; the conservative keyword version ships as the safe reversible baseline. severity: "block" vs "flag" both escalate; block additionally stamps a higher-risk marker.

Tech Stack: NestJS 11 (red-line matcher util, ConfigAssetsService boundaries reader, ConversationsService persist-site wiring, ExpertQueueService escalation), Jest.

Scope note: Long-lived branch feat/specialist-value-output-p2plus. Relatively independent (Explore: persist site is common, flag-independent). Commit per sub-task. This is the LAST Phase 3 task.

Verified reality (Explore) shaping this plan:

  • Persist chokepoint EXISTS, OUTSIDE if (assembling): main dispatch sanitizeBidiChars (:3419) → scrubAgentReplyPaths (:3423) → msgRepo.save (:3437). The check slots between the scrub and the save. msgRepo.save (:3437) runs BEFORE the auto-send decision (resolveAutoReplyPolicy ~:3688, autoResponded ~:3745), so a hit computed at persist time can set a var the L3745 gate reads. regenerateDraft chokepoint: after ~:6697 scrub, before ~:6732 save (no auto-send path there — pure flag/metadata stamp).
  • redLines NOT in scope at persist — SkillEvaluation drops them (skill-trigger.service.ts returns only {skillLayerText, sourceRefs, abstain}), and skillEval is if (assembling)-block-scoped. Must re-fetch via getPublishedSkills (returns full SkillContentT incl. redLines; fail-open, dual-scope, flag-independent).
  • soul boundaries STRUCTURED in schema (soul.schema.ts: boundaries: {never: string[], escalate: string[]}) but only prose reaches runtime (getPublishedSoul returns rendered soulMd). Must BUILD a structured reader.
  • Reuse verificationRequired to hold the draft (L3751 tightening-only AND-term bars auto-send) — no new hold mechanism; the queue item is already created pending/held.
  • escalationReason is an unconstrained string — add "red_line_violation" with no migration; insert a branch in the escalation cascade (~:3901) at highest priority.
  • Model the check structure on scrubAgentReplyPaths (private async helper, own flag, pure inner fn, fail-open, stable log event) — but it CLASSIFIES (returns hits), doesn't rewrite.

Boundary (P3.5 does NOT do):

  • Does NOT auto-adjudicate true violation (conservative over-flag — DECISION POINT).
  • Does NOT rewrite/redact the draft (only flags + holds).
  • Does NOT add an LLM call on the persist path.
  • Does NOT gate on prompt_assembler_enabled (own flag, runs on all drafts).
  • Does NOT change the agent (NestJS persist-side only).
  • Does NOT expose red-line detail to clients (metadata allowlist keeps it Expert-internal).

Pre-existing baseline: verify against branch base 490517fb. No agent change → agent evals unaffected (but re-run to confirm nothing shifted).

File Structure​

Create:

  • api/src/conversations/red-line-check.util.ts — pure checkRedLines(draft, rules): RedLineResult. Deterministic keyword/topic substring match of each rule's significant terms against the draft. Never-throws. Defines the matcher seam (upgradeable to semantic later). RedLineRule = { id, rule, severity, source: "skill"|"soul" }.
  • api/src/conversations/red-line-check.util.spec.ts — pure unit test.

Modify:

  • api/src/config-assets/config-assets.service.ts — add getPublishedSoulBoundaries({orgId, specialistId}): Promise<{ never: string[]; escalate: string[] } | null> (structured boundaries reader; same dual-scope + fail-open as getPublishedSoul).
  • api/src/conversations/conversations.service.ts:
    • Add private async resolveRedLineRules(orgId, specialistId): Promise<RedLineRule[]> — re-fetch published skills' redLines (getPublishedSkills) + soul boundaries (getPublishedSoulBoundaries), flatten into RedLineRule[] (skill redLines keep their severity; soul never→severity block, escalate→severity flag). Fail-open ([]).
    • Add private async checkDraftRedLines(draft, ctx): Promise<RedLineHit> — own flag (red_line_check_enabled, default ON), resolve rules, run pure checkRedLines, return { barsAutoSend, violations }. Fail-open. Modeled on scrubAgentReplyPaths.
    • Main dispatch: call it after the scrub, before save; on hit set a redLineBars bool + stamp metadata.redLines; add && !redLineBars to the auto-send conjunction (:3745) OR set verificationRequired; add the red_line_violation escalation branch (:3901); pass to createQueueItem.
    • Regenerate: same check + metadata stamp (no auto-send path — flag only).
  • api/src/conversations/conversations.module.ts — no new provider (checks are ConversationsService methods; util is a pure import).
  • api/src/conversations/CLAUDE.md, api/src/config-assets/CLAUDE.md — document.

Task 1: pure red-line matcher (red-line-check.util.ts)​

Files: Create api/src/conversations/red-line-check.util.ts + spec.

Precedent: evaluateSkillTriggers (skill-trigger.util.ts) + classifyTask (task-classification.util.ts) — deterministic, total, never-throws substring matching. Mirror.

Design: conservative over-flag. For each rule, extract significant terms from rule.rule (the operator text) and check if they appear in the draft (case-insensitive substring). A term hit → the rule "fires" (flag for Expert; NOT an assertion of true violation). Significant-term extraction: lowercase, split on non-word, drop stopwords + short tokens (<4 chars), require ≥1 significant term to co-occur. (Simple + conservative — err toward firing.)

  • Step 1: Write the failing test
import { checkRedLines, type RedLineRule } from "./red-line-check.util";

const rule = (over: Partial<RedLineRule> = {}): RedLineRule => ({
  id: "no-refund-promise", rule: "never promise refunds to the customer",
  severity: "block", source: "skill", ...over,
});

describe("checkRedLines (conservative over-flag)", () => {
  it("fires when a rule's significant terms appear in the draft", () => {
    const r = checkRedLines("Sure, we can promise a full refund today.", [rule()]);
    expect(r.violations.map((v) => v.id)).toContain("no-refund-promise");
    expect(r.barsAutoSend).toBe(true); // any hit bars auto-send
  });

  it("does not fire when no significant term appears", () => {
    const r = checkRedLines("Thanks for reaching out, how can I help?", [rule()]);
    expect(r.violations).toEqual([]);
    expect(r.barsAutoSend).toBe(false);
  });

  it("block and flag severities both fire; block is marked higher-risk", () => {
    const r = checkRedLines("here is a refund", [rule({ severity: "flag" }), rule({ id: "b", severity: "block" })]);
    expect(r.violations.length).toBe(2);
    expect(r.hasBlock).toBe(true);
  });

  it("carries source (skill vs soul) on each violation", () => {
    const r = checkRedLines("a refund", [rule({ source: "soul" })]);
    expect(r.violations[0].source).toBe("soul");
  });

  it("is total — never throws on empty/garbage", () => {
    expect(checkRedLines("", []).violations).toEqual([]);
    expect(checkRedLines("x", [rule({ rule: "" })]).barsAutoSend).toBe(false);
  });
});
  • Step 2: Implement
/**
 * Red-line post-draft check (P3.5). Pure, deterministic, total — never throws.
 * CONSERVATIVE OVER-FLAG: if a red-line rule's significant terms appear in the
 * draft, the rule FIRES (→ Expert review). This does NOT assert the draft
 * truly violates the rule — a red-line is a safety boundary where a silent
 * miss is far worse than a false alarm, and the platform is always-HITL by
 * default, so we escalate anything touching a boundary and let the Expert
 * adjudicate. Matcher SEAM: a semantic/LLM matcher could replace `fires()`
 * with the same RedLineRule→hit contract; this keyword version is the safe,
 * deterministic, no-LLM-on-the-critical-path baseline (P3.5 DECISION POINT).
 */
export type RedLineSeverity = "block" | "flag";
export interface RedLineRule {
  id: string;
  rule: string;
  severity: RedLineSeverity;
  source: "skill" | "soul";
}
export interface RedLineViolation {
  id: string;
  severity: RedLineSeverity;
  source: "skill" | "soul";
  matchedTerms: string[];
}
export interface RedLineResult {
  violations: RedLineViolation[];
  barsAutoSend: boolean; // any violation → true
  hasBlock: boolean;     // any block-severity violation
}

const STOPWORDS = new Set([
  "never","always","must","should","the","a","an","to","of","for","and","or",
  "do","not","dont","with","that","this","when","from","your","our","you",
  "will","can","are","is","be","in","on","at","as","it","if","any","all",
]);
const MIN_TERM_LEN = 4;

function significantTerms(rule: string): string[] {
  return (rule.toLowerCase().match(/[a-z0-9]+/g) ?? [])
    .filter((t) => t.length >= MIN_TERM_LEN && !STOPWORDS.has(t));
}

export function checkRedLines(draft: string, rules: RedLineRule[]): RedLineResult {
  const hay = (typeof draft === "string" ? draft : "").toLowerCase();
  const list = Array.isArray(rules) ? rules : [];
  const violations: RedLineViolation[] = [];
  for (const r of list) {
    const terms = significantTerms(r.rule ?? "");
    if (terms.length === 0) continue; // a rule with no significant terms can't fire
    const matched = terms.filter((t) => hay.includes(t));
    if (matched.length > 0) {
      violations.push({ id: r.id, severity: r.severity, source: r.source, matchedTerms: matched.slice(0, 8) });
    }
  }
  return {
    violations,
    barsAutoSend: violations.length > 0,
    hasBlock: violations.some((v) => v.severity === "block"),
  };
}
  • Step 3: Run → PASS (5). npx tsc --noEmit clean.
  • Step 4: Commit feat(conversations): pure conservative red-line matcher (over-flag, deterministic) (P3.5)

Task 2: structured soul-boundaries reader​

Files: Modify api/src/config-assets/config-assets.service.ts + spec.

Context: getPublishedSoul (P3.4) returns rendered soulMd; the structured boundaries.never[]/escalate[] (in SoulContent) are flattened into markdown. P3.5 needs them structured.

  • Step 1: Read getPublishedSoul (config-assets.service.ts:442+), soul.schema.ts boundaries field ({never: string[], escalate: string[]}).

  • Step 2: Failing test — getPublishedSoulBoundaries returns {never, escalate} for a published soul; null on no-soul / parse-fail / error (fail-open). Mirror the getPublishedSoul test setup.

  • Step 3: Implement (parallels getPublishedSoul but returns the structured boundaries):

async getPublishedSoulBoundaries(params: {
  orgId: string; specialistId: string;
}): Promise<{ never: string[]; escalate: string[] } | null> {
  if (!params.orgId || !params.specialistId) return null;
  try {
    const asset = await this.assetRepo.findOne({
      where: { assetType: "soul", scope: "org_instance", orgId: params.orgId,
               specialistId: params.specialistId, slug: SOUL_ASSET_SLUG, archivedAt: IsNull() },
    });
    if (!asset?.publishedVersionId) return null;
    const version = await this.versionRepo.findOneBy({ id: asset.publishedVersionId });
    if (!version) return null;
    const parsed = SoulContent.safeParse(version.content);
    if (!parsed.success) return null;
    return { never: parsed.data.boundaries?.never ?? [], escalate: parsed.data.boundaries?.escalate ?? [] };
  } catch (err) {
    this.logger.warn(`published-soul-boundaries lookup failed for org=${params.orgId} specialist=${params.specialistId}: ${err instanceof Error ? err.message : String(err)}`);
    return null;
  }
}

Confirm the real SoulContent boundaries field path (parsed.data.boundaries.never). If the schema names differ, adapt.

  • Step 4: Run → PASS. npx tsc --noEmit clean.
  • Step 5: Commit feat(config-assets): getPublishedSoulBoundaries structured reader (P3.5)

Task 3: wire the check into both persist sites + escalation + auto-send bar​

Files: Modify api/src/conversations/conversations.service.ts + prompt-assembler-wiring.spec.ts (or a focused red-line spec).

  • Step 1: Read the persist chain (main :3419-3483, regenerate :6694-6732), the auto-send gate (:3745-3757, the !(taskClassification?.verificationRequired ?? false) term), the escalation cascade (:3901-3927), and the createQueueItem call site (~:4084-4096). Confirm getPublishedSkills returns content.redLines and getPublishedSoulBoundaries (Task 2) exists.

  • Step 2: Failing test (wiring): a published skill with a block redLine whose terms appear in the agent's drafted reply → the persisted draft has metadata.redLines with the fired rule AND (if auto-reply were enabled) the draft is NOT auto-sent (verificationRequired/bar). And: no red-line hit → no metadata.redLines, unchanged behavior. Mock getPublishedSkills to return a skill with redLines: [{id, rule, severity: "block"}] and make the agent chat mock reply contain the rule's terms.

  • Step 3: Implement resolveRedLineRules + checkDraftRedLines (private methods, modeled on scrubAgentReplyPaths):

private async resolveRedLineRules(
  orgId: string | undefined, specialistId: string | undefined,
): Promise<RedLineRule[]> {
  if (!orgId || !specialistId) return [];
  const rules: RedLineRule[] = [];
  try {
    const skills = this.configAssets ? await this.configAssets.getPublishedSkills({ orgId, specialistId }) : [];
    for (const s of skills) for (const rl of s.content.redLines ?? [])
      rules.push({ id: `${s.slug}:${rl.id}`, rule: rl.rule, severity: rl.severity, source: "skill" });
    const boundaries = this.configAssets ? await this.configAssets.getPublishedSoulBoundaries({ orgId, specialistId }) : null;
    if (boundaries) {
      for (const n of boundaries.never) rules.push({ id: `soul:never:${rules.length}`, rule: n, severity: "block", source: "soul" });
      for (const e of boundaries.escalate) rules.push({ id: `soul:escalate:${rules.length}`, rule: e, severity: "flag", source: "soul" });
    }
  } catch (err) {
    this.logger.warn(`red-line rule resolution failed (non-fatal): ${(err as Error).message}`);
  }
  return rules;
}

private async checkDraftRedLines(
  draft: string, ctx: { orgId?: string; specialistAssignmentId?: string; specialistId?: string },
): Promise<RedLineResult> {
  const empty: RedLineResult = { violations: [], barsAutoSend: false, hasBlock: false };
  try {
    const flag = await this.featureFlags?.resolve("red_line_check_enabled", {
      orgId: ctx.orgId, specialistAssignmentId: ctx.specialistAssignmentId,
    });
    if (flag && flag.enabled === false) return empty; // default ON: only an explicit disable skips
    const rules = await this.resolveRedLineRules(ctx.orgId, ctx.specialistId);
    if (rules.length === 0) return empty;
    const result = checkRedLines(draft, rules);
    if (result.violations.length > 0) {
      this.logger.warn(`red_line_check event=violation count=${result.violations.length} block=${result.hasBlock} ids=${result.violations.map((v) => v.id).join(",")}`);
    }
    return result;
  } catch (err) {
    this.logger.warn(`red-line check failed (non-fatal): ${(err as Error).message}`);
    return empty; // fail-open: never block the pipeline
  }
}

The flag default-ON convention: mirror scrubAgentReplyPaths — resolve the flag, treat only an explicit enabled === false as off (absent/unset → on). Confirm the exact convention scrubAgentReplyPaths uses and match it.

  • Step 4: Wire main dispatch — after the scrub (~:3426), before save:
    const redLine = await this.checkDraftRedLines(agentReply, {
      orgId: dispatchOrgId, specialistAssignmentId, specialistId: conv.specialistId,
    });

Stamp metadata in the msgRepo.create({... metadata: {...}}) (additive): ...(redLine.violations.length ? { redLines: { violations: redLine.violations, hasBlock: redLine.hasBlock } } : {}). Add && !redLine.barsAutoSend to the auto-send conjunction (:3745). In the escalation cascade (:3901), add a top-priority branch: if (redLine.barsAutoSend) escalationReason = "red_line_violation"; and ensure the queue item is created (it already is, when not auto-responded). Pass classificationSignals = fired rule ids (optional) to createQueueItem. If hasBlock, consider riskLevel bump to high (reuse existing risk handling — confirm the riskLevel var).

  • Step 5: Wire regenerate — after ~:6697 scrub, before ~:6732 save: same checkDraftRedLines (ctx {orgId: conv.orgId, specialistId: conv.specialistId} — note no assignmentId at that site, matches the scrub asymmetry) + the additive metadata.redLines stamp. No auto-send path there.

  • Step 6: Run tests — wiring red-line tests pass; ALL existing conversations/wiring/persona/template tests green (no red-line rules → no behavior change); flag-OFF assembler byte-equality intact (red-line check is independent, runs on both, but a no-rules Specialist sees no change). npx tsc --noEmit + npm run lint clean.

  • Step 7: Commit feat(conversations): red-line post-draft check → bar auto-send + escalate + metadata (P3.5)

Task 4: docs + full verification​

Files: Modify api/src/conversations/CLAUDE.md, api/src/config-assets/CLAUDE.md.

  • Step 1: Docs

    • conversations/CLAUDE.md — new Constraint bullet: P3.5 red-line post-draft check (checkDraftRedLines at both persist sites, after scrub / before save, OUTSIDE if (assembling) — runs on ALL drafts, own flag red_line_check_enabled default ON). Conservative over-flag (pure red-line-check.util.ts keyword match — DECISION POINT: fires on term co-occurrence, does NOT auto-adjudicate; matcher is a seam upgradeable to semantic). Sources: published skills' redLines (re-fetched via getPublishedSkills) + soul boundaries.never(→block)/escalate(→flag) (via new getPublishedSoulBoundaries). Hit → bars auto-send (&& !redLine.barsAutoSend in the L3745 conjunction, reusing the always-HITL hold), escalationReason: "red_line_violation" (unconstrained string, no migration), metadata.redLines stamp (Expert-internal — NOT in the client-view allowlist). Fail-open (never blocks the pipeline). No agent change; no draft rewrite.
    • config-assets/CLAUDE.md — add getPublishedSoulBoundaries next to getPublishedSoul (structured {never, escalate}; same dual-scope + fail-open).

  • Step 2: Self-check no machine paths — grep the two CLAUDE.md → exit 1.

  • Step 3: Full verification

    • cd api && npx tsc --noEmit && npm run lint → clean.
    • npx jest src/conversations/ src/config-assets/ src/feature-flags/ --runInBand → pass; no-rules behavior unchanged; flag-OFF assembler byte-equality intact.
    • cd agent && <venv>/bin/python -m pytest evals/test_prompt_echo_guard.py evals/test_chat_helpers.py evals/test_prompt_assembly_meta.py -q → pass (no agent change — confirm nothing shifted).

  • Step 4: Commit docs docs(P3.5): red-line post-draft check + getPublishedSoulBoundaries + over-flag decision

  • Step 5: No further commit — proceed to Phase-3 P3.5 task-level review + then a WHOLE-Phase-3 final review (P3.1–P3.5). Dispatch code-reviewer: verify the check runs on all drafts (not gated on assembler), fail-open (never blocks the pipeline / never 500s a turn), no-rules Specialists see byte-identical behavior, red-line detail stays Expert-internal (not client-exposed), and the auto-send bar genuinely holds a violating draft.

Self-Review Checklist​

Spec coverage (design §6.5 P3.5, master plan R2.5 runtime half): red-line post-draft check ✅ (Task 1 matcher + Task 3 wire); skill.redLines + soul boundaries sources ✅ (Task 2 boundaries reader + Task 3 resolveRedLineRules); violation → Expert queue flag ✅ (Task 3 escalationReason + verificationRequired/bar); persist-side NestJS, no agent change ✅ (D-5).

Type consistency: RedLineRule/RedLineResult/RedLineViolation (Task 1) consumed by resolveRedLineRules/checkDraftRedLines (Task 3). getPublishedSoulBoundaries (Task 2) → resolveRedLineRules. barsAutoSend → auto-send conjunction; violations → metadata.

Safety net: own flag default ON (explicit-disable-only to skip); fail-open at every level (rule resolution, check, flag resolve → empty result, never blocks the pipeline); no-rules Specialist → empty → zero behavior change; conservative over-flag → misses err toward MORE Expert review (safe for a boundary); metadata Expert-internal (client allowlist). No agent change → no echo-guard risk.

Boundary: no auto-adjudication; no draft rewrite; no LLM on persist path; not gated on assembler; no client exposure.

DECISION POINT surfaced: conservative keyword over-flag is the reversible default (safe baseline for a safety boundary); semantic/LLM matcher is an upgrade behind the pure checkRedLines seam, flagged for user confirmation.

Last updated on by Degen