Skip to main content

P4.2c — Remove correction auto-ingest (flag + direct-ingest branch) + accept ADR-037 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Execute the expand-contract contract step of ADR-037 Decision 3: now that the human-approved proposal flow has shipped (P4.2a producer + P4.2b approval UI), delete the legacy correction_auto_ingest_enabled flag and the entire correction-refinement.processor.ts direct-KB-ingest pipeline it gated, and flip ADR-037 to Accepted.

Architecture: The processor's ONLY job is direct-ingest, so removing "the direct-ingest branch" means removing the whole processor + its BullMQ queue + hourly tick + metrics enrolment. P4.2a's LoopProposalProducerService already owns the same corrections.processed_at IS NULL backlog (as human-approved proposals), so processed_at semantics cleanly converge to "attempted to turn into a proposal". Backend + docs only; zero frontend change (the flag was never rendered in the Ops feature-flag UI).

Tech Stack: NestJS 11, TypeORM 0.3, BullMQ, Jest, ADR markdown.

Context the implementer needs (verified against real code @ 8c2d693d)​

ADR-037 Decision 3 (the contract): "The flag AND the processor's direct-ingest branch are removed once the human-approved proposal flow (master plan P4.2) ships; P4.2's consumer takes over the unprocessed backlog." P4.2a (LoopProposalProducerService) + P4.2b (approval UI) shipped → this is that removal.

Critical distinction: api/src/haystack/jobs/correction-refinement.processor.ts (LEGACY, being removed) ≠ api/src/config-assets/loop-proposal-producer.service.ts (P4.2a, the NEW pipeline, DO NOT TOUCH). Both read corrections WHERE processed_at IS NULL; the new one uses its own distilCorrectionToGolden, NOT the legacy categorize. Removing the legacy one does not affect P4.2a.

Verified deletion scope:

  • Flag correction_auto_ingest_enabled: defined api/src/feature-flags/feature-flags.constants.ts:45-50 (FLAG_KEYS + comment) + :106-107 (DEFAULT_FLAG_VALUES + comment). Golden assertion api/src/feature-flags/feature-flag.service.spec.ts:53 (one line in a full expect(FLAG_KEYS).toEqual([...])). Frontend: NONE — frontend/.../feature-flags/page.tsx FLAG_KEYS (6 keys) + api.ts FeatureFlagKey union never listed it. Zero frontend change.
  • Processor correction-refinement.processor.ts (631 lines): one linear chain process() → _processInContext() → processOneCorrection(), terminal = direct-ingest. No hidden second responsibility (correctionType is only passed through as metadata, never branched on). Integer-delete the file.
  • Queue/tick wiring in api/src/haystack/haystack.module.ts: import (:24-27), registerQueue(CORRECTION_REFINEMENT_QUEUE) (:80-83), provider (:116), implements OnModuleInit + @InjectQueue constructor (:136-140), onModuleInit() hourly tick (:142-157), and the OnModuleInit import (:1).
  • Metrics api/src/common/queue-metrics.collector.ts: constructor @Optional() @InjectQueue("correction-refinement") (:41-43) + injectedQueues entry (:108) + a comment ref (:76).
  • Tests: api/test/correction-refinement.spec.ts (808 lines, all processor behaviour) — integer-delete, BUT migrate the processed_at column exists / defaults NULL schema test (:620-640) — it tests entity/column semantics P4.2a also depends on, not processor behaviour. api/test/llm-callsite-metering.integration.spec.ts: import (:73) + the categorisation meters purpose='classify' it block (:221-255) — delete just that block + import (the file has other callsite tests that stay).
  • The one external symbol reference to the processor is that metering test (Object.create(CorrectionRefinementProcessor.prototype)). All other private methods (categorize/buildRefinementDoc/hashKeyLearning/findExistingRefinement/appendToExistingDocument/normalizeCategorization) are processor-internal (the kb-seeding.service.ts categorize* are same-name-different-thing). Few-shot (correction_fewshot_enabled / learning.retrieveSimilarCorrections) shares NO symbol with the processor.

TWO Chesterton-fence checks that MUST be grep-verified during implementation (do NOT blind-delete):

  1. haystack.module.ts:93-103 LlmModule + FeatureFlagsModule imports — a comment says they're for CorrectionRefinementProcessor. Before removing them, grep whether GoldenAnswerSyncProcessor / ArtifactIngestionProcessor / any other haystack provider injects LlmService / FeatureFlagService. Remove an import ONLY if no surviving provider needs it; otherwise keep it (and drop the now-wrong comment reference).
  2. haystack.module.ts:7,75 Correction entity import + TypeOrmModule.forFeature([...Correction]) — grep whether any surviving haystack service uses the Correction repo. If none, it may be removed; if uncertain, leaving it is harmless (a registered-but-unused entity is not an error). Prefer safe: keep unless clearly the sole consumer was the processor.

Docs to flip (docs-only, mirror P4.1's ADR-035 accept 026695ad): ADR-037 Status + Decision 3 + Verification section; OPEN_DECISIONS.md:115 (CD-30); optional master-plan P4.2 marker; stale "correction-refinement → KB flywheel" comments in auto-correction.listener.ts:11-12 / expert-queue.service.ts:160 / entities.ts:1511 (update the downstream to "loop-proposal producer (P4.2a)"; note auto-correction.listener is the correction PRODUCER — do NOT delete it).

DECISION POINTS (reversible defaults — user may override)​

  • DP-1 — Integer-delete the processor, not gut-and-keep. Its sole responsibility is direct-ingest; a gutted "read backlog, do nothing" tick is pure wasted DB load (P4.2a already reads the same backlog). ADR-037 authorises removing the branch, which is the whole processor. Reversible: git history retains it.
  • DP-2 — Migrate the schema test, don't drop it. The processed_at column test has standalone value (P4.2a depends on the column semantics). Moving it to a surviving spec keeps that guarantee.

Task 1: Remove the processor wiring (queue, tick, metrics) — make it unreferenced​

Files:

  • Modify: api/src/haystack/haystack.module.ts

  • Modify: api/src/common/queue-metrics.collector.ts

  • Step 1: Grep the two Chesterton fences FIRST (decide before editing).

cd api
# Fence 1: does any SURVIVING haystack provider need LlmModule / FeatureFlagsModule?
grep -rn "LlmService\|FeatureFlagService\|featureFlags\|llm:" src/haystack/jobs/golden-answer-sync.processor.ts src/haystack/jobs/artifact-ingestion.processor.ts src/haystack/*.service.ts
# Fence 2: does any SURVIVING haystack file use the Correction repo?
grep -rn "Correction" src/haystack/ | grep -v correction-refinement

Record the result. If a surviving provider injects LlmService/FeatureFlagService, KEEP that module import (only drop the comment naming the processor). If none uses Correction, you may drop it from forFeature; if unsure, keep it.

  • Step 2: Edit haystack.module.ts — remove: the { CORRECTION_REFINEMENT_QUEUE, CorrectionRefinementProcessor } import (:24-27); the registerQueue({ name: CORRECTION_REFINEMENT_QUEUE, ... }) block (:80-83); CorrectionRefinementProcessor from providers (:116); the implements OnModuleInit, the @InjectQueue(CORRECTION_REFINEMENT_QUEUE) constructor param, and the whole onModuleInit() method (:136-157). If onModuleInit was the class's only reason to implement OnModuleInit, remove OnModuleInit from the @nestjs/common import (:1). Apply the Step-1 decision on LlmModule/FeatureFlagsModule/Correction.

  • Step 3: Edit queue-metrics.collector.ts — remove the @Optional() @InjectQueue("correction-refinement") ... constructor param (:41-43) and the { name: "correction-refinement", queue: ... } entry in injectedQueues (:108). Fix the comment at :76 that references correction-refinement (leave only surviving examples).

  • Step 4: Verify it still compiles (the processor file still exists but is now unreferenced):

cd api && npx tsc --noEmit 2>&1 | tail -8

Expected: no NEW errors from these edits. (There may be an "unused" note or the still-present processor file — that's removed next task.)

  • Step 5: Commit
cd /Users/daniel/Project/hp/humanwork/.worktrees/specialist-value-output-p2plus
git add api/src/haystack/haystack.module.ts api/src/common/queue-metrics.collector.ts
git commit -m "refactor(haystack): unwire correction-refinement queue/tick/metrics (P4.2c)"

Task 2: Delete the processor + migrate the schema test​

Files:

  • Delete: api/src/haystack/jobs/correction-refinement.processor.ts

  • Delete: api/test/correction-refinement.spec.ts

  • Modify: api/test/loop-proposal-producer.spec.ts (receive the migrated schema test)

  • Modify: api/test/llm-callsite-metering.integration.spec.ts

  • Step 1: Read the schema test to migrate — api/test/correction-refinement.spec.ts:620-640 (the processed_at column exists / defaults to NULL test). Read it verbatim so you can port its assertions.

  • Step 2: Port that schema test into api/test/loop-proposal-producer.spec.ts as a new it (or a small describe("corrections.processed_at column")). It should assert the corrections.processed_at column exists and defaults to NULL, using the same datasource-build pattern the producer spec already uses (it already builds a sqlite datasource with Correction). Keep the assertion's intent identical; adapt the setup to the producer spec's existing helpers. Run it:

cd api && npx jest test/loop-proposal-producer.spec.ts

Expected: PASS (existing producer tests + the migrated schema test).

  • Step 3: Delete the processor + its spec:
cd /Users/daniel/Project/hp/humanwork/.worktrees/specialist-value-output-p2plus
git rm api/src/haystack/jobs/correction-refinement.processor.ts api/test/correction-refinement.spec.ts
  • Step 4: Edit api/test/llm-callsite-metering.integration.spec.ts — remove the CorrectionRefinementProcessor import (:73) and the single it("haystack: correction categorisation meters purpose='classify' ...") block (:221-255). Leave the file's other callsite tests untouched. Verify the file still parses + its remaining tests pass:
cd api && npx jest test/llm-callsite-metering.integration.spec.ts

Expected: PASS (remaining callsite tests; the deleted block gone).

  • Step 5: Compile check (the dangling import from the deleted processor must be gone):
cd api && npx tsc --noEmit 2>&1 | grep -iE "correction-refinement|CorrectionRefinement" || echo "no dangling correction-refinement references"

Expected: the "no dangling..." line.

  • Step 6: Commit
git add api/test/loop-proposal-producer.spec.ts api/test/llm-callsite-metering.integration.spec.ts
git commit -m "refactor(haystack): delete correction-refinement processor + spec; migrate processed_at schema test (P4.2c)"

Task 3: Remove the feature flag​

Files:

  • Modify: api/src/feature-flags/feature-flags.constants.ts

  • Modify: api/src/feature-flags/feature-flag.service.spec.ts

  • Step 1: Read feature-flags.constants.ts:44-51 and :105-108 to see the exact comment blocks around the flag.

  • Step 2: Remove the flag from FLAG_KEYS (:50) and its OQ-205/ADR-037 comment block (:45-49), and from DEFAULT_FLAG_VALUES (:107) and its comment (:106). Leave the surrounding flags intact.

  • Step 3: Remove the golden assertion line — feature-flag.service.spec.ts:53 (the correction_auto_ingest_enabled entry inside the expect(FLAG_KEYS).toEqual([...]) array). The sibling expect(rows.map(...).sort()).toEqual([...FLAG_KEYS].sort()) at :58-59 derives from FLAG_KEYS and needs no manual edit.

  • Step 4: Run the flag spec (the golden assertion must now match the trimmed FLAG_KEYS):

cd api && npx jest test/feature-flag.service.spec.ts src/feature-flags/feature-flag.service.spec.ts 2>&1 | tail -8

(Use whichever path the spec lives at — grep -rl "FLAG_KEYS).toEqual" api/src api/test.) Expected: PASS — FLAG_KEYS golden assertion green with the flag removed.

  • Step 5: Commit
git add api/src/feature-flags/feature-flags.constants.ts api/src/feature-flags/feature-flag.service.spec.ts
git commit -m "refactor(feature-flags): remove correction_auto_ingest_enabled flag (P4.2c)"

Task 4: Accept ADR-037 + update stale references​

Files:

  • Modify: docs/decisions/037-runtime-readonly-and-human-approved-promotion.md

  • Modify: docs/decisions/OPEN_DECISIONS.md

  • Modify: docs/superpowers/plans/2026-07-07-specialist-value-output-master-plan.md (optional marker)

  • Modify: api/src/learning/auto-correction.listener.ts, api/src/expert-queue/expert-queue.service.ts, api/src/common/entities.ts, api/src/haystack/haystack-exception.filter.ts, api/src/main.ts (stale comments)

  • Step 1: Flip ADR-037 (docs/decisions/037-...md), mirroring P4.1's ADR-035 accept (git show 026695ad -- docs/decisions/035*.md for the exact style):

    • Line 3: **Status**: Proposed → **Status**: Accepted (2026-07-09; P4.2 shipped — proposal flow live, legacy direct-ingest removed)
    • Decision 3 (:29): change "...are removed once the human-approved proposal flow (master plan P4.2) ships; P4.2's consumer takes over the unprocessed backlog." → "...are removed — P4.2a (LoopProposalProducerService) + P4.2b (approval UI) shipped, and P4.2c executed this contract step. P4.2a's producer consumes the unprocessed backlog as human-approved proposals."
    • Verification (:43-46): the correction-refinement.spec.ts gate-block reference is now stale (spec deleted). Replace with the current guarantees: P4.2a producer spec (backlog consumed flag-independently as proposals) + feature-flag.service.spec.ts (the flag is no longer in FLAG_KEYS).

  • Step 2: Update OPEN_DECISIONS.md (~:115, CD-30): "pending acceptance" → "accepted (2026-07-09)"; "removed when P4.2 ships" → "removed (P4.2c)".

  • Step 3: Update stale flywheel comments — in auto-correction.listener.ts:11-12, expert-queue.service.ts:160, entities.ts:1511: change "correction-refinement → KB flywheel" (or similar) to reference the P4.2a loop-proposal producer as the downstream consumer. In haystack-exception.filter.ts:38 and main.ts:388, replace the CorrectionRefinementProcessor / "correction-refinement" example with a surviving one (e.g. GoldenAnswerSyncProcessor / "golden-answer-sync"). These are comment-only; do not change logic. Do NOT modify auto-correction.listener's logic — it is the correction producer and stays.

  • Step 4: Optional — mark P4.2 done in master-plan.md:167 if that file tracks per-task status (mirror how P4.1 marked D4 closed).

  • Step 5: Verify no stale code refs remain:

cd /Users/daniel/Project/hp/humanwork/.worktrees/specialist-value-output-p2plus
grep -rn "correction_auto_ingest_enabled\|CorrectionRefinementProcessor\|CORRECTION_REFINEMENT_QUEUE" api/src api/test | grep -v node_modules || echo "no code references remain"

Expected: "no code references remain" (docs may still mention the flag historically — that's fine in ADR/OPEN_DECISIONS as accepted-history).

  • Step 6: Commit
git add docs/ api/src/learning/auto-correction.listener.ts api/src/expert-queue/expert-queue.service.ts api/src/common/entities.ts api/src/haystack/haystack-exception.filter.ts api/src/main.ts
git commit -m "docs: accept ADR-037; retire correction-refinement flywheel references (P4.2c)"

Self-review checklist (run after implementation)​

  • Contract fidelity: flag gone from FLAG_KEYS + DEFAULT_FLAG_VALUES; processor + queue + tick + metrics enrolment gone; ADR-037 Accepted; P4.2a untouched.
  • Zero frontend change: git diff <base>..HEAD -- frontend/ EMPTY.
  • P4.2a intact: git diff <base>..HEAD -- api/src/config-assets/loop-proposal-producer.service.ts api/src/config-assets/loop-proposal.distil.ts EMPTY (the new pipeline is not touched).
  • No dangling refs: the Task-4 grep prints "no code references remain".
  • Chesterton fences honoured: LlmModule/FeatureFlagsModule/Correction in haystack.module removed ONLY if grep proved no surviving consumer; auto-correction.listener logic untouched; schema test migrated not dropped.
  • processed_at semantics: only P4.2a writes it now (verified sole backlog reader).

Verification (verify skill, after all tasks)​

  • cd api && npx tsc --noEmit → clean, no correction-refinement refs.
  • cd api && npx jest feature-flag loop-proposal llm-callsite-metering haystack config-assets 2>&1 | tail → all green (flag golden assertion, migrated schema test, metering file minus deleted block, haystack module wiring regression, P4.2a producer regression). Do NOT run full npm test (seed-eval spec self-rewrites migrations under full runs).
  • cd api && npm run build → EXIT 0 (catches any OnModuleInit/import residue in haystack.module).
  • cd api && npm run lint → clean.
  • Grep sweep (Task 4 Step 5) → "no code references remain".
Last updated on by Degen