ADR-040 — Runtime thesis: preserve the execution engines, unify the control plane

Status: Accepted (2026-07-20) Date: 2026-07-20 Author / Owner: zheng@humanity.org (agent runtime) Supersedes: the draft 034-agent-runtime-evolution.md on the unmerged docs/adr-034-agent-runtime-evolution branch (never merged to dev; its "retire Hermes" thesis is reversed here — see §6). The ADR-034 number on dev belongs to ADR-034 Specialist Prompt-Template Scoping, an unrelated doc. Refs: PR #4277 coordination thread (the decision this ADR records), ADR-029 (tool execution runtime), ADR-033 (professional Specialist), ADR-035 (eval sandbox & golden sets), ADR-037 (runtime read-only toward assets), ADR-038 (ECS-canonical). Design context: docs/superpowers/specs/2026-07-15-unified-agent-abstraction-design.md (cross-channel unified assembly), docs/superpowers/specs/2026-07-07-unified-config-asset-model.md (config-asset model + assembler).

0. TL;DR

The prior runtime-evolution draft (ADR-034 on the #3408 branch, owner sign-off 2026-06-29) proposed retiring the Python Hermes runtime and rebuilding the agent layer in-process inside NestJS. That draft never merged, and in the two weeks after sign-off the platform shipped Phase 0–4 of the specialist-value-output program (PR #4277) — config-assets, the NestJS prompt assembler feeding Hermes, the P4.1 eval publish gate, cost control, and the MCP-server tool carrier. That shipped shape is the answer to the structural problems the draft named, but it reaches it a different way.

Revised thesis, recorded here as an Accepted decision: the execution engines stay; the control plane unifies.

• Hermes stays as the long-term text /chat execution engine. Tavus stays as the video execution engine. Both are stateless as to authoritative business-session state — that lives in NestJS (messages + conversation.agent_context) — so the retirement plan and its adapter-migration risks fall away. ("Stateless" here means no authoritative business state, not no local cache: Hermes still keeps an agent-local --resume session cache as a continuity optimization — see §1.2 for that known debt.)
• What unifies is the NestJS control plane around them: Specialist config assets, layered prompt assembly, the publish-time eval gate, cost/budget control, and the tool gateway all consolidate in NestJS — which is exactly what Phase 0–4 already built.

This ADR does not introduce new code. It records the thesis reversal so the design intent is written down (the prior draft's "delete agent/" inventory would otherwise still read as the plan of record), reconciles the eval/runtime doc conflict raised in the PR #4277 thread, and fixes the ADR-034 number collision.

1. Context

1.1 Why this ADR exists

Three facts collided:

1. A draft proposed retiring Hermes. 034-agent-runtime-evolution.md (owner sign-off 2026-06-29, on branch docs/adr-034-agent-runtime-evolution, never merged) proposed deleting the entire agent/ directory and moving LLM execution in-process into NestJS on the Vercel AI SDK. Its §5.3 "delete entirely: agent/" swept agent/evals/ and the /eval/run endpoint.
2. The eval stack then became load-bearing. PR #4277 (Phase 0–4) wired agent/evals/ + /eval/run as the execution engine behind the P4.1 publish gate (eval-gate.processor.ts → /eval/run). Deleting agent/ would delete the gate's engine. This was raised as a coordination note in the PR thread (@AlexHumanity) and accepted.
3. The number collided. On dev, ADR-034 was assigned to Specialist Prompt-Template Scoping, and eval methodology / runtime-read-only decisions landed as ADR-035 and ADR-037. The runtime-evolution draft has no home number on dev.

Rather than rewrite the stale draft in place under a colliding number, this ADR records the revised decision on dev under a fresh number and marks the draft superseded.

1.2 The problems the prior draft named are real — its conclusion was not

The prior draft's problem analysis (its §1.3) still holds and is worth preserving as the motivation for the control-plane unification, not for retirement:

Problem (from the ADR-034 draft §1.3)	Still real?	How the shipped control plane addresses it — without retiring Hermes
Black-box subprocess → no in-flight tool authorization; only interception point is HERMES_TOOLSETS at process start	Yes	Tool access now flows through the NestJS-owned toolset manifest + MCP-server carrier (config-assets tool_binding → HERMES_TOOLSETS); authorization lives in NestJS, Hermes remains the executor of an already-authorized toolset
runtime/session_map.json local file violates the active-active rule	Still live, not yet resolved	The authoritative business session state is in NestJS (messages + conversation.agent_context), but live /chat still uses the agent-local Hermes resume cache — workspace_layout.py persists runtime/session_map.json (falling back to Redis) and hermes_client.py passes the recovered id via --resume. So this is a preserved-with-known-debt item, not "Hermes is stateless — remove the cache." A pod restart / re-route drops the Hermes-side resume and the conversation continues from the NestJS-held history (correct output, lost intra-Hermes session continuity). Making Hermes truly stateless (drive --resume purely off NestJS-held state, or drop resume) is a follow-up, not a shipped fact. The unified-agent spec §2.1 "runtime 状态归属" row describes the target, not today's /chat.
stdout parsing is a fragile interface (F1–F9 pitfalls)	Yes, ongoing	Contained, not eliminated — and the containment lives in the Python agent, not NestJS: main.py's parse_hermes_output / clean_hermes_response normalize Hermes' -Q stdout into structured JSON before it crosses the HTTP boundary; NestJS AgentClient then consumes that JSON (response.json()), it does not re-parse stdout. So Hermes-output hardening belongs in the agent's parser layer; the eval gate is a downstream backstop that catches a scored regression at publish time, not the parser itself. The fragility is inherent to a CLI executor and is the standing operational cost of the preserve decision (see §5 Consequences).
Original per-org-deployment selection assumption no longer holds (ADR-028/ADR-038)	Yes	Acknowledged; centralized ECS execution does not by itself require replacing the executor
We pay the full Hermes architectural tax for capabilities we don't use	Partially	The tax is real but bounded; the cost of a full in-process reimplementation (contract drift across 5 business call sites, prompt-parity risk, dual state management) is judged higher than the tax — this is the core reversal

The reversal, in one line: the structural fix was to move the control (assets, authorization, gate, budget — and, progressively, assembly) into NestJS — not to move the execution out of Hermes. Phase 0–4 delivered that direction without retiring the executor, and it works. One honest caveat on assembly: the NestJS-side prompt assembler ships dark (prompt_assembler_enabled defaults OFF); on the default path Hermes still performs its own final assembly (chat_helpers.py build_hermes_prompt_with_meta — absent agent_context.assembled_prompt ⇒ legacy self-assembly, byte-identical to before). So "assembly is unified in NestJS" is the migration target and opt-in path, not the default-on state today.

2. Decision

D1. Hermes is the long-term text execution engine; it is not being retired. The agent/ directory, the Hermes subprocess path, and the /chat + /eval/run endpoints stay. agent/evals/ + /eval/run are explicitly preserved as the execution engine behind the ADR-035 / P4.1 publish gate.

D2. Tavus is the long-term video execution engine. Per the unified-agent spec, both runtimes are stateless executors; neither holds authoritative business-session state.

D3. The control plane unifies in NestJS. Config assets (Soul / Skill / KB-curation / tool_binding), the layered prompt assembler, the publish-time eval gate, cost/budget control, and the tool gateway are NestJS-owned. Hermes and Tavus consume the output of that control plane. This is the shape PR #4277 shipped and the unified-agent spec formalizes (channel-neutral AssembledConfig → per-channel adapters). Rollout caveat: for the prompt-assembly layer specifically this is a migration in progress, not a completed cutover — the NestJS assembler is behind prompt_assembler_enabled (default OFF), and when it's OFF Hermes still self-assembles (see §1.2). The direction is set and the opt-in path is live; "Hermes no longer assembles" is not yet true on the default path, so no one should remove the agent-side assembly on the strength of this ADR.

D4. No in-process LLM-execution rewrite of the chat pipeline. The AI SDK does live in NestJS (api/src/llm/, per ADR-034-E2 in root CLAUDE.md) for in-process tasks (summary, subject, KB refinement, embeddings, meeting/marketing/Tavus summarisation) — but the main /chat draft pipeline and Expert↔AI consultation deliberately stay on Hermes via AgentClient. The prior draft's "5 business call sites to switch" migration is not undertaken.

D5. The eval methodology and asset-write policy are owned elsewhere and referenced, not restated. Eval sandbox / golden-set ownership / versioning / Langfuse positioning are ADR-035 (Accepted). Runtime-is-read-only-toward-assets is ADR-037 (Accepted). This ADR owns only the runtime execution thesis and defers eval/asset policy to those.

3. Reconciliation with the PR #4277 coordination thread

The PR #4277 thread worked through the ADR-034-draft §-by-§ conflicts before the direction was revised. This section records where each landed under the revised thesis, so the thread's commitments have a written home.

Draft § (on #3408)	Thread outcome	Status under this ADR
§5.3 "delete entirely: agent/" (sweeps the gate's eval engine)	Accepted the objection; agreed the eval engine must survive	Resolved by reversal. With no retirement, there is no deletion inventory and no adapter migration. The eval engine is a statement of standing architecture (D1), not a migration guard.
§5.4 prompt port (agent/prompts/*.txt → api/src/llm/prompts/*.ts) + "prompts and eval harness migrate together" constraint	Accepted the same-milestone constraint	Conditional guard — kept, not dissolved. The port most likely never happens, but the earlier claim that "eval exercises the production pipeline by construction" was too strong and is withdrawn: /eval/run runs the Hermes binary against the golden set via the eval harness, which is not guaranteed to be the exact same assembly/parse path as production /chat (different entrypoint; and with prompt_assembler_enabled OFF the two could assemble differently). So the parity guard is a real, load-bearing constraint, not a formality: should a prompt port (or an assembler-default flip) ever execute, prompts and the eval harness migrate in the same milestone, with a byte-identical (hash-compare) check of the assembled prompt across old/new pipelines before cutover — and, ideally, /eval/run should be wired to drive the same assembly path production uses so the gate can't green-light a prompt that live traffic would render differently.
§7.1 eval_observations table	Accepted: reconcile before adding a fourth ground-truth table	Deferred to eval ownership (ADR-035). Land as an extension/materialization over the three shipped signals (#3323 edit-signal, calibration analytics, PR #4277 message_feedback), not a parallel store.
§7.2 Langfuse	Converges with what shipped	Done / referenced. Integration is shipped (API-side @langfuse/otel bootstrap + AI SDK telemetry; Hermes-side tracing incl. the flush/env fixes; self-hosted dev instance with reply-quality eval rules). Remaining work is rollout ops (staging/prod instances, porting eval rules), not integration. Positioning (sync-copy + runner/dashboard, not gate authority) is ADR-035 §"Tooling positions".
§7.3 promptfoo CI gate	Accepted: drop it	Dropped. The single holistic llm-rubric assertion is the judging pattern the noise study measured as unreliable, and a third golden-case format is a drift generator. If it ever returns, it must be a thin runner over the same eval_golden_scenarios rows, with K-vote / item gates / quorum / blockers / paired compare as entry criteria. Known gap accepted: today's gate fires on asset publish, so edits to the file-based role templates in git have no golden-scenario coverage; those templates change rarely, so no tooling is stood up for it now (revisit triggers: the §5.4 port executes, or template churn increases).
golden-store ownership + versioning	Rule agreed in-thread	Owned by ADR-035 (git = authoring truth for platform-global synthetic cases, seeded → eval_golden_scenarios as serving copy; DB-native for future org-scoped / promoted-from-real-conversation PII-bounded cases). Versioning state: eval_runs.scenario_snapshot pins the executed case set (caseKey + role); rows currently mutate in place (no version column) — git history is the version record while the only edit path is a migration.

3.1 Eval-tier boundary (platform gate blocks; org-tier advises)

A boundary agreed after the thread, recorded here so the two eval tiers don't drift into one:

Tier	Golden source	Owner	Gate behavior on publish
Platform-global	eval_golden_scenarios (platform synthetic set, ADR-035)	the eval-methodology track (ADR-035)	Hard gate — the shipped P4.1 hallucination gate: a hallucination on the platform quality bar blocks the org_instance publish (publishing → blocked). Unchanged by this ADR.
Org-scoped	an org's own golden / promoted-from-real-conversation cases (PII-bounded, DB-native — the ADR-035-deferred set)	the runtime/control-plane track	Advisory only — records scores/warnings; never blocks a publish.

Rationale: the platform bar is a floor every org must clear, so it gates; an org's own cases are a signal to that org, and letting them block publishes would turn one org's noisy golden into a self-inflicted outage. The org-scoped advisory tier is net-new work, not built here — this ADR only fixes the boundary so it isn't conflated with the platform gate when it lands.

3.2 Content-hash follow-up (implemented in this PR)

The one code follow-up committed in the thread — write a per-case content hash into eval_runs.scenario_snapshot at run time — is implemented in this PR (not deferred). EvalGateProcessor now stamps each scored case as {id, role, contentHash}, where contentHash is computeContentHash (the single hashing source, config-assets/CLAUDE.md) over the case's scored fields (question / goldenAnswer / mustInclude / mustNotInclude / context / kb* — identity/toggle fields excluded). This pins the verdict to the exact scored content, so a past EvalRun stays reproducible even though eval_golden_scenarios rows mutate in place (no version column). Full row versioning (aligned with the config-assets draft/version machinery) still waits until an authoring UI actually exists — this hash is the minimal reproducibility floor, and it directly hardens the org-scoped tier (§3.1), whose cases will be edited via UI. Files: api/src/config-assets/eval-golden-scenario.hash.ts (+ .hash.spec.ts), wired at eval-gate.processor.ts.

4. Scope

In scope (this ADR): the runtime execution thesis (D1–D4), the reconciliation record (§3), the eval-tier boundary (§3.1), and the content-hash reproducibility floor shipped in this PR (§3.2). Superseding the ADR-034 draft and fixing the number collision.

Out of scope (owned elsewhere, referenced):

• Eval sandbox / golden-set / Langfuse methodology → ADR-035.
• Runtime-read-only-toward-assets / improvement-loop-proposes → ADR-037.
• Cross-channel unified assembly (Hermes ↔ Tavus config & session consistency) → docs/superpowers/specs/2026-07-15-unified-agent-abstraction-design.md (still Draft; this ADR only fixes its runtime-thesis premise as Accepted — both executors preserved).
• Long-term memory (working_memory / observation_log, Anthropic Memory Tool) → the draft proposed these; they are not adopted by this ADR and remain a future decision if pursued.

5. Consequences

• Positive: the design intent is written down under a valid number; the "delete agent/" plan of record is retired; the eval engine's status is unambiguous (preserved); the PR #4277 thread's commitments have a durable home; no code churn, no migration risk, no prompt-parity risk incurred.
• Standing cost (accepted): the stdout-parsing fragility of the Hermes CLI executor (F-class pitfalls in agent/CLAUDE.md) remains the operational tax of the preserve decision. It is contained by the NestJS-side contract (AgentClient normalization) and by the eval gate catching regressions at publish time, but it is not eliminated. Future Hermes upstream changes may produce new F-class bugs — this is the known trade for not reimplementing execution.
• Neutral: the AI SDK footprint in NestJS (api/src/llm/) is unchanged — it keeps its in-process tasks; it does not absorb the /chat pipeline.

6. Supersession of the ADR-034 draft

The draft 034-agent-runtime-evolution.md (branch docs/adr-034-agent-runtime-evolution, never merged) is superseded by this ADR on its central thesis: its TL;DR "retire the Python Hermes runtime … build in-process inside NestJS" is reversed. Its problem analysis (§1.3) is preserved as motivation (see §1.2 above). Its decisions D1–D12, replacement inventory (§5), memory implementation (§6), and phased retirement roadmap (§8) are not adopted. Because that branch never merged, no on-disk doc requires editing on dev; the supersession is recorded here and should be noted on the draft if that branch is ever revived or landed.

7. Re-evaluation triggers

This decision should be revisited if any of the following hold:

1. Hermes upstream instability crosses a threshold — e.g. F-class stdout-parsing bugs recur at > 1/quarter and the NestJS contract layer can no longer absorb them cheaply → re-evaluate the in-process execution option the draft proposed.
2. The control plane needs a capability Hermes structurally cannot expose — e.g. true in-flight per-tool-call approval mid-turn (not just per-turn toolset gating) becomes a product requirement → re-evaluate.
3. A second stateless executor beyond Hermes/Tavus is needed and maintaining N CLI adapters costs more than one in-process engine → re-evaluate.
4. The unified-agent spec (§8-Q2) resolves the assembler placement toward NestJS in a way that makes Hermes a thin enough shell that removing it is nearly free → re-evaluate the preserve decision on cost grounds.

8. References

Internal (on dev):

• ADR-029 — tool execution runtime (control-plane-owned toolset authorization)
• ADR-033 — professional Specialist architecture
• ADR-035 — eval sandbox & golden sets (owns eval methodology)
• ADR-037 — runtime read-only toward assets
• ADR-038 — Railway deprecation / ECS-canonical
• Root CLAUDE.md — Architecture request flow; "LLM call layer (ADR-034 E2)" (the api/src/llm/ boundary)
• agent/CLAUDE.md — Hermes /chat lifecycle + F-class pitfalls

Design specs:

• docs/superpowers/specs/2026-07-15-unified-agent-abstraction-design.md — cross-channel unified assembly (Hermes/Tavus), stateless-executor premise. (Lands on dev via its own doc commit; forward-referenced here.)
• docs/superpowers/specs/2026-07-07-unified-config-asset-model.md — config-asset model + assembler (§6 nine-layer assembly, §11 assembler placement)

Superseded:

• 034-agent-runtime-evolution.md (branch docs/adr-034-agent-runtime-evolution, unmerged) — thesis reversed by this ADR