Skip to main content

P4.7 Hermes runtime spike — results (real binary, 2026-07-09)

Spike prerequisite for the P4.7 tool-dispatch implementation plan (direction A: reuse the existing gateway as the unified tool layer + a thin MCP adapter exposing business tools to Hermes; approval carrier = ExpertQueueItem). Validates 4 assumptions against the REAL hermes binary (v0.17.0, upstream dc5ef20d; source under the operator's ~/.hermes/hermes-agent, hermes on PATH) rather than reasoning from source alone. Map ≠ Territory: the 2026-07-08 spike recorded main.py:699 as the bug site; on this branch it has drifted to main.py:750-755 — re-derived from the running code, not copied from the old line number.

Assumption 1 — -t '<json>' yields "Unknown toolsets" (proves the main.py wiring bug) — ✅ CONFIRMED​

Real-code bug site (current branch):

  • agent/main.py:752 — toolsets_json = json.dumps(req.tools_manifest) if req.tools_manifest else ""
  • agent/main.py:755 — request_hermes_client = HermesClient(toolsets=toolsets_json) if toolsets_json else hermes_client
  • agent/hermes_client.py:353-354 — if self.toolsets: args.extend(["-t", self.toolsets])

So the AgentRun toolsManifest (a list of {name, description, input_schema} dicts) is json.dumps-ed and handed to -t whole. hermes chat -t is documented "Comma-separated toolsets to enable" and cli.py:3639-3641 splits the value and validate_toolsets each token (allowing tokens that are keys in mcp_servers — t not in mcp_names), else prints Warning: Unknown toolsets: … and loads nothing.

Real run:

$ hermes chat -Q -q 'hi' -t '[{"name":"order_lookup","description":"x"}]' --max-turns 1
Warning: Unknown toolsets: [{"name":"order_lookup", "description":"x"}]
(exit 0)

The entire JSON string is treated as ONE toolset name, fails validation, and Hermes silently loads no tools (exit 0, warning only — not an error). The gateway's tool manifest never reaches Hermes; the draft path is effectively tool-free regardless of what toolsManifest contains.

P4.7 fix implication: pass comma-separated toolset NAMES to -t, not json.dumps(tools_manifest). In direction A those names are the MCP server name(s) that expose the intersected business tools (active skills.tools ∩ bindings.toolSlug), e.g. -t mcp-humanwork (or whatever the MCP server is named in cli-config). The manifest's descriptions/schemas are the MCP server's concern, not -t.

Assumption 2 — an mcp_servers entry becomes a -t-selectable toolset — ✅ CONFIRMED​

Mechanism (source): cli.py:3638 mcp_names = set((CLI_CONFIG.get("mcp_servers") or {}).keys()); cli.py:3639 flags a -t token invalid only if not validate_toolset(t) AND t not in mcp_names. So an mcp_servers KEY is accepted as a toolset even though it's not a built-in.

Confirmed against the real config + binary:

  • ~/.hermes/config.yaml already defines mcp_servers: { github, notion, slack } (two transports present: stdio command/args/env for github+slack, HTTP url/auth for notion).
  • Built-in toolset whitelist check (from toolsets import validate_toolset): terminal/delegation/web → True; order_lookup/humanwork/files → False (a business-tool name is NOT a built-in toolset — it MUST come via MCP). For each mcp name: validate_toolset('github')=False BUT in mcp_names=True → -t admits it.
  • Real run hermes chat -Q -q '…' -t 'github' --max-turns 1 → NO "Unknown toolsets" warning (exit 0), vs assumption 1's JSON which warned immediately. -t <mcp-name> is accepted as a toolset, not rejected. (-Q suppresses the MCP connection banner so the actual tool call isn't visible in quiet mode, and github's MCP needs auth — but the load-bearing claim for P4.7, "an mcp_servers key is a selectable toolset", holds.)

P4.7 implication: configure mcp_servers: { humanwork: { url or command … } } in the per-org Hermes cli-config (baked into the image / injected), then pass -t humanwork (the fix for assumption 1). The Humanwork MCP server's handler calls the existing /v1/runtime/tool-gateway/execute; credentials stay in the control plane (the MCP server only knows the gateway URL + runner token).

Assumption 3 — write tool under HERMES_GATEWAY_SESSION=1 in a non-interactive subprocess — ✅ CONFIRMED (but the 2026-07-08 premise changed in v0.17.0)​

Map ≠ Territory: the earlier spike recorded "gateway-approval staging (HERMES_GATEWAY_SESSION → writes staged to disk, uncommitted, tools/approval.py:136)". In v0.17.0 that is NOT what the flag does. Verified against tools/approval.py:

  • _is_gateway_approval_context() (:136-154): HERMES_GATEWAY_SESSION=1 → returns True (comment :139 explicitly calls it a "Legacy gateway integration" flag; newer paths bind HERMES_SESSION_PLATFORM via contextvars).
  • The dangerous-command approval fork (:1364-1379):
    • non-interactive AND non-gateway (no HERMES_GATEWAY_SESSION, no interactive): the dangerous command is AUTO-APPROVED — :1369 return {"approved": True, "message": None} (with a warning "set HERMES_INTERACTIVE or HERMES_GATEWAY_SESSION to require approval"). A headless hermes chat -Q subprocess with no gateway flag AUTO-RUNS dangerous writes.
    • is_gateway (HERMES_GATEWAY_SESSION=1): :1371-1379 submit_pending(...) then return {"approved": False, ...} → the command is parked pending an external resolver. In a non-interactive subprocess there is no listener, so (per the cron warning at :143-148) it "submit[s] a pending approval with no listener and block[s] the job indefinitely."

So NOT disk-staging — it's either auto-approve (flag off → unsafe) or submit-pending-and-block (flag on, no listener → hangs). Neither is acceptable for the draft path. This is a source-level certainty; a real run would only demonstrate the hang we must avoid (and would need --yolo/a genuinely dangerous command to trigger the terminal-tool path, which is not how business tools flow anyway).

Assumption 4 — how "pending approval" reaches the MCP handler — ✅ CONFIRMED (design consequence of #3)​

Because Hermes's own approval mechanism is unusable for the draft path (assumption 3), approval must NOT ride Hermes at all. The correct flow (direction A + option a, now source-justified):

  • A business tool is a MCP tool, not a Hermes terminal/dangerous-command — so it never enters the tools/approval.py terminal-approval fork above. The Humanwork MCP server's handler calls /v1/runtime/tool-gateway/execute.
  • The gateway (control plane) inspects the tool's accessClass. For write: it does NOT perform the real side effect; it enqueues an ExpertQueueItem (HITL, mirroring the P3.5 escalationReason pattern) and returns a normal JSON tool result to the MCP handler, e.g. {status: "queued_for_approval", requestId: …}.
  • The MCP handler returns that JSON as the tool's output. Hermes sees an ordinary successful tool result (not a pending-approval / blocked state) and continues its turn — no dependency on HERMES_GATEWAY_SESSION, no subprocess hang. The real side effect happens later, server-side, when an Expert releases the ExpertQueueItem.

P4.7 approval carrier = ExpertQueueItem (already locked by the user); the gateway is the enforcement point; Hermes is never trusted to gate writes.

Spike verdict — all 4 assumptions resolved​

  1. ✅ -t '<json>' → Unknown toolsets (the main.py:750-755 / hermes_client.py:354 bug is live) — P4.7 fix: pass comma-separated toolset names.
  2. ✅ an mcp_servers key is a -t-selectable toolset — P4.7: register mcp_servers: { humanwork: … }, select with -t humanwork.
  3. ✅ HERMES_GATEWAY_SESSION is legacy + either auto-approves (off) or blocks-with-no-listener (on) — not usable; do not rely on Hermes-side approval.
  4. ✅ pending-approval is a control-plane concern — gateway returns a normal "queued" tool result; ExpertQueueItem is the carrier; Hermes never blocks.

Net: direction A is validated end-to-end at the source+binary level. P4.7 implementation can now be planned: (i) fix the -t wiring bug (comma-separated names), (ii) build the thin Humanwork MCP server whose handlers call the existing gateway execute endpoint, (iii) register it in the per-org Hermes cli-config mcp_servers + select the intersected active-skills.tools ∩ bindings.toolSlug toolset via -t, (iv) gateway enforces write-class → ExpertQueueItem, returning a "queued" result. No code assumption rests on unverified Hermes behaviour anymore.

Notes on the running binary​

  • hermes chat flags present in v0.17.0: -q/--query, -t/--toolsets ("Comma-separated toolsets to enable"), -s/--skills, -m/--model, -Q/--quiet ("for programmatic use"), --max-turns N, --yolo ("Bypass all dangerous command approval prompts … implies MCP servers …"), --pass-session-id, --tui, --cli, --dev.
  • --yolo help text explicitly ties it to MCP servers — consistent with the spike's direction-A conclusion that MCP is the load-bearing carrier.
  • Toolset validation happens at arg-parse time, BEFORE any LLM call — so assumption 1 is verifiable cheaply (no model spend) as shown.
Last updated on by Degen