Humanwork Sandbox Supervisor Complete Implementation Plan

2026-06-05 SHIPPED: humanwork sandbox supervisor landed via #1538 and a series of hardening follow-ups (#1727 sandbox artifact persistence boundaries; see docs/architecture/HUMANWORK_SANDBOX_SUPERVISOR.md for the as-shipped design). This plan is preserved for historical reference; treat the current code under runners/sandbox-supervisor/ + api/src/runtime-control-plane/ as the source of truth.

For Hermes: Use subagent-driven-development to execute this plan task-by-task.

Goal: Ship the final Humanwork runtime sandbox substrate: API dispatches typed sandbox capability operations to a Python worker shim; the worker talks to a long-lived Humanwork sandbox supervisor daemon over a Unix socket; the supervisor launches the existing Zig capability shell inside a supervised bubblewrap sandbox; the supervisor emits trusted evidence; the API fails closed when evidence or host primitives are missing.

Threat claim: hostile user-space runtime isolation. Kernel-zero-day / separate guest-kernel isolation is out of scope unless ADR-022 is reopened. Firecracker is contingency-only for separate-kernel requirements, failed proof gates, or impossible host classes.

---

Source-of-truth chain

Read before writing code:

1. docs/architecture/ISOLATION_SANDBOXING_BINARIES.md — layered architecture thesis.
2. docs/decisions/022-sandbox-substrate.md — substrate decision.
3. docs/architecture/HUMANWORK_SANDBOX_SUPERVISOR.md — complete technical design.
4. docs/decisions/020-isolation-classification.md — isolation matrix.
5. docs/_archive/RIG_RUNTIME_HARD_CUT_CUTOVER.md (archived 2026-06-04) — runtime cutover gate context.
6. Rig bwrap backend: https://github.com/humanity-org/rig/blob/main/packages/runtime/src/control-plane/runtime/sandbox/backend-bwrap.ts
7. Rig sandbox orchestrator: https://github.com/humanity-org/rig/blob/main/packages/runtime/src/control-plane/runtime/sandbox/orchestrator.ts
8. bubblewrap project: https://github.com/containers/bubblewrap

Current Humanwork seams:

• api/src/runtime-control-plane/sandbox-capability-executor.service.ts — dispatch, production evidence validation, status/exception mapping.
• api/src/runtime-control-plane/sandbox-capability-gateway.service.ts — policy + run-event ledger.
• runners/sandbox-worker/worker.py — current direct spawn path to remove from production.
• runners/zig-capability-shell/ — typed executor to keep inside the sandbox.
• agent/evals/test_sandbox_capability_ops.py — hostile-code deny battery to expand.

External primitive docs:

• Linux namespaces: https://man7.org/linux/man-pages/man7/namespaces.7.html
• bubblewrap: https://github.com/containers/bubblewrap
• seccomp filter: https://docs.kernel.org/userspace-api/seccomp_filter.html
• cgroup v2: https://docs.kernel.org/admin-guide/cgroup-v2.html
• Landlock: https://docs.kernel.org/userspace-api/landlock.html
• Linux project quotas: xfs_quota / ext4 prjquota

---

Branch policy and implementation gate

Use one implementation branch, not a stack.

• Current docs PR: PR #1444 (https://github.com/humanity-org/humanwork/pull/1444); use the live PR head from GitHub, not a hard-coded docs branch name.
• Implementation branch after cosign: feat/humanwork-sandbox-supervisor.

Tasks 2-35 are production implementation tasks. Do not start production wiring until ADR-022 has explicit AlexD / Paul / Eusden cosign recorded in docs/decisions/022-sandbox-substrate.md. Experimental spikes before cosign must remain non-production and cannot wire the production API/worker path.

---

End-state architecture to build

API SandboxCapabilityGatewayService
  -> TypedSandboxCapabilityExecutor
  -> RemoteSandboxWorkerBackend
  -> sandbox-worker HTTP shim
  -> sandbox-worker supervisor_protocol.py
  -> sandbox-supervisor long-lived daemon over Unix socket
  -> operation manifest + in-flight registry
  -> linux-bwrap-supervisor backend
  -> bwrap sandbox with namespaces/cgroups/seccomp/Landlock/net policy
  -> sandbox-init
  -> Zig capability-shell
  -> supervisor-owned SandboxResult + substrate evidence
  -> API evidence verification + run-event ledger

No production path may invoke capability-shell outside the supervisor. No production path may report worker success without supervisor-owned linux-bwrap-supervisor evidence.

---

Final decisions locked by this plan

These are implementation decisions, not options.

Area	Final decision
Supervisor process model	Long-lived daemon over a Unix socket. --run-json is test/smoke only and cannot be the production worker path.
Wire schema	Canonical JSON is camelCase. Internal Python/Zig may use snake_case only behind explicit conversion tests. Unknown wire fields reject.
Evidence trust boundary	Supervisor constructs the outer SandboxResult and evidence. Child/capability-shell output is untrusted payload and cannot set or override evidence fields.
Operation policy	A versioned operation manifest maps API ops -> capability-shell ops -> network/allowlist/workspace/seccomp policy. No manifest entry, no production dispatch.
Executable allowlist	allowedExecutables: [] means deny-all. Entries are canonical in-sandbox absolute executable paths only; manifest aliases, bare names, PATH lookup, relative paths, and globs are forbidden. Ops that spawn child processes require a non-empty manifest-approved allowlist. Caller env passthrough is forbidden.
Resource policy	Production requires cgroup v2 pids/memory/cpu/io plus workspace byte quota. Missing required controllers or quota support is NO_GO, not degraded success.
Workspace writes	Writable output/log/tmp areas are bounded by per-op workspace quota and output/log byte caps. No unbounded host writable bind exists.
Seccomp	bwrap receives a default-deny seccomp BPF program by FD. OCI JSON is only an outer container profile and is not the inner bwrap profile.
Landlock	Landlock evidence is always present. If host ABI is available, sandbox-init must apply it or fail closed. If unavailable, evidence records kernel_unsupported; Landlock is defense-in-depth, not the only filesystem boundary.
Network off	bwrap unshares network namespace; no route, metadata, raw socket, host localhost, or direct egress.
Network proxy-only	Implemented as supervised netns/veth + host proxy + nftables allow-only-proxy. If required host primitives or proxy config are missing, dispatch returns proxy_only_unavailable; no silent downgrade to off.
Deployment substrate	Native/systemd dedicated worker host is the recommended production host class. Dedicated K8s node is allowed only if cgroup v2 delegation, project quotas, bwrap namespaces, seccomp, and CAP_NET_ADMIN for proxy-only are proven. Railway/nested Docker is NO_GO for production hostile-code mode.
CI/smoke	CI may report host-gated NO_GO on GitHub runners. NO_GO is not a production green. Smoke must prove a real eligible host before cutover.
Rollback	Rollback disables/routes away from sandbox capability execution or to a previous supervisor build. It never routes to unsandboxed direct spawn.

---

Canonical protocol and evidence contract

API -> worker / worker -> supervisor dispatch JSON

Wire fields are camelCase. env is not a valid field.

{
  "id": "op_...",
  "orgId": "org_...",
  "runId": "run_...",
  "correlationId": "...",
  "op": "artifact_read",
  "args": {},
  "timeoutMs": 30000,
  "stdioMaxBytes": 262144,
  "network": "off",
  "allowedExecutables": []
}

Required validation:

• id, orgId, runId, correlationId, op are non-empty strings with bounded length and safe character set.
• args is JSON object only; no raw shell, raw env, or arbitrary command string passthrough.
• timeoutMs is clamped to configured max; stdioMaxBytes is clamped to configured max.
• network is exactly off or proxy-only.
• allowedExecutables is an array of canonical in-sandbox absolute executable paths only; empty means deny-all. Manifest aliases, bare executable names, PATH lookup, wildcard/glob entries, relative paths, and host paths outside the sandbox mount namespace reject before dispatch. The supervisor executes only argv[0] values that byte-match an allowed absolute path after mount-namespace setup.
• Unknown fields reject in production.

Operation manifest

Create runners/sandbox-supervisor/policy/operation-manifest.json and generated Zig/Python types from it. This table is the minimum production manifest; any API op not represented here rejects before worker dispatch.

API op	Capability-shell op	Spawns child executable?	Allowlist requirement	Network	Writable roots	Notes
artifact_read	artifact_read	false	allowedExecutables must be empty	off	none	Read declared input/artifact paths only.
artifact_write	artifact_write	false	allowedExecutables must be empty	off	/workspace/output	Output path must normalize under output root.
exec_argv	exec_argv	true	non-empty explicit allowlist required	off or proxy-only if manifest permits	/workspace/output, /workspace/tmp, /workspace/logs	Shell metacharacters are data, not shell syntax.
browser_action	browser_action	true	non-empty explicit browser/runtime allowlist required	proxy-only only	/workspace/output, /workspace/tmp, /workspace/logs	If current shell exposes browser_step, add adapter or rename before cutover; no unmapped op ships.

Acceptance: API allowed-op list, worker protocol, supervisor manifest, and capability-shell handler names must match in tests. No production op ships with TODO, fallback handler, or unmapped shell op.

Supervisor-owned result JSON

The supervisor returns the outer envelope. Child output is nested only in stdout, stderr, and operation-specific payload fields. If a child prints JSON containing sandboxBackend, limits, cleanup, hashes, or versions, those fields are treated as plain stdout and cannot satisfy evidence gates.

{
  "schemaVersion": 1,
  "id": "op_...",
  "status": "ok",
  "eventType": "sandbox.exec.finished",
  "sandboxBackend": "linux-bwrap-supervisor",
  "supervisorVersion": "...",
  "supervisorBuildId": "sha256:...",
  "bwrapVersion": "...",
  "bwrapPath": "/usr/bin/bwrap",
  "capabilityShellHash": "sha256:...",
  "operationManifestHash": "sha256:...",
  "sandboxProfileHash": "sha256:...",
  "seccompProfileHash": "sha256:...",
  "requestedNetwork": "off",
  "effectiveNetwork": "off",
  "limits": {
    "timeoutMs": 30000,
    "stdioMaxBytes": 262144,
    "pidsMax": 64,
    "memoryMaxBytes": 536870912,
    "cpuMax": "50000 100000",
    "ioMax": [{ "major": 8, "minor": 0, "rbps": 10485760, "wbps": 10485760, "riops": 1000, "wiops": 1000 }],
    "workspaceMaxBytes": 1073741824,
    "workspaceOutputMaxBytes": 268435456,
    "workspaceLogsMaxBytes": 67108864
  },
  "landlock": {
    "available": true,
    "enabled": true,
    "abi": 4,
    "rulesetApplied": true,
    "reason": null
  },
  "cleanup": {
    "processTreeReaped": true,
    "cgroupKilled": false,
    "cgroupRemoved": true,
    "workspaceTornDown": true,
    "quotaReleased": true
  },
  "exitCode": 0,
  "durationMs": 1234,
  "stdout": "",
  "stderr": "",
  "stdoutBytes": 0,
  "stderrBytes": 0,
  "reason": null
}

Production API rules:

• status="ok": require every evidence field above; otherwise throw BadGatewayException and ledger sandbox_capability.failed.
• status="denied": throw ForbiddenException with sanitized reason; ledger sandbox_capability.blocked. Pre-dispatch denials such as proxy_only_unavailable do not count as sandbox success.
• status="timeout": throw RequestTimeoutException; ledger sandbox_capability.failed.
• status="cancelled": return/throw the existing cancellation shape; ledger sandbox_capability.cancelled.
• status="error" or unknown: throw BadGatewayException; ledger sandbox_capability.failed.
• sandboxBackend must equal linux-bwrap-supervisor exactly.
• Hash fields must match sha256:<64 lowercase hex> and must be computed by the supervisor from exact manifest/profile/seccomp bytes used for that op.
• requestedNetwork must equal dispatch; effectiveNetwork must equal enforced mode on ok.
• limits must equal configured production policy; cpuMax="max ..." is not production success.
• Cleanup object is required and canonical: processTreeReaped, cgroupKilled, cgroupRemoved, workspaceTornDown, quotaReleased. On success, processTreeReaped, cgroupRemoved, workspaceTornDown, and quotaReleased must be true; cgroupKilled is factual evidence (true if supervisor used cgroup subtree kill during timeout/cancel/reaper fallback, false for normal exit) and is not required to be true.

Canonical cleanup evidence object:

{
  "processTreeReaped": true,
  "cgroupKilled": false,
  "cgroupRemoved": true,
  "workspaceTornDown": true,
  "quotaReleased": true
}

This object is required in every supervisor result. The API validator must check presence and boolean type for all five fields. For status=ok, processTreeReaped, cgroupRemoved, workspaceTornDown, and quotaReleased must be true; cgroupKilled is required factual evidence and may be false on normal exit or true when the supervisor used cgroup subtree kill during timeout/cancel/reaper fallback.

---

Resource policy

Production defaults:

Limit	Default	Enforcement
pidsMax	64	cgroup v2 pids.max before exec
memoryMaxBytes	536870912	cgroup v2 memory.max before exec
cpuMax	50000 100000	cgroup v2 cpu.max before exec
ioMax	10 MiB/s read+write and 1000 read/write IOPS per resolved worker data device	cgroup v2 io.max before exec
workspaceMaxBytes	1 GiB	total per-op workspace project quota before exec
workspaceOutputMaxBytes	256 MiB	/workspace/output directory cap enforced by supervisor accounting + workspace quota
workspaceLogsMaxBytes	64 MiB	/workspace/logs directory cap enforced by supervisor accounting + workspace quota
stdioMaxBytes	256 KiB returned inline	combined stdout/stderr drain cap and maximum inline stdio bytes in the result envelope
stdioMaxBytes applies only to stdout/stderr captured inline in the supervisor result. It does not cap files written under /workspace/output; that directory is capped by workspaceOutputMaxBytes. /workspace/logs is capped by workspaceLogsMaxBytes. The whole per-op workspace remains capped by workspaceMaxBytes. Production validators must reject legacy byte-limit aliases and accept only these canonical field names.

Readiness is NO_GO unless the worker can:

• create child cgroups under $HUMANWORK_SANDBOX_CGROUP_ROOT;
• write pids.max, memory.max, cpu.max, and io.max for the worker data device;
• remove child cgroups after completion;
• apply project quotas under $HUMANWORK_SANDBOX_ROOT or an approved equivalent with identical hard byte enforcement;
• use cgroup.kill or an equivalent complete subtree kill path.

Race-free launch sequence:

1. Create op workspace and apply quota.
2. Create op cgroup and write limits.
3. Spawn bwrap child paused before exec using clone3(CLONE_INTO_CGROUP) when available, otherwise fork child blocked on a pipe, move it to cgroup, then release it to exec.
4. Attach bwrap profile/seccomp/net policy.
5. Start drain/reaper before releasing user code.

Fast-start fork bombs must fail under the configured pids limit; there is no attach-after-exec race.

---

Filesystem and cleanup policy

Workspace layout:

$HUMANWORK_SANDBOX_ROOT/runs/<org>/<run>/<op>/
  input/      read-only bind into sandbox
  output/     read-write, quota-bound
  logs/       read-write, quota-bound
  tmp/        read-write, quota-bound
  etc/        generated synthetic /etc files
  root/       synthetic root metadata/profile material

Safety requirements:

• Org/run/op path segments are normalized safe IDs; traversal, empty, absolute path, UTF-8 ambiguity, control characters, and separator injection reject.
• Per-op dirs are mode 0700; parent roots are not writable by sandbox UID.
• Cleanup uses fd-relative operations (openat2/RESOLVE_BENEATH/O_NOFOLLOW or Zig equivalent) and never follows symlinks.
• Malicious symlinks/hardlinks/FIFOs/devices created inside writable roots cannot cause cleanup to delete or open outside the op directory.
• Output/log collection enforces byte caps before ledger persistence.
• Cleanup failure never turns a failed op into success; success requires quota release and workspace teardown evidence.

Forbidden production binds:

• host home;
• .ssh;
• Docker socket;
• repo root by default;
• cloud credential files;
• /var/run, /run, /proc from host except explicitly generated proxy socket path;
• host /etc wholesale;
• sibling run directories;
• broad /sys.

Synthetic /etc only includes passwd, group, nsswitch.conf, plus read-only loader/TLS roots when present. TLS roots do not grant network access.

---

bwrap profile policy

The generated profile must be deterministic and hashable. Golden tests assert exact argv for every manifest profile.

Required production profile properties:

• --die-with-parent;
• --new-session;
• user namespace with sandbox UID/GID mapping;
• mount namespace;
• pid namespace;
• IPC namespace;
• UTS namespace;
• network namespace per network policy;
• tmpfs/synthetic root;
• sandbox-scoped /proc;
• minimal /dev only;
• no broad /sys;
• read-only runtime binary binds;
• read-only input binds;
• quota-bound writable output/tmp/log binds;
• synthetic /etc binds only;
• --clearenv and supervisor-generated minimal env;
• no new privileges;
• all capabilities dropped;
• seccomp BPF attached by FD;
• --info-fd for bwrap metadata plus --sync-fd as the mandatory pause gate when proxy-only netns setup is required. --info-fd alone does not pause user execution.

Profile hash must change when mounts, network, seccomp, limits, manifest, runtime binary path/hash, or env policy changes.

---

Seccomp policy

The inner bwrap policy is a generated classic seccomp BPF program passed by FD to bwrap --seccomp. The existing OCI JSON profile remains an optional outer container/K8s profile and cannot satisfy the inner evidence contract.

Production seccomp is default-deny per architecture (x86_64, aarch64). Unknown architecture is NO_GO.

Policy files:

runners/sandbox-supervisor/policy/seccomp/base-allowlist.json
runners/sandbox-supervisor/policy/seccomp/exec-argv-allowlist.json
runners/sandbox-supervisor/policy/seccomp/browser-action-allowlist.json

Every allowlist entry must name syscall, architecture, argument constraints where relevant, and reason. Tests must explicitly cover denial for:

• ptrace, process_vm_readv, process_vm_writev;
• mount, umount2, fsopen, fsmount, move_mount, open_tree, pivot_root from user code;
• setns, unshare, namespace-changing clone/clone3 flags;
• bpf, perf_event_open, keyring syscalls;
• io_uring_setup, io_uring_enter, io_uring_register unless explicitly required by a runtime profile;
• open_by_handle_at, fanotify_init, module loading, syslog, reboot;
• raw sockets, AF_PACKET, unsafe AF_NETLINK, and socket types outside manifest policy.

Denied probes assert syscall return/errno through native helper or ctypes with use_errno=True; exit-code-only checks are audit theater and fail review.

---

Network policy

network=off

• bwrap creates a network namespace with no usable route.
• No veth, host loopback, metadata endpoint, DNS, raw socket, or direct egress exists.
• Probes must attempt direct TCP, UDP, DNS, metadata IP, host localhost, and raw socket creation.

network=proxy-only

Production proxy-only is implemented, not left open:

1. Host runs a Humanwork egress proxy bound to a supervisor-owned port/socket and enforcing the operation manifest's egress allowlist.
2. Supervisor launches bwrap with network namespace using --info-fd for namespace metadata and --sync-fd to pause before user exec. --info-fd is not a pause primitive.
3. Supervisor reads the sandbox netns metadata from --info-fd, keeps --sync-fd unreleased, and configures a veth pair into the sandbox netns.
4. Supervisor's in-flight operation registry atomically leases one free /30 block from the proxy-only pool before veth/IP/route setup. Production default pool: 169.254.128.0/17 subdivided into non-overlapping /30 operation leases. Host side uses the first usable IP, sandbox side uses the second usable IP, network/broadcast addresses remain unused.
5. Lease ownership is tied to the op slot lifecycle and released only after netns teardown, route removal, and registry cleanup. Occupied /30 blocks are never reused, retried, or shared while an op is in flight.
6. Supervisor installs nftables/iptables rules in the sandbox netns allowing only TCP to the leased host-proxy IP:port and denying all other egress, metadata ranges, and host-local addresses.
7. Supervisor releases the sandbox to exec.
8. Evidence records proxy config hash, leased /30, host/sandbox IPs, and effective network proxy-only.

Host requirements: CAP_NET_ADMIN for supervisor, nftables/iptables availability, configured HUMANWORK_SANDBOX_PROXY_ENDPOINT, and an available proxy-only /30 lease. If any are absent and an op requests proxy-only, dispatch returns:

{
  "status": "denied",
  "reason": "proxy_only_unavailable",
  "requestedNetwork": "proxy-only",
  "effectiveNetwork": null
}

This is not a sandbox success and cannot be downgraded to off silently. Pool exhaustion, duplicate lease detection, ip addr add failure, route insertion conflict, or nftables rule conflict all return proxy_only_unavailable; the supervisor must never retry with an occupied range or ignore route/IP conflicts. Browser/network-capable operations that require proxy-only remain blocked until proxy-only readiness is GO.

---

Landlock policy

Landlock evidence object is always required.

• If host Landlock ABI is available, sandbox-init must apply rules before execing capability-shell; failure to apply is fail-closed.
• If unavailable, evidence is { "available": false, "enabled": false, "abi": null, "rulesetApplied": false, "reason": "kernel_unsupported" } and production may still succeed because mount namespace + bind policy + project quota remain the load-bearing filesystem boundary.
• If available but disabled by config in production, readiness is NO_GO.

---

Deployment model

Recommended production host: dedicated native/systemd worker host.

Systemd requirements:

• pinned bubblewrap package or pinned bundled bwrap binary with hash recorded;
• sandbox-supervisor and capability-shell installed under read-only runtime path;
• HUMANWORK_SANDBOX_ROOT on filesystem with project quotas enabled;
• cgroup v2 delegated to the service (Delegate=yes) with pids/memory/cpu/io controllers writable;
• seccomp enabled;
• no Docker socket mounted;
• CAP_NET_ADMIN only when proxy-only profile is enabled;
• supervisor Unix socket readable/writable only by sandbox-worker service user.

Dedicated K8s node requirements:

• node proves user/mount/pid/ipc/uts/net namespaces, seccomp, bwrap, cgroup v2 delegation, and workspace project quota;
• pod/container profile permits only the minimum primitives needed by supervisor;
• no cluster-wide Docker socket or host home bind;
• smoke job proves actual cgroup/quota/netns behavior on the node.

Railway, generic nested Docker, and locked-down PaaS workers are production NO_GO for hostile-code sandbox mode unless they prove every primitive above. Outer Docker/K8s may package the worker; it is not the product sandbox boundary.

---

Implementation tasks

Task 1: Lock ADR/design references and cosign gate

Objective: Ensure implementation cannot drift to direct spawn, bwrap-alone, Firecracker, or open-ended phases.

Files:

• Modify: docs/decisions/022-sandbox-substrate.md
• Modify: docs/architecture/HUMANWORK_SANDBOX_SUPERVISOR.md

Acceptance criteria: ADR states Humanwork supervisor + bwrap is target shipped substrate for hostile user-space isolation; Firecracker is separate-kernel contingency only; implementation tasks after Task 1 require cosign; no doc contains unresolved substrate decisions for proxy-only, Landlock, or host class.

Task 2: Add canonical protocol schemas and generated types

Objective: Make API/worker/supervisor agree on one wire contract.

Files:

• Create: runners/sandbox-worker/supervisor_protocol.py
• Create: runners/sandbox-worker/tests/test_supervisor_protocol.py
• Create: runners/sandbox-supervisor/policy/protocol.schema.json
• Create: runners/sandbox-supervisor/src/protocol.zig
• Create: runners/sandbox-supervisor/tests/protocol_test.zig
• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Test: api/test/sandbox-capability-executor.service.spec.ts

Acceptance criteria: Wire JSON is camelCase; unknown fields reject; env rejects; snake_case appears only in internal structs behind explicit conversion tests; result schema requires all production evidence fields including limits.cpuMax, limits.ioMax, and workspace/output/log caps.

Task 3: Add operation manifest

Objective: Centralize API op -> capability-shell op -> policy mapping.

Files:

• Create: runners/sandbox-supervisor/policy/operation-manifest.json
• Create: runners/sandbox-supervisor/src/operation_manifest.zig
• Create: runners/sandbox-supervisor/tests/operation_manifest_test.zig
• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Modify: runners/zig-capability-shell/src/main.zig if handler names do not match manifest

Acceptance criteria: Every API-allowed op has exactly one manifest entry; every manifest shell op has a real capability-shell handler; unmapped ops fail closed; browser_action is either implemented as a shell handler or explicitly adapted from existing browser handler before production cutover.

Task 4: Create long-lived supervisor daemon skeleton

Objective: Implement the production process model.

Files:

• Create: runners/sandbox-supervisor/build.zig
• Create: runners/sandbox-supervisor/src/main.zig
• Create: runners/sandbox-supervisor/src/server.zig
• Test: runners/sandbox-supervisor/tests/server_test.zig

Implementation detail: Support --version, --health --json, and --serve-unix $HUMANWORK_SANDBOX_SUPERVISOR_SOCK. --run-json may exist only as a test/smoke helper and must be rejected as production worker mode.

Acceptance criteria: Unix socket server starts with restrictive permissions; health reports primitive readiness; daemon owns in-flight registry; tests prove --run-json is not used by production worker config.

Task 5: Add worker-side supervisor client

Objective: Replace direct subprocess policy with typed daemon RPC.

Files:

• Modify: runners/sandbox-worker/worker.py
• Create: runners/sandbox-worker/supervisor_client.py
• Test: runners/sandbox-worker/tests/test_supervisor_client.py

Acceptance criteria: Worker sends canonical dispatch JSON over Unix socket; request size/time caps remain; supervisor socket path is fixed config, not user input; network errors produce not-ready/fail-closed responses.

Task 6: Enforce supervisor-owned evidence envelope

Objective: Prevent sandboxed code from forging substrate proof.

Files:

• Create: runners/sandbox-supervisor/src/evidence.zig
• Test: runners/sandbox-supervisor/tests/evidence_test.zig
• Test: runners/sandbox-worker/tests/test_malicious_child_evidence.py

Acceptance criteria: Only supervisor code can set evidence fields; child stdout/stderr containing fake evidence remains payload; API rejects fake evidence from worker fakes; hashes derive from exact manifest/profile/seccomp/runtime bytes used.

Task 7: Package supervisor runtime assets

Objective: Make the shipped worker contain every binary/profile it needs.

Files:

• Modify/create: runners/sandbox-worker/Dockerfile
• Modify/create: runners/sandbox-worker/Makefile
• Modify: package.json or repo build script if present
• Create: runners/sandbox-supervisor/policy/seccomp/

Acceptance criteria: Build artifact/image includes sandbox-supervisor, capability-shell, pinned bwrap or pinned package requirement, operation manifest, seccomp profile assets, probe helpers, and version metadata; missing asset makes worker health NO_GO.

Task 8: Add workspace manager with quotas and race-safe cleanup

Objective: Bound writable state and make cleanup non-exploitable.

Files:

• Create: runners/sandbox-supervisor/src/workspace.zig
• Test: runners/sandbox-supervisor/tests/workspace_test.zig

Acceptance criteria: Safe ID normalization; 0700 dirs; project quota applied before exec; cleanup uses fd-relative no-follow traversal; malicious symlink/hardlink/FIFO/device cleanup probes cannot escape op dir; quota release evidenced.

Task 9: Generate synthetic rootfs and /etc

Objective: Support dynamic-linked runtime binaries without broad host binds.

Files:

• Create: runners/sandbox-supervisor/src/rootfs.zig
• Test: runners/sandbox-supervisor/tests/rootfs_test.zig

Acceptance criteria: Synthetic root contains only declared files/dirs; generated /etc/passwd, /etc/group, nsswitch.conf are sandbox-only; loader/TLS roots are read-only and do not imply network; host .ssh, cloud creds, Docker socket, repo root, broad /etc, broad /run, broad /sys absent.

Task 10: Implement deterministic bwrap profile builder

Objective: Generate the exact sandbox argv/profile.

Files:

• Create: runners/sandbox-supervisor/src/bwrap_profile.zig
• Test: runners/sandbox-supervisor/tests/bwrap_profile_test.zig

Acceptance criteria: Golden tests assert required namespace/hardening flags; forbidden host binds never appear; profile hash changes when mounts/network/limits/seccomp/manifest/runtime/env policy changes; all caps dropped; no-new-privs enforced.

Task 11: Add host primitive readiness probe

Objective: Block deployment on hosts that cannot enforce the boundary.

Files:

• Create: runners/sandbox-supervisor/src/probe.zig
• Test: runners/sandbox-supervisor/tests/probe_test.zig

Acceptance criteria: Probe reports bwrap path/version/hash, user/mount/pid/ipc/uts/net namespaces, cgroup v2 controllers, cgroup.kill, seccomp BPF support, project quotas, Landlock ABI, nftables/proxy-only primitives, and runtime asset hashes. Production health is NO_GO when any required primitive for the enabled profile is missing.

Task 12: Add cgroup v2 manager with race-free attach

Objective: Enforce pids/memory/cpu/io before user code starts.

Files:

• Create: runners/sandbox-supervisor/src/cgroup.zig
• Test: runners/sandbox-supervisor/tests/cgroup_test.zig

Acceptance criteria: Writes pids.max, memory.max, cpu.max, io.max; spawn enters cgroup before exec via clone3(CLONE_INTO_CGROUP) or paused-child attach; fast-start fork bomb is contained; cleanup removes cgroup; missing controller is NO_GO.

Task 13: Add workspace storage and output/log caps

Objective: Prevent disk exhaustion and huge evidence payloads.

Files:

• Modify: runners/sandbox-supervisor/src/workspace.zig
• Create: runners/sandbox-supervisor/src/output_caps.zig
• Test: runners/sandbox-supervisor/tests/output_caps_test.zig

Acceptance criteria: Workspace quota enforces workspaceMaxBytes; output/log caps enforce configured bytes; disk-fill probe fails with intended reason; stdout/stderr drain caps kill/report output_cap_exceeded; ledger never stores oversized raw output.

Task 14: Generate default-deny bwrap seccomp BPF

Objective: Deny syscall escape surface with a real inner profile.

Files:

• Create: runners/sandbox-supervisor/src/seccomp.zig
• Create: runners/sandbox-supervisor/policy/seccomp/base-allowlist.json
• Create: runners/sandbox-supervisor/policy/seccomp/exec-argv-allowlist.json
• Create: runners/sandbox-supervisor/policy/seccomp/browser-action-allowlist.json
• Test: runners/sandbox-supervisor/tests/seccomp_test.zig

Acceptance criteria: Generated BPF is passed by FD to bwrap; seccomp hash is exact BPF bytes; default action is deny; allowlist is arch-specific; dangerous syscall probes fail with expected errno; OCI JSON is not accepted as inner profile evidence.

Task 15: Add sandbox-init and Landlock helper

Objective: Apply filesystem defense-in-depth inside bwrap before capability-shell exec.

Files:

• Create: runners/sandbox-supervisor/src/sandbox_init.zig
• Create: runners/sandbox-supervisor/src/landlock.zig
• Test: runners/sandbox-supervisor/tests/landlock_test.zig

Acceptance criteria: If Landlock ABI is available, ruleset applies before exec or op fails; if unavailable, explicit evidence records kernel_unsupported; host-path read/write path-trick probes fail when ABI exists.

Task 16: Implement network=off

Objective: Make no-network mode real.

Files:

• Create: runners/sandbox-supervisor/src/network.zig
• Test: runners/sandbox-supervisor/tests/network_off_test.zig

Acceptance criteria: Direct TCP/UDP/DNS/metadata/localhost/raw-socket probes fail under off; effective network evidence is off; no proxy config is consulted.

Task 17: Implement proxy-only

Objective: Make proxy-only finite and fail-closed.

Files:

• Modify: runners/sandbox-supervisor/src/network.zig
• Create: runners/sandbox-supervisor/src/proxy_netns.zig
• Test: runners/sandbox-supervisor/tests/network_proxy_only_test.zig

Acceptance criteria: With proxy config and CAP_NET_ADMIN, supervisor launches bwrap with --info-fd and --sync-fd, keeps --sync-fd unreleased until the veth/IP/routes/nftables allow-only-proxy path is committed, leases a unique /30 from the thread-safe registry pool, and evidence includes proxy config hash plus leased /30/host/sandbox IPs; tests fail if --info-fd is present without --sync-fd; concurrent proxy-only ops receive non-overlapping leases; lease release follows op teardown; pool exhaustion or IP/route conflict returns proxy_only_unavailable; direct egress/metadata/localhost still fail; no silent downgrade to off.

Task 18: Add process runner and reaper

Objective: Execute bwrap/capability shell and guarantee teardown.

Files:

• Create: runners/sandbox-supervisor/src/process.zig
• Test: runners/sandbox-supervisor/tests/process_test.zig

Acceptance criteria: Runner starts output drain before exec release; timeout/cancel kills full process tree using process group and cgroup kill; no child remains alive; exit/duration/output bytes captured; cleanup evidence accurate.

Task 19: Add in-flight operation registry and cancel RPC

Objective: Support durable cancellation from API through worker to supervisor.

Files:

• Create: runners/sandbox-supervisor/src/registry.zig
• Modify: runners/sandbox-supervisor/src/server.zig
• Modify: runners/sandbox-worker/worker.py
• Test: runners/sandbox-worker/tests/test_worker_cancel.py
• Test: runners/sandbox-supervisor/tests/registry_test.zig

Acceptance criteria: POST /v1/sandbox/ops/:id/cancel is idempotent; cancel active op reaps exact cgroup while original /ops request is in flight; cancel unknown op returns idempotent status; registry owns thread-safe proxy-only /30 pool take/release semantics; concurrent lease tests prove no overlapping ranges and pool exhaustion returns proxy_only_unavailable; race tests cover complete-before-cancel and cancel-before-start.

Task 20: Wire worker to supervisor and delete production direct spawn

Objective: Remove worker.py -> capability-shell production execution.

Files:

• Modify: runners/sandbox-worker/worker.py
• Test: runners/sandbox-worker/tests/test_worker.py

Acceptance criteria: Production worker invokes supervisor daemon only; AST/grep test fails on production subprocess.run([SHELL or equivalent direct capability-shell path; test fakes exist only behind test-only dependency injection and cannot satisfy production evidence gates.

Task 21: Move executable allowlist out of request env

Objective: Stop treating env as policy transport.

Files:

• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Modify: runners/sandbox-worker/worker.py
• Modify: runners/zig-capability-shell/ if it still reads only env policy
• Test: api/test/sandbox-capability-executor.service.spec.ts

Acceptance criteria: Caller-provided env rejects; allowedExecutables is required field; empty list deny-all; entries must be canonical in-sandbox absolute executable paths; manifest aliases, bare names, PATH lookup, wildcard/glob entries, relative paths, and host paths outside the sandbox mount namespace reject; child-spawning ops reject unless manifest supplies a non-empty approved allowlist; supervisor may generate minimal internal env for capability-shell but never passes caller env through.

Task 22: Align capability-shell operation handlers

Objective: Ensure the sandbox executor implements exactly what the manifest exposes.

Files:

• Modify: runners/zig-capability-shell/src/main.zig
• Modify/create tests under runners/zig-capability-shell/tests/

Acceptance criteria: Shell has real handlers for every manifest shell op; handler names match manifest or explicit adapter; no fallback/default handler executes arbitrary command; shell output schema is payload-only, not substrate evidence.

Task 23: Require supervisor evidence in API production mode

Objective: Prevent fake or unsandboxed worker success.

Files:

• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Test: api/test/sandbox-capability-executor.service.spec.ts

Acceptance criteria: Production evidence mode is enabled when NODE_ENV=production or SANDBOX_REQUIRE_SUPERVISOR_EVIDENCE=true; status=ok requires complete evidence; missing/inconsistent evidence fails closed; sandboxBackend != linux-bwrap-supervisor fails closed; no production degraded-success mode.

Task 24: Map worker statuses to API exceptions and ledger event types

Objective: Stop treating denied/error/timeout as success.

Files:

• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Modify: api/src/runtime-control-plane/sandbox-capability-gateway.service.ts
• Test: api/test/sandbox-capability-gateway.service.spec.ts

Acceptance criteria: ok -> success ledger; denied -> ForbiddenException and blocked ledger; timeout -> timeout exception and failed ledger; cancelled -> cancelled ledger; error/unknown -> bad gateway and failed ledger; sanitized reasons only.

Task 25: Persist substrate metadata in run ledger

Objective: Make sandbox evidence inspectable after execution without leaking payloads/secrets.

Files:

• Modify: api/src/runtime-control-plane/sandbox-capability-gateway.service.ts
• Test: api/test/sandbox-capability-gateway.service.spec.ts

Acceptance criteria: Ledger stores redacted backend/version/build/profile/seccomp/manifest/Landlock/limits/network/cleanup metadata; never stores raw secrets or oversized stdout/stderr; enough evidence remains to reconstruct which boundary ran each op.

Task 26: Add direct hostile-code probes

Objective: Prove the substrate against adversarial user-space behavior.

Files:

• Modify: agent/evals/test_sandbox_capability_ops.py
• Create: runners/sandbox-supervisor/tests/probes/

Acceptance criteria: Probes attempt path escape, symlink cleanup escape, hardlink cleanup escape, /proc sibling inspection, ptrace, mount, setns, unshare, bpf, io_uring, keyring, fork bomb, memory pressure, CPU burn, disk fill, high IO, raw egress, metadata, host localhost, raw sockets, env dump, rootfs write, executable allowlist bypass, and malicious child fake-evidence. Every probe asserts intended denial reason/layer.

Task 27: Add audit-theater guards

Objective: Prevent false green tests.

Files:

• Modify: agent/evals/test_sandbox_capability_ops.py
• Create: runners/sandbox-supervisor/tests/probes/native/

Acceptance criteria: Syscall probes call syscalls directly via native helpers or ctypes(use_errno=True); tests assert probe code actually ran; wide-open substrate makes the deny battery fail; no os.system("not-a-real-command") style pass is accepted.

Task 28: Add bwrap profile parity tests against Rig reference

Objective: Reuse Rig mechanics without importing host conveniences.

Files:

• Create: runners/sandbox-supervisor/tests/backend_parity_test.zig

Acceptance criteria: Parity covers namespace flags, tmpfs root, read-only runtime binds, workspace binds, network-off/proxy-only semantics; tests explicitly assert Humanwork does not bind .ssh, broad repo root, host home, Docker socket, host /etc, or broad /run.

Task 29: Wire build/image/systemd/K8s deployment

Objective: Make the substrate deployable on real hosts.

Files:

• Modify/create: runners/sandbox-worker/Dockerfile
• Create: deploy/systemd/humanwork-sandbox-worker.service
• Create: deploy/systemd/humanwork-sandbox-supervisor.service
• Modify: infra/k8s/sandbox-worker.yaml
• Create: infra/k8s/sandbox-worker-node-requirements.md

Acceptance criteria: Native/systemd install is documented and testable; bwrap pinned; cgroup delegation configured; project quotas required; supervisor socket permissions correct; K8s manifest states exact node/primitives and remains NO_GO without them; no deployment mounts Docker socket or host home.

Task 30: Add readiness smoke script

Objective: Block rollout on missing host primitives or fake evidence.

Files:

• Create: api/scripts/smoke-sandbox-supervisor.ts
• Create/modify: .github/workflows/sandbox-substrate-smoke.yml

Acceptance criteria: Script calls worker health, validates primitive readiness, runs minimal artifact_read/exec_argv op, validates full evidence schema, runs deny subset, and emits GO only on real eligible hosts. GitHub-hosted runner lacking primitives reports NO_GO/skipped host gate, not green production readiness.

Task 31: Add CI workflow for deterministic units

Objective: Keep implementation regressions out before host-gated smoke.

Files:

• Create/modify: .github/workflows/sandbox-supervisor.yml

Acceptance criteria: CI runs zig build test in supervisor, Zig capability-shell tests, Python worker tests, API executor/gateway specs, schema generation drift checks, seccomp allowlist validation, and static bypass grep/AST checks.

Task 32: Add operator runbook and rollback path

Objective: Give operators exact GO/NO_GO and rollback criteria.

Files:

• Create: docs/runbooks/sandbox-worker-supervisor.md
• Modify: docs/_archive/RIG_RUNTIME_HARD_CUT_CUTOVER.md (archived 2026-06-04 — any further updates land in the archive)

Acceptance criteria: Runbook covers native/systemd and dedicated K8s node deployment, bwrap pinning, cgroup delegation, project quotas, proxy-only primitives, Landlock evidence, smoke commands, rollback by routing away/disabling sandbox capability execution, and explicitly forbids rollback to unsandboxed direct spawn.

Task 33: Add production cutover gate

Objective: Prevent partial substrate from serving production hostile-code traffic.

Files:

• Modify: deployment config/docs for sandbox worker
• Modify: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
• Test: API production gate tests

Acceptance criteria: Production cutover requires ADR cosign, host smoke GO, CI units green, full evidence validator enabled, direct-spawn bypass test green, and hostile-code deny battery green on eligible host. Missing any gate keeps production disabled.

Task 34: Add post-cutover bypass audit

Objective: Catch residual direct-spawn or fake-backend paths after wiring.

Files:

• Create: api/scripts/audit-sandbox-bypass.ts
• Test: script fixture tests if repo has script tests

Acceptance criteria: Audit scans API, worker, deploy manifests, env samples, and runbook for backend=none, sandbox=off, direct capability-shell, raw bwrap success, degraded-success, broad host binds, and env passthrough. Any production bypass pattern fails CI.

Task 35: Final hostile-code acceptance run

Objective: Prove the final point of the whole sandboxing requirement.

Files:

• Modify: PR body / release checklist if used
• Persist: smoke output artifact from eligible host

Acceptance criteria: On an eligible worker host, the full matrix passes: filesystem, symlink cleanup, network off, proxy-only, process isolation, syscall deny, env scrub, executable allowlist, resource limits, IO/storage limits, timeout, cancel, cleanup, evidence schema, ledger metadata, and bypass audit. Production evidence from the run is attached to the release/cutover record.

---

Final acceptance criteria

• ADR-022 records Humanwork Supervisor + bwrap backend as the target shipped substrate, with Firecracker as separate-kernel contingency only.
• The architecture doc has no open sandbox substrate questions; proxy-only, Landlock, host class, cgroup, storage, and seccomp are final decisions.
• No production operation path invokes the capability shell outside the supervisor.
• No production operation path can bypass the supervisor.
• No production config/env can select none, direct spawn, raw-bwrap success, or degraded-success backend as accepted evidence.
• Supervisor production mode is a long-lived Unix-socket daemon with in-flight registry and cancel RPC.
• Canonical wire schema is camelCase, strict, versioned, and tested API <-> worker <-> supervisor.
• Supervisor constructs evidence; child output cannot forge backend, hashes, limits, cleanup, or versions.
• Operation manifest maps every API op to exact capability-shell handler and policy.
• allowedExecutables=[] is deny-all; child-spawning ops require explicit non-empty manifest-approved allowlist of canonical in-sandbox absolute executable paths only; manifest aliases, bare names, PATH lookup, wildcard/glob entries, and relative paths reject.
• Production API fails closed if worker response lacks complete linux-bwrap-supervisor evidence.
• Worker readiness fails when bwrap, namespaces, cgroup v2 pids/memory/cpu/io, project quotas, seccomp, runtime assets, or required proxy-only primitives are missing.
• bwrap profile uses user/mount/pid/ipc/uts/net namespaces, synthetic root, minimal /dev, sandbox /proc, no broad host binds, no-new-privs, dropped caps, and seccomp FD.
• Cgroup pids/memory/cpu/io and workspace quota are applied before user code execs.
• Seccomp inner profile is generated default-deny BPF, not OCI JSON theater.
• Landlock evidence is always present; available Landlock ABI must be applied or fail closed.
• network=off denies all direct egress; proxy-only is explicit --sync-fd-paused veth/nftables/proxy or proxy_only_unavailable.
• Workspace cleanup is fd-relative/no-follow and race-safe against symlink/hardlink/device tricks.
• Hostile-code deny battery proves filesystem, network, process, syscall, env, executable allowlist, resource, IO/storage, timeout, cancel, cleanup, and evidence trust behavior.
• CI units and host-gated smoke cannot report fake green production readiness on ineligible GitHub runners.
• Rollback never means unsandboxed direct spawn.