Skip to main content

ADR-022: Sandbox substrate for runtime capability execution

Date: 2026-05-28 Status: Target locked for implementation; production cutover is NO_GO until AlexD / Paul / Eusden cosign is actually recorded. Humanwork Sandbox Supervisor with supervised bubblewrap backend is the target shipped substrate for hostile user-space runtime isolation.

2026-06-01 — Renumbered from ADR-008 → ADR-022 to resolve a numbering collision with ADR-008 (KB architecture K1 + K2). Previously lived at docs/architecture/decisions/008-sandbox-substrate.md; the secondary docs/architecture/decisions/ ADR root has been folded into the primary docs/decisions/ tree.

2026-06-01 — Discussion update: Slack thread 1780054357 (cosign ask) saw pushback on Firecracker/gVisor. Paul challenged necessity/complexity; AlexD proposed skipping Firecracker/gVisor and writing a small controlled layer like the existing capability shell.

2026-06-02 — Current proposal: ship a Humanwork-owned supervisor substrate that uses bubblewrap as the Linux isolation mechanism. The self-written part is the product/runtime contract: policy translation, profile generation, workspace materialization, cgroups, proxy/no-network semantics, cancellation/reaper, evidence, and fail-closed API verification. bubblewrap stays load-bearing as the namespace/mount/seccomp executor; it is not dropped.

2026-06-03 — Implementation lock: Tasks may build the supervisor substrate, but production readiness remains NO_GO while ADR cosign is pending. This repository does not record AlexD / Paul / Eusden cosign yet; runtime health must report that honestly instead of fabricating signatures or treating implementation progress as approval.

Production cutover / cosign gate​

Production hostile-code sandbox traffic is disabled unless all of these are true:

  1. AlexD, Paul, and Eusden cosign ADR-022 in repo history or an approved release record.
  2. Host smoke reports GO on an eligible worker host.
  3. CI units, schema drift checks, direct-spawn bypass checks, evidence validation, and hostile-code deny battery are green.
  4. The cutover verification envelope is GitHub-provider verified and signed by an out-of-repo Ed25519 cutover attestation key; supervisor health verifies it with configured trusted public key(s), not local JSON shape.
  5. Worker/supervisor health reports complete host primitive readiness.

The current state is NO_GO: adr_cosign_pending. No code path may convert that pending state into production readiness with an env flag, fake signature, unsigned provider-shaped envelope, direct spawn fallback, raw bwrap success, or degraded-success backend.

Context​

Runtime runners now route file/process/browser/network operations through typed capability operations and a compiled capability shell. That removes raw shell execution from the Python adapter, but it is not enough by itself: hostile user-space code can still attempt syscalls, process escape, filesystem traversal, resource exhaustion, and network exfiltration unless the worker owns a real OS isolation boundary.

The architecture deck rule remains explicit: do not fake sandboxing in the API process. If the deployment target cannot provide required Linux primitives, run sandbox workers somewhere that can.

Current dev state at the time of this proposal:

  • api/src/runtime-control-plane/sandbox-capability-executor.service.ts validates typed ops, rejects raw shell-shaped input, caps time/output, and dispatches to a remote worker through SANDBOX_WORKER_URL.
  • runners/zig-capability-shell/ enforces path checks, argv-only exec, scrubbed child env, and executable allowlists.
  • runners/sandbox-worker/worker.py is a Python HTTP adapter that directly runs the Zig capability shell with subprocess.run; the new target replaces that direct spawn with supervisor-owned sandbox launch.
  • runners/sandbox-worker/seccomp.json, infra/k8s/sandbox-worker.yaml, and firecracker/vmconfig.json are scaffolds; they do not by themselves provide the product substrate contract.

Options​

OptionBoundaryReuseOps burdenVerdict
A. Docker/task boundary onlyContainer runtime policy; no Humanwork per-op contractMediumLowNot enough; inherits Docker/platform limits
B. Firecracker microVMSeparate guest kernel + VM boundaryLow unless snapshottingMedium-highContingency only; not a roadmap follow-up
C. Hardened K8s podPod seccomp/NetworkPolicy/rootfs hardeningMediumMedium-highUseful deployment wrapper, not product substrate
D. gVisor/runscUser-space kernelMediumLow-mediumContested; compatibility/ops surface for default path
E. bwrap aloneNamespaces/mount jail via external binaryMediumLowMechanism only; missing product contract/evidence/reaper
F. Humanwork Supervisor + bwrap backendbwrap kernel primitives + Humanwork lifecycle/evidence/cgroups/proxy contractHighMediumRecommended target substrate
G. Humanwork direct Linux backendReimplement bwrap-style primitive orchestration ourselvesHighHighFuture fallback only if bwrap becomes a proven blocker
H. WASI/WasmtimeCapability model for WASM modulesHigh for WASM onlyMediumFuture plugin lane, not general Python/Node/browser runtime

Feasibility and decision quality​

Feasible: yes, if the target is hostile user-space runtime isolation and not kernel-escape isolation.

Good decision: yes, with gates, because it matches the existing layered architecture better than Firecracker as a default substrate and avoids prematurely reimplementing a namespace/mount jailer.

  • The control plane, manifest, no-ambient-secret policy, and Zig capability shell already remove the broad authority path.
  • The missing layer is a deterministic worker-side OS boundary and lifecycle contract.
  • bubblewrap gives the Linux primitive composition we need today: user/pid/mount/net namespaces, bind plan, tmpfs root, --die-with-parent, --new-session, seccomp FD support, and no host Docker daemon dependency.
  • Humanwork-owned supervisor gives the part bwrap does not: policy translation, evidence schema, cgroups, proxy semantics, operation registry, cancellation, process-tree reaper, TTL cleanup, and API fail-closed integration.
  • The only Firecracker-only property is a separate guest kernel. That should be an explicit escalation requirement, not an assumed roadmap follow-up.

Risk: sandboxing is sharp. The decision stays good only if implementation is evidence-driven: no production bypass, hostile probes, fail-closed production mode, no unsandboxed fallback, and explicit host-primitives readiness.

Decision proposal​

Target substrate: Humanwork Sandbox Supervisor with supervised bubblewrap backend.

The supervisor is a shipped Humanwork service/binary. It receives already-authorized typed operations from the worker endpoint, materializes a per-run/per-op workspace, builds a deterministic bwrap profile, attaches cgroups and optional supervisor-side preflight checks, launches a sandbox-init/capability-shell process under bwrap, captures bounded output, emits structured events, and tears down the slot.

bubblewrap is part of the target solution, not something we dropped. The boundary is:

API → sandbox-worker → Humanwork supervisor → bwrap sandbox → sandbox-init/capability-shell

The Humanwork-owned part is the substrate contract and enforcement glue. bubblewrap is the Linux primitive executor. A direct Linux backend is not the initial target; it only reopens if bwrap itself becomes a measured blocker after the supervisor contract and deny battery exist.

Firecracker feature coverage we need​

The goal is not to recreate a microVM. The goal is to cover the Firecracker properties Humanwork actually needs for the runtime threat model with a smaller supervisor+bwrap substrate.

Firecracker property people wantedSupervisor + bwrap coverage
Per-run clean machine shapeper-op/per-run sandbox slot + synthetic rootfs + no mutable cross-run state
Filesystem isolationbwrap mount namespace + pivot_root-style tmpfs root + bind allowlist + read-only root + optional Landlock via sandbox-init
Process isolationbwrap pid namespace + sandbox-scoped /proc + process-group/cgroup reaper
Resource containmentsupervisor-owned cgroups v2 pids/memory/cpu/io limits + timeout/cancel
Network isolationbwrap net namespace off or explicit proxy-only path; no host network fallback
Syscall reductionbwrap --seccomp profile + no_new_privs + dropped capabilities
No ambient secretsworker/supervisor env scrub + generated minimal /etc + no host credential mounts
Auditabilitysupervisor version, bwrap version, profile hash, seccomp hash, limits, denial reason, cleanup evidence

The one Firecracker feature we are not covering ourselves is the separate guest kernel / hypervisor boundary. That is not needed for the current Humanwork runtime target unless kernel-escape resistance becomes an explicit product/security requirement.

No-bypass invariant​

The supervisor is not optional in production. The final design must make bypass structurally impossible:

  • runners/sandbox-worker/worker.py must not execute capability-shell directly for production operations.
  • There is no production backend=none, sandbox=off, degraded-success, or direct-spawn escape hatch.
  • linux-bwrap-supervisor is the only backend that can satisfy production evidence gates.
  • Raw bwrap invocation without Humanwork supervisor evidence is not accepted.
  • API production mode accepts only sandboxBackend=linux-bwrap-supervisor plus required evidence fields.
  • Timeout/cancel cleanup is owned by the supervisor/cgroup boundary, not by API best-effort cleanup alone.

Threat-model boundary​

The hostile-code claim is precise:

  • In scope: adversarial user-space code trying to read host files, dump env, escape workspace paths, fork bomb, inspect sibling processes, open raw network egress, abuse allowed interpreters, or bypass typed capability policy.
  • Required boundary: bwrap user/mount/pid/ipc/net namespaces, synthetic/minimal root, seccomp syscall policy, cgroups v2 limits, no ambient secrets, no host convenience mounts, proxy-only/no-network enforcement, process-tree reaper, and event evidence.
  • Defense in depth: sandbox-init may add Landlock where available before execing the capability shell.
  • Out of scope: a kernel zero-day / malicious kernel-level escape class that requires a separate guest kernel. That is the only real Firecracker-shaped reason to reopen the substrate decision.

Consequences​

  • runners/sandbox-worker/worker.py stops invoking the capability shell directly and delegates to the supervisor.
  • Add runners/sandbox-supervisor/ as a shipped long-lived Humanwork daemon/service with a Unix-socket RPC surface, in-flight registry, cancel RPC, and linux-bwrap-supervisor backend.
  • Add bwrap as an explicit worker-host dependency, pinned and probed by readiness checks; add cgroup v2 pids/memory/cpu/io, project quota, seccomp BPF, and proxy-only primitive readiness gates.
  • Worker hosts must support bwrap requirements: unprivileged user namespaces or an approved equivalent, mount namespaces, cgroups v2, seccomp, and required filesystem layout. If a platform blocks those primitives, deploy sandbox workers on a native/systemd or dedicated K8s node class that permits them.
  • Firecracker is not a planned follow-up. Reopen it only if proof gates fail or a separate guest-kernel threat model becomes product scope.

Acceptance gates​

  • Raw shell remains impossible.
  • Production worker never invokes capability-shell directly; it talks to the long-lived supervisor daemon over the Unix socket.
  • Canonical protocol is strict camelCase JSON; caller-provided env and unknown fields reject.
  • Operation manifest maps every API op to a concrete capability-shell handler, network policy, workspace policy, and executable allowlist requirement.
  • Empty allowedExecutables is deny-all; child-spawning ops require explicit non-empty manifest-approved allowlists of canonical in-sandbox absolute executable paths only. Manifest aliases, bare executable names, PATH lookup, wildcard/glob entries, relative paths, and host paths outside the sandbox mount namespace reject before dispatch.
  • Env secret scrub is proven inside the sandboxed process.
  • Supervisor constructs evidence; sandboxed child output cannot forge sandboxBackend, hashes, versions, limits, cleanup, or Landlock fields.
  • ptrace, process VM syscalls, mount, new mount API, setns, namespace-changing clone/clone3, unshare, bpf, io_uring, keyring syscalls, raw sockets, raw egress, metadata endpoints, and host-local network probes are denied.
  • Network modes are enforced as off or explicit proxy-only; no silent downgrade and no host network fallback.
  • proxy-only is supervised netns/veth + proxy + nftables allow-only-proxy with a thread-safe per-op /30 lease from 169.254.128.0/17, or dispatch returns proxy_only_unavailable on missing primitives, pool exhaustion, or IP/route conflict.
  • Root filesystem is synthetic/read-only; writes are limited to quota-bound output/tmp/log slots.
  • Workspace cleanup is fd-relative/no-follow and race-safe against symlink/hardlink/device tricks.
  • Timeout/cancel kills the full sandbox process tree and cgroup subtree and leaves no live child.
  • Cgroup pids/memory/cpu/io limits and workspace quota are applied before user code execs.
  • Events include supervisor version/build, bwrap version/path, capability-shell hash, operation manifest hash, profile hash, seccomp hash, limits including cpuMax, ioMax, stdioMaxBytes, workspaceOutputMaxBytes, and workspaceLogsMaxBytes, optional/explicit Landlock status, exit code, duration, stdio byte counts, denial reason, and canonical cleanup result (processTreeReaped, cgroupKilled, cgroupRemoved, workspaceTornDown, quotaReleased).
  • Hostile-code deny battery proves the boundary with direct probes, not exit-code theater.
  • Host readiness is NO_GO when bwrap, namespaces, cgroup v2 controllers, project quotas, seccomp BPF, required proxy-only primitives, or runtime assets are missing.
  • CI cannot report fake production green on ineligible GitHub runners; host-gated smoke must distinguish NO_GO from GO.
  • No production operation path can bypass the supervisor, select an unsandboxed backend, accept raw bwrap evidence, or report degraded sandbox success.
  • Rollback never routes to unsandboxed direct spawn.

Canonical cleanup evidence object:

{
  "processTreeReaped": true,
  "cgroupKilled": false,
  "cgroupRemoved": true,
  "workspaceTornDown": true,
  "quotaReleased": true
}

This object is required in every supervisor result. The API validator must check presence and boolean type for all five fields. For status=ok, processTreeReaped, cgroupRemoved, workspaceTornDown, and quotaReleased must be true; cgroupKilled is required factual evidence and may be false on normal exit or true when the supervisor used cgroup subtree kill during timeout/cancel/reaper fallback.

References and source-of-truth chain​

Authoritative Humanwork docs:

Code seams on dev this ADR intentionally targets:

  • API dispatch/evidence seam: api/src/runtime-control-plane/sandbox-capability-executor.service.ts
  • API ledger seam: api/src/runtime-control-plane/sandbox-capability-gateway.service.ts
  • Current direct spawn path to remove: runners/sandbox-worker/worker.py
  • Existing typed executor to keep: runners/zig-capability-shell/
  • Existing scaffolds to either absorb or supersede: runners/sandbox-worker/seccomp.json, infra/k8s/sandbox-worker.yaml, runners/sandbox-worker/firecracker/vmconfig.json

External / adjacent references:

Last updated on by Degen