Skip to main content

RIG Runtime Hard-Cut Cutover Runbook

Status (2026-06-04): Cutover status is tracked in PR #1559 (dev→staging→main promotion flow) and ADR-028 (retire-agent-runtime-per-org-railway, impl PR #1584 merged). The legacy /v1/tasks authority returns HTTP 410 Gone (agent/routers/tasks.py); shared-Railway + ecs_per_org are the only supported AgentRuntimeKinds. Reuse this runbook only as a rehearsal template or post-mortem reference — do not re-enable legacy authority paths.

This runbook cuts production/staging to the Nest runtime control plane as the single authority for runs, events, approvals/input, tools, artifacts, model calls, runner claims, sandbox capability ops, and unresolved ingress.

Do not use this to re-enable legacy authority paths. If a gate fails, contain or forward-fix through runtime rows/events; do not fall back to /tools/execute, legacy Python task authority, org-less execution, raw shell, broad runner secrets, or persisted signed artifact URLs.

Roles​

  • Cutover lead: owns freeze/reopen calls and go/no-go.
  • API operator: runs migrations, backfill, smoke checks, and DB evidence capture.
  • Runner operator: verifies manifests, registration, heartbeats, and runner bootstrap gate.
  • Frontend/Ops checker: verifies Ops/Expert/client surfaces are reading runtime DTOs.
  • Incident lead: owns containment, rollback criteria, and forward-fix execution.

Required evidence​

Record these in the release ticket before reopening traffic:

  • Git SHA and deployed API/agent/frontend versions.
  • Migration job output with SYNCHRONIZE=false.
  • Freeze start time and freeze high-water mark.
  • Backfill dry-run output before apply.
  • Backfill apply output, if this is the real cutover and not rehearsal.
  • Smoke script JSON output.
  • Runner manifest IDs, ready runner registration IDs, and heartbeat timestamps.
  • Screenshots or logs for /ops/agent-runs, /ops/approvals, Expert approval surface, and client approval/input cards.
  • Quarantine sample for at least one unresolved/unknown ingress path in staging rehearsal.

Pre-freeze prerequisites​

All are blockers:

  1. API build passes.
  2. Postgres migrations pass against staging/production-like Postgres with SYNCHRONIZE=false.
  3. Legacy authority routes are disabled or return 410 Gone in production mode.
  4. Runtime action paths append run_events and write scoped agent_runs.
  5. RLS/session context is fail-closed unless app.current_role_domain = 'internal' is explicitly set by a worker/backfill/smoke transaction.
  6. Runner event signatures are verified before runner-originated events persist.
  7. RUNNER_JWT_SECRET is set per environment, separate from user JWT_SECRET; runner tokens must carry aud=runner.
  8. Runner/sandbox env does not include broad provider/vendor/R2/platform secrets.
  9. Artifact code issues short-lived leases only at response time; no signed URLs are persisted.
  10. Unknown/mismatched channel ingress is quarantined in unresolved_channel_events and does not execute runtime work.
  11. Ops/Expert/client UI can read runtime runs and approval/input requests.

Freeze and high-water marks​

  1. Announce the freeze window.
  2. Disable client-visible auto-send and any non-runtime mutation entry points at the edge/feature-flag layer.
  3. Put runners into drain mode or stop new runner claims. Existing in-flight work may finish only if it is already runtime-backed.
  4. Stop or pause webhook workers that can create conversations/tasks/queue items outside the runtime path. Webhooks may still ACK and quarantine unknown ingress.
  5. Capture the high-water mark from the DB:
SELECT now() AS freeze_high_water_created_at;
SELECT max(created_at) AS conversations_high_water FROM conversations;
SELECT max(created_at) AS expert_queue_high_water FROM expert_queue;
SELECT max(created_at) AS agentic_tasks_high_water FROM agentic_tasks;
SELECT max(created_at) AS agent_runs_high_water FROM agent_runs;
SELECT max(created_at) AS run_events_high_water FROM run_events;

Use the freeze_high_water_created_at ISO timestamp as --high-water-created-at for dry-run, apply, and smoke so the rehearsal/cutover has one consistent boundary.

Drain​

  1. Wait for in-flight non-runtime handlers to finish or be explicitly cancelled.
  2. Confirm there are no legacy tool/task/model side-effect jobs still running.
  3. Confirm no runner claims are accepted while the backfill applies.
  4. Confirm no org-less ingress is queued for execution; unknown ingress must be in quarantine/remediation only.

Suggested checks:

SELECT status, count(*) FROM agent_runs GROUP BY status ORDER BY status;
SELECT status, count(*) FROM unresolved_channel_events GROUP BY status ORDER BY status;
SELECT status, count(*) FROM runner_registrations GROUP BY status ORDER BY status;

Backfill runtime rows​

The backfill script is dry-run by default. It finds active source rows at or before the high-water mark that do not have runtime coverage:

  • active expert_queue rows -> paused conversation_draft agent_runs with expert_queue_item_id.
  • active agentic_tasks rows -> paused agentic_task agent_runs with agentic_task_id.
  • active conversations without active queue/task coverage -> paused conversation_draft agent_runs.

Inserted runs are deliberately paused with metadata.backfill.containment = 'paused_until_reopen' and a first run.backfilled ledger event. This avoids accidental runner pickup during the freeze.

Dry-run:

cd api
DATABASE_URL="$DATABASE_URL" \
  npx ts-node scripts/backfill-runtime-runs.ts \
  --dry-run \
  --high-water-created-at="$FREEZE_HIGH_WATER_CREATED_AT"

Apply during the approved freeze window only:

cd api
DATABASE_URL="$DATABASE_URL" \
RIG_RUNTIME_BACKFILL_CONFIRM=apply-runtime-runs \
RIG_RUNTIME_POLICY_HASH="$CURRENT_POLICY_HASH" \
RIG_RUNTIME_TOOL_MANIFEST_HASH="$CURRENT_TOOL_MANIFEST_HASH" \
RIG_RUNTIME_PROMPT_BUNDLE_HASH="$CURRENT_PROMPT_BUNDLE_HASH" \
  npx ts-node scripts/backfill-runtime-runs.ts \
  --apply \
  --high-water-created-at="$FREEZE_HIGH_WATER_CREATED_AT"

Then rerun dry-run. The total missing runtime rows must be 0 for the same high-water mark.

Runner bootstrap gate​

Before reopening, all blockers must pass:

  1. At least one active, unexpired runner_manifests row exists for the target environment/org set.
  2. Required runner registrations are ready and have heartbeat fresher than the cutover threshold (default 120 seconds).
  3. No ready runner registration has a stale/missing heartbeat.
  4. Runner manifest hash, policy hash, tool manifest hash, and event schema version match the deployed API expectations.
  5. Runner callback signature verification is active; unsigned runner events are rejected and not persisted.
  6. Runner JWTs are signed with RUNNER_JWT_SECRET, include aud=runner, and are not accepted by user JWT auth paths.
  7. Runner env has only scoped runtime credentials; no broad provider/vendor/R2/platform secrets.

Run the smoke/go-no-go script:

cd api
DATABASE_URL="$DATABASE_URL" \
FRONTEND_BASE_URL="https://staging.h853.work" \
  npx ts-node scripts/smoke-runtime-cutover.ts \
  --required-ready-runners=1 \
  --heartbeat-max-age-seconds=120 \
  --high-water-created-at="$FREEZE_HIGH_WATER_CREATED_AT"

The script exits 0 only for GO. A NO_GO report is a blocker unless the incident lead explicitly documents a non-blocking false positive.

Sandbox supervisor production gate​

Before reopening any hostile-code sandbox capability traffic, also run the sandbox worker/supervisor gate from docs/runbooks/sandbox-worker-supervisor.md.

Required blockers:

  1. ADR-022 has recorded AlexD / Paul / Eusden cosign.
  2. api/scripts/smoke-sandbox-supervisor.ts emits GO on an eligible native/systemd host or a dedicated K8s node with proven primitives.
  3. Deterministic CI units are green, including supervisor Zig tests, capability-shell tests, worker tests, API evidence tests, schema drift checks, seccomp validation, and static bypass audit.
  4. API production evidence validator is enabled and rejects missing/fake linux-bwrap-supervisor evidence.
  5. Direct-spawn bypass test/audit is green.
  6. Hostile-code deny battery is green on the eligible host.

Missing any gate keeps production sandbox capability execution disabled. Rollback must never route to backend=none, sandbox=off, direct spawn, raw bwrap success, degraded success, or unsandboxed worker success.

Reopen​

  1. Re-enable runtime-backed workers and webhook processing.
  2. Allow runner claims only after the runner bootstrap gate is green.
  3. Re-enable frontend/Ops traffic.
  4. Keep default HITL: no channel send unless policy and approval state explicitly allow it.
  5. Keep legacy authority paths disabled; do not reopen them for convenience.
  6. Watch the first 30 minutes:
SELECT status, count(*) FROM agent_runs GROUP BY status ORDER BY status;
SELECT type, count(*) FROM run_events WHERE created_at >= now() - interval '30 minutes' GROUP BY type ORDER BY count(*) DESC;
SELECT status, count(*) FROM approval_requests GROUP BY status ORDER BY status;
SELECT status, count(*) FROM unresolved_channel_events GROUP BY status ORDER BY status;
SELECT status, count(*) FROM runner_registrations GROUP BY status ORDER BY status;

Backfilled paused rows should be advanced only by approved runtime code or an audited one-off forward-fix. Do not bulk-flip them to runner-claimable states without confirming HITL and policy state.

Go/no-go criteria​

GO​

  • Backfill dry-run reports Total missing runtime rows: 0 at the freeze high-water mark.
  • Smoke script status is GO.
  • Ops /ops/agent-runs and /ops/approvals show scoped runtime data.
  • Expert and client approval/input surfaces can resolve staged requests.
  • Unknown Slack/Telegram/Teams/email/WhatsApp/WeChat ingress remains quarantined.
  • New channel sends require explicit runtime policy + approval state.

NO-GO​

Any of these block reopen:

  • Migration failure or schema drift in runtime tables.
  • RLS internal context missing for workers/backfills/smoke or org context not fail-closed.
  • Active source rows missing runtime coverage after apply.
  • No fresh ready runner registration.
  • Stale ready runner registration still eligible for claims.
  • Runner callback signatures not enforced.
  • Legacy authority path executes a side effect.
  • Org-less ingress can execute runtime work.
  • Tool/model/artifact/sandbox path bypasses runtime gateways.

Forward-fix containment​

Prefer forward-fix when runtime rows/events are intact and the issue is scoped:

  • Quarantine bad ingress by inserting/updating unresolved_channel_events; do not execute it.
  • Mark broken runner registrations draining, unhealthy, or disabled and let healthy runners claim only compatible manifests.
  • Append compensating run_events for observable state correction; do not edit ledger rows in place.
  • Patch metadata with an audited script only when the correction is secret-free and org-scoped.
  • Keep affected runs paused, waiting_approval, or waiting_input until policy is clear.

Rollback criteria​

Rollback is allowed only before reopen or when containment cannot protect clients. Rollback means closing new runtime work and restoring the previous deployed build; it does not mean re-enabling legacy authority routes for side effects.

Rollback if:

  • Data corruption affects multiple orgs and cannot be isolated with RLS/runner disablement.
  • Runtime event ledger is missing or duplicating events for new production work.
  • Approval/input state can result in unauthorized sends.
  • Runners accept unsigned events or incompatible manifests.
  • Tool/model/artifact/sandbox gateway bypass is observed in production.

Rollback steps:

  1. Re-freeze ingress and runner claims.
  2. Disable affected runner registrations.
  3. Preserve all runtime rows/events for forensic analysis.
  4. Deploy last known safe build with legacy authority still disabled, or route all affected operations to HITL/manual remediation.
  5. Announce NO-GO/rollback evidence and assign forward-fix owners.

Staging rehearsal checklist​

Run this end-to-end on staging before production:

  1. Deploy migrations and app builds with production-like env (SYNCHRONIZE=false).
  2. Register at least one runner from a signed active manifest.
  3. Create representative active conversations, Expert queue items, agentic tasks, approval requests, user input requests, artifacts, and unresolved ingress events.
  4. Execute freeze and capture high-water marks.
  5. Run backfill dry-run, apply, rerun dry-run.
  6. Run smoke/go-no-go script.
  7. Verify RLS by checking an org-scoped caller cannot see another org's run.
  8. Verify unknown/mismatched channel webhook goes to quarantine and creates no run.
  9. Verify approval resolution from Ops/Expert/client surfaces.
  10. Verify /ops/agent-runs details show events, policy decisions, tool calls, artifacts, approvals, and input requests with redacted payloads.
  11. Reopen staging and watch 30 minutes of run/event/approval/quarantine/runner metrics.
  12. Attach evidence to the release ticket.
Last updated on by Degen