Graceful Shutdown Runbook

Owner: backend infra Last updated: 2026-05-27 (#808) Related: ADR-018 (multi-pod safety contract), MVP P0 NFR FINAL EN §Graceful Shutdown Contract

When this matters

• ECS rolling deploys (every PR merge to main → staging → prod)
• Auto-scaling scale-in events (off-peak hours)
• Manual aws ecs stop-task invocations
• Ctrl+C during local dev (same code path so dev behaviour matches prod)

Without an explicit drain routine, NodeJS receives SIGTERM and exits without firing NestJS shutdown hooks. In-flight BullMQ jobs are abandoned, their HTTP-side caller never gets a response, and the task-termination window (default 30s for ECS) is wasted on a process that already left.

Shutdown sequence (by design)

ECS sends SIGTERM
    │
    ▼
process.on('SIGTERM') handler in api/src/main.ts
    │
    ▼
app.close()  ← NestJS Lifecycle
    │
    ├── 1. stop accepting new HTTP connections (http.Server.close)
    │
    ├── 2. drain in-flight HTTP handlers (default 30s Nest grace)
    │
    ├── 3. onModuleDestroy on every module (parallel where independent):
    │      • channels-inbound BullMQ worker — await current 'email-route' /
    │        'whatsapp-route' job, then close ioredis connection
    │      • telegram-inbound BullMQ worker — same for 'route' jobs
    │      • scheduler BullMQ worker — current cron tick completes
    │      • background-tasks BullMQ worker — current job completes
    │      • correction-refinement BullMQ worker — current correction completes
    │      • Socket.io Redis adapter — closes pub/sub connections
    │      • Throttler Redis storage — closes connection
    │      • Sentry SDK — flushes pending events (5s timeout)
    │
    └── 4. TypeOrmCoreModule — closes PG connection pool last
        │
        ▼
process.exit(0)   ← within the ECS stopTimeout window

ECS sends SIGKILL after stopTimeout (default 30s — must be ≥120s in our task definition, see below).

Required ECS configuration

{
  "stopTimeout": 120,
  "containerDefinitions": [...]
}

Why 120s:

• Max LLM call timeout = 60s
• BullMQ job processing a held conversation can include 1× LLM call
• Worker close awaits current job → up to 60s
• • 30s HTTP grace + 30s safety margin = 120s

A value below 60s risks SIGKILL hitting mid-LLM-call and leaving Postgres messages row half-written (we have transactional protection at the DB layer, so no data corruption — but the HTTP caller times out and retries, doubling load).

Where to set: Terraform module ecs/task-definitions/humanwork-api.tf (owned by @ethumanity ECS PR).

What we explicitly do NOT do

• ❌ Forceful job cancellation. BullMQ has Worker.close(force: true) — never use it. In-flight jobs run to completion or fail and retry; never kill mid-execution.
• ❌ Custom timeout-then-kill. If app.close() hangs longer than ECS stopTimeout, ECS SIGKILLs us automatically. Adding our own kill on top creates double-kill races.
• ❌ Sentry flush after process.exit. The Sentry handler is wired via app.getHttpAdapter().getHttpServer().on('close') so it fires as part of step 1 (HTTP server close), not after exit.

Verification

Local smoke test

# Start the API
cd api && npm run start:dev

# In another terminal:
kill -TERM $(pgrep -f "node.*main.js")

# Expected log order:
# [Shutdown] SIGTERM received, draining...
# [BullMQModule] Stopping queue ...
# [TypeOrmModule] Connection closed
# [Shutdown] drain complete, exiting

Total elapsed: typically <2s on dev (no real jobs in flight). Prod with in-flight LLM jobs: 30–90s depending on what's running.

Integration test

api/test/graceful-shutdown.spec.ts (#808) spawns a NestJS process, enqueues a slow BullMQ job, sends SIGTERM, asserts:

• exit code = 0
• elapsed time < 30s
• BullMQ job marked completed (not abandoned)
• HTTP server returned 503 to a request sent during drain

Production deploy check

After any ECS rolling deploy, in CloudWatch query Sentry-task-stop events:

ECS service event: "deployment completed"
  - Tasks stopped: N
  - Tasks stopped via SIGKILL: 0 ← MUST be 0
  - Tasks stopped within stopTimeout: N ← MUST equal "Tasks stopped"

A single SIGKILL means a task didn't drain within 120s → investigate which queue was holding it.

Code references

File	Purpose
api/src/main.ts (around line 211)	process.on('SIGTERM' | 'SIGINT') handler
api/src/main.ts:37	app.enableShutdownHooks() — wires NestJS lifecycle
api/src/channels/channels-inbound.processor.ts	WorkerHost — auto-closes via app.close()
api/src/channels/telegram/telegram-inbound.processor.ts	same
api/src/scheduler/scheduler.processor.ts	same
api/src/common/background-tasks.processor.ts	same
docs/decisions/018-multi-pod-safety-contract.md AA-8	hard requirement