ADR-018: Multi-Pod Safety Contract (Active-Active Hard Rules AA1-AA10)

Date: 2026-05-28 Status: Accepted Issue: #937 (this ADR) Implements / codifies: P0-9 Multi-Instance Active-Active Safety Bundle (13 sub-tasks, complete 2026-05-28) Authors: Tom Tang, Eusden, claude Related: ADR-013 (dedup keys), ADR-019 (data consistency), ADR-021 (throttler rate-limit — renumbered from ADR-020 in #1229)

Context

The h.work platform deploys as multi-instance active-active in staging and prod (PR #863 established the local 2-pod stack for verification). This is a hard architectural commitment — N pods serve traffic simultaneously behind an LB with no sticky sessions; any pod can handle any request from any user. Pod restarts (rolling deploy, scaling event, ECS task drain) MUST NOT lose business state or produce duplicate side effects.

Without explicit rules, the natural Node.js single-process patterns (new Map() for cache, setInterval for cron, setImmediate for async work) break in surprising ways once N > 1. The damage modes are subtle (extra cron fires, message duplicates, lost in-flight work) and don't show up in single-pod tests.

This ADR codifies the 10 rules every active-active codebase change MUST follow ("hard rules" AA1-AA10), enumerates the banned patterns + approved alternatives, and links each rule to the sub-task that implements it.

Decision: 10 Hard Rules

#	Rule	Implemented by	Pre-existing or this session
AA1	All in-memory state externalized to Redis or Postgres	9a (Redis CB), 9d (Tavus dedup)	Mostly pre-existing; 9d cleaned up in #934
AA2	All cron / scheduled tasks run exactly once globally	9b (Scheduler → BullMQ), 9c (@Cron → leader election)	Pre-existing
AA3	All write operations idempotent	9d / 9e / 9f / 9g	9g + 9f delivered in #929 / #936
AA4	All async work persisted (BullMQ, not setImmediate)	9e / 9f	9f delivered in #936
AA5	All writes have race protection (PG unique + atomic UPDATE...WHERE...RETURNING)	9g / 9j	Delivered in #929 / #932
AA6	LB has no sticky session	(Infra) ALB target group stickiness.enabled=false	External (Emanuele ECS PR)
AA7	Health checks fast + independent of in-flight state	P0-2 (Redis adapter), P0-7 health probes	Mostly pre-existing
AA8	Graceful shutdown ≥ 2× max in-flight LLM timeout	SIGTERM handler in main.ts	Pre-existing; ECS stopTimeout ≥ 120s external
AA9	DB connection pool across pods does not exceed PG max_connections	P0-1 RDS Proxy (pool × pod_count ≤ Proxy_max × 0.8)	External (Emanuele ECS PR)
AA10	Secrets / config from a single source of truth	P0-6 AWS Secrets Manager	External (Emanuele ECS PR)

Code-level rules (AA1-AA5, AA7-AA8) are 100% complete in dev branch as of 2026-05-28. Infra rules (AA6/AA9/AA10) ship with Emanuele's ECS PR.

Banned patterns → approved replacements

❌ Don't	✅ Use	Why ban
private xxxMap = new Map() for state-across-requests	Redis SET/HSET/SET NX	Per-pod Map is incoherent; pod A doesn't know about pod B's writes
setInterval(...) cron	BullMQ Queue.add(name, data, { repeat: { cron: ... }, jobId: stable })	Fires N times in N-pod deployment
setImmediate(() => doWork()) for business work	BullMQ persistent queue (see channels-outbound.processor.ts)	Lost on pod restart between schedule + execution
Sticky-session assumptions (same user → same pod)	None — code must be pod-independent	ALB doesn't pin; any code assuming it is broken
findOne → check → save for state transitions	UPDATE ... WHERE ... RETURNING * (atomic claim)	TOCTOU: two pods both pass check, both write
INSERT then catch generic error	INSERT ... ON CONFLICT (key) DO NOTHING RETURNING * with explicit unique index	Catches the wrong exception class; misses concurrent inserts
In-process counter (let n = 0) for rate limit / metric	Redis INCR + EXPIRE, or ThrottlerStorageRedisService	Each pod has its own counter → effective limit × N
In-process distributed lock (Promise mutex)	PG pg_advisory_xact_lock(hashtext(key))	Lock is process-local; other pods don't see it
In-process circuit breaker	Redis-backed CB (see circuit-breaker.util.ts:24)	Other pods don't know the breaker is open

Stateful infrastructure layer (single source of truth)

Each piece of state lives in EXACTLY ONE of:

• Postgres — durable business data (conversations, messages, users, audit log)
• Redis — fast shared coordination (dedup, rate limits, leader election, Socket.io fan-out, circuit breaker state, throttler counters)
• R2 (Cloudflare object storage) — large blobs, artifact storage, specialist manifests
• Haystack pgvector tables — vector embeddings and chunk metadata
• BullMQ (backed by Redis) — durable async work queue
• Per-Org Agent sidecar (EFS workspace) — per-org agent runtime state

Nothing business-critical lives in pod memory. Pod memory holds only ephemeral request-scoped state (the current HTTP handler stack frame, DB connection pool, parsed JWT, etc.).

Options considered

1. Sticky sessions + per-pod state — Rejected: blocks linear scale-out (you can't add a pod without bouncing sessions); also breaks under any rebalance event (cordon, drain, autoscale). Half the value of active-active comes from "any request to any pod"; sticky undoes that.
2. Active-passive — Rejected: half resources wasted; in-flight work lost on every switchover; switchover takes seconds-minutes (vs zero for active-active rolling deploy).
3. Leader-only writes — Rejected: write throughput bottleneck. Election storms during high churn.
4. Per-feature opt-in (some features active-active, others sticky) — Rejected: matrix of "which features work multi-pod" becomes a tribal-knowledge minefield. All or nothing is simpler to reason about.

Consequences

Positive:

• Rolling deploy with zero downtime (CodeDeploy minimumHealthyPercent=100)
• Linear scale: add N pods → N× capacity (no leader bottleneck)
• Pod death = client transparent (LB removes failed target; in-flight work persists in BullMQ + reloads from PG)
• Verified by scripts/load/socketio-fanout-test.mjs (Redis adapter cross-pod fan-out) + docs/runbooks/local-multi-pod.md cross-pod tests

Negative:

• More boilerplate for "obvious" patterns (a counter is a Redis call, not let n = 0)
• All new code needs to be reviewed against these 10 rules; CI cannot fully enforce them
• Local dev complexity: 2-pod stack reveals bugs single-pod doesn't (the entire purpose of docker-compose.multi-pod.yml)

Validation

• Local: cross-pod tests in docs/runbooks/local-multi-pod.md (round-robin, pod-kill, leader election, Socket.io fan-out) — all green as of 2026-05-28
• Staging: scripts/test-nfr-smoke.sh automated probes (6 PASS / 0 FAIL / 8 SKIP on local; staging-only items defer)
• Production: Emanuele ECS PR brings AA6/AA9/AA10 online

Decision authority + enforcement

PRs that change any of the following MUST cite this ADR + the relevant AA rule in the description:

• New setInterval / setImmediate / in-process Map for cross-request state
• Removal of @InjectQueue in favor of direct invocation
• Removal of any WHERE status = :expected atomic-claim predicate
• Changes to the Redis client setup or BullMQ defaults

This is a workflow rule, not a CI gate — reviewers must remember to ask "does this conflict with ADR-018?". Adding a lint check is a follow-up consideration.

Reference

• MVP P0 NFR FINAL §"Active-Active Hard Constraints (10 rules)" — original AA1-AA10 table
• CLAUDE.md §"Deployment Model" + §"Shared-state implementation patterns" — operational guidance
• docs/runbooks/local-multi-pod.md — verification procedures
• ADR-013 (dedup keys), ADR-019 (data consistency contract)
• All P0-9 PRs: #870, #909, #911, #929, #932, #934, #936 + pre-existing 9a/9b/9c/9e/9h/9i/9k/9l/9m. Drift-fix follow-ups (2026-06-01 sweep): #1288, #1289, #1290, #1292, #1294, #1296.