Local 2-pod active-active dev setup

Owner: backend infra Last updated: 2026-05-27 (#881 — full validation pass) Compose file: docker-compose.multi-pod.yml Nginx config: scripts/local/nginx-multipod.conf Cross-pod test script: scripts/load/socketio-fanout-test.mjs

When this is useful

The default docker compose up runs a single api container. That's fine for most dev, but it doesn't catch:

• Socket.io Redis adapter cross-pod fan-out bugs (PR #449)
• BullMQ jobId dedup races (P0-9e/9f, #827)
• Leader election (P0-9c) — needs > 1 pod to test at all
• Throttler Redis storage convergence (P0-9k)
• SIGTERM drain during rolling deploy (PR #836 / #808)
• DLQ retry worker idempotency under concurrent pods (#744)

For those, run the 2-pod local stack.

Pre-requisites

The multi-pod compose builds the api container in the Dockerfile's default final stage (runner — compiled dist/), unlike the default override (docker-compose.override.yml) which sets target: builder for hot-reload. That means env vars required at boot must actually be set in .env. Common ones that may be missing in a stale local .env:

# Generate if missing — values shown here are placeholders, run openssl
# rand to get real ones, never commit:
JWT_SECRET=<openssl rand -hex 48>
MASTER_ENCRYPTION_KEY=<openssl rand -hex 32>

RUNNER_JWT_SECRET (api, #866) and HUMANWORK_ORG_SLUG / PLATFORM_API_TOKEN (agent, #870) have dev fallbacks — they're not required in NODE_ENV=development. Set them only if you want to mirror prod behaviour.

If any required secret is absent the api containers will start, fail boot with <NAME> is required for ..., and the LB will return 502. Check docker logs humanwork-api-1 after first boot if so.

Boot

# Bring up postgres + redis + agent + api-1 + api-2 + nginx-lb
docker compose \
  -f docker-compose.yml \
  -f docker-compose.multi-pod.yml \
  up -d

# Watch both pods come up
docker compose logs -f api-1 api-2

http://localhost:3000 is the nginx LB; it round-robins between api-1:3000 and api-2:3000.

Verify it's actually 2 pods

# Each curl should land on a different pod — look for X-Pod-Endpoint
for i in 1 2 3 4; do
  curl -sI http://localhost:3000/health | grep -i x-pod-endpoint
done

Expected output rotates between the two upstreams:

X-Pod-Endpoint: 172.18.0.5:3000
X-Pod-Endpoint: 172.18.0.6:3000
X-Pod-Endpoint: 172.18.0.5:3000
X-Pod-Endpoint: 172.18.0.6:3000

If both lines show the same IP, nginx fell back to a single upstream — check that both api containers are healthy: docker compose ps.

Smoke-test cross-pod behavior

Socket.io fan-out (item #6)

wscat doesn't speak the Socket.io protocol — use the bundled Node script instead. It opens two Socket.io clients (nginx round-robin lands them on different pods), publishes a synthetic agent_message_sent via @socket.io/redis-emitter, and asserts both sockets receive it.

cd scripts/load
npm install              # one-time
npm run socketio-fanout

Expected output:

Socket.io cross-pod fan-out test → http://localhost:3000
  ✓ JWT acquired (superadmin → 'experts' room)
  ✓ A: connected sid=...
  ✓ B: connected sid=...
  → publishing via @socket.io/redis-emitter
  ✓ A received agent_message_sent
  ✓ B received agent_message_sent

PASS — Redis adapter cross-pod fan-out verified

Exit code 0 = pass, 1 = fan-out broken. The script auto-acquires a superadmin JWT via /auth/demo-login and reads the Redis password from env (REDIS_PASSWORD, default HwDev2026).

If it fails: check REDIS_URL is set on both pods, and that PUBSUB CHANNELS '*' on Redis lists socket.io-request#/notifications# plus one socket.io-response#/notifications#<uid># per pod (two if 2 pods are subscribed).

Pod-kill resilience (item #11)

# Start a slow load
while true; do curl -sf http://localhost:3000/health > /dev/null; sleep 0.1; done &
LOAD_PID=$!

# Kill one pod mid-stream
docker stop humanwork-api-1

# Verify the load script keeps getting 200s (nginx fails over to api-2)
# After ~10s, expect zero errors in your terminal

# Restart the killed pod
docker start humanwork-api-1
# Watch logs — leader election should rebalance cleanly:
docker compose logs api-1 | grep -i 'leader\|election'

# Stop load
kill $LOAD_PID

Pass criteria: no failed curl, no double-cron-fire after restart, no socket message duplication.

Leader election (P0-9c)

The leader lock value is a hostname-derived id (e.g. b49b4a048d45-1-4psjd0), not the literal api-1/api-2. Match it against docker exec humanwork-api-1 hostname / humanwork-api-2 hostname.

The Redis password in .env (REDIS_PASSWORD) is not the compose default — locally it's typically HwDev2026. Extract it once from a running pod's REDIS_URL:

PW=$(docker exec humanwork-api-1 sh -c 'echo $REDIS_URL' \
     | sed -E 's|.*://:([^@]+)@.*|\1|')

# Current leader + TTL
docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning GET hwork:leader:lock'
docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning TTL hwork:leader:lock'

Failover recipe (verified working in #881):

# Identify leader → kill it
LEADER_HOST=$(docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning GET hwork:leader:lock' \
  | cut -d- -f1)
if [ "$(docker exec humanwork-api-1 hostname)" = "$LEADER_HOST" ]; then
  docker stop humanwork-api-1
else
  docker stop humanwork-api-2
fi

# Wait > LEADER_LOCK_TTL (30s) for the lock to expire, then verify the
# other pod has taken over (new value, fresh TTL ~28s):
sleep 35
docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning GET hwork:leader:lock'
docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning TTL hwork:leader:lock'

# Recovery check (no split-brain): restart the stopped pod, verify it
# does NOT steal the lock — count of BullMQ scheduler keys should stay
# the same (~80, not ~160):
docker start humanwork-api-1   # or api-2
sleep 20
docker exec -e PW="$PW" humanwork-redis sh -c \
  'redis-cli -a "$PW" --no-auth-warning KEYS "bull:scheduler:repeat:*" | wc -l'

Pass criteria: leader value changes after 35 s wait; lock TTL fresh (~28 s) after failover; scheduler key count stable after restart.

NFR smoke against 2-pod local

# After standing up the multi-pod stack:
TOKEN_A=$(curl -sS -X POST http://localhost:3000/auth/demo-login \
  -H 'Content-Type: application/json' \
  -d '{"role":"org_admin","orgSlug":"acmefinancial"}' | jq -r '.token')
TOKEN_B=$(curl -sS -X POST http://localhost:3000/auth/demo-login \
  -H 'Content-Type: application/json' \
  -d '{"role":"org_admin","orgSlug":"techcorp"}' | jq -r '.token')

STAGING_BASE_URL=http://localhost:3000 \
STAGING_FRONTEND_URL=http://localhost:3001 \
STAGING_TOKEN_ORG_A="$TOKEN_A" \
STAGING_TOKEN_ORG_B="$TOKEN_B" \
bash scripts/test-nfr-smoke.sh

Expected on the local stack (post-#879): 6 PASS / 0 FAIL / 8 SKIP, exit 0.

• Items 1, 4, 8, 9 PASS unconditionally
• Item 3 (cross-org IDOR) SKIPs because local /auth/demo-login returns the same token regardless of orgSlug — the probe needs two distinct real users; on staging it runs normally
• Items 6 and 11 stay SKIP in the smoke script — but the cross-pod recipes above cover them manually. Items 5, 7, 10, 12 require infra (Sentry, RDS, channel_failures seed, real queue item) and only run against staging.

What's now validated locally

MVP P0 NFR	Validated by	Script
P0-3 Socket.io Redis adapter	Cross-pod fan-out (§Socket.io fan-out)	scripts/load/socketio-fanout-test.mjs
P0-9c Leader election	Failover + no split-brain (§Leader election)	shell recipe
P0-9e/9f BullMQ jobId dedup	Scheduler key count stable across pod restart	shell recipe
P0-11 Pod kill mid-deploy	50 reqs through nginx, 0 errors (§Pod-kill resilience)	shell recipe
/health, /ready reachability, webhook dedup, prod-invariant grep	scripts/test-nfr-smoke.sh	smoke script

Things still needing staging:

• P0-1 Sentry alert path → real DSN
• P0-2 DLQ retry → real channel_failures rows
• P0-9k Throttler convergence → multi-pod soak with high QPS
• RDS Multi-AZ failover (P0-?) → AWS infra

Tear down

docker compose \
  -f docker-compose.yml \
  -f docker-compose.multi-pod.yml \
  down

Then go back to the default override (single api pod):

docker compose up -d

Gotchas

• Don't mix the multi-pod compose with docker-compose.override.yml — the override defines a single api service with hot-reload volumes; combining them would reactivate that service alongside api-1 / api-2 and you'd have 3 pods competing for the same Redis keys. The override is automatically picked up unless you pass -f docker-compose.yml -f docker-compose.multi-pod.yml explicitly (this excludes docker-compose.override.yml).
• Hot reload isn't wired on api-1 / api-2 — each container builds the default final stage (runner) of api/Dockerfile, not the builder stage that the dev docker-compose.override.yml uses. To pick up code changes: rebuild with docker compose -f docker-compose.yml -f docker-compose.multi-pod.yml up -d --build.
• Single Postgres / single Redis — this is local convenience, not a Multi-AZ test. NFR smoke item #7 (RDS failover) stays SKIP locally.
• Single agent — agent is stateless, multi-pod adds no signal there. Keep agent as one container.
• Single Frontend — same logic, no Next.js multi-pod story to test locally.
• Healthcheck IPv6/IPv4 trap (lesson from #872/#874) — in python:slim and alpine base images, localhost resolves to ::1 only. If the service under check binds IPv4-only (Next.js dev server, nginx listen 80;), wget http://localhost:... will fail Connection refused even when the service is up. Always use 127.0.0.1 in compose healthcheck.test for containers whose service binds IPv4-only. Node's HTTP server binds dual-stack :::3000 so the api healthcheck is unaffected — netstat -tln inside the container tells you which side a given service is on.
• Redis password is from .env, not compose default — the compose ${REDIS_PASSWORD:-Hw2026!Redis#SecureDefault} fallback is rarely used; local stacks typically have REDIS_PASSWORD=HwDev2026 (or whatever the team-shared .env says). Extract it from a running pod: docker exec humanwork-api-1 sh -c 'echo $REDIS_URL' | sed -E 's|.*://:([^@]+)@.*|\1|'. The password contains ! and # in the default — they break URL parsing in redis client libs, prefer passing as a separate password field.

Reference

• ADR-018 §AA1–AA10 (multi-pod safety contract)
• NFR smoke runbook: docs/runbooks/staging-smoke-runbook.md
• #863 — issue tracking this setup
• #870 / #873 / #876 — boot bug fixes surfaced by this stack
• #879 — NFR smoke false-positive cleanup
• #881 — this runbook's validation pass