Skip to main content

Runbook: AWS / ECS Production Outage

Status (2026-06-29): Production API + agent run on AWS ECS Fargate behind a public ALB, with RDS PostgreSQL 16 (Multi-AZ) fronted by RDS Proxy and ElastiCache Redis 7 (#1101, staging boot validated #1303). The full stack is Terraform under infra/terraform/modules/{network,data,ecs-cluster,ecs-service,alb,monitoring,iam,secrets} composed by modules/stack. Frontend stays on Vercel. This runbook covers the case where the AWS-hosted prod substrate is degraded — ECS, ALB, RDS, RDS Proxy, ElastiCache, or a CloudWatch alarm firing against them.

This is the AWS-stack counterpart to railway-outage.md (now scoped to the dev/staging Railway substrate). For the broad first-response checklist and per-surface remediation (API down, agent errors, rollback, feature-flag kill-switches), the primary doc is incident-response.md — it already leads with the ECS path. Use this runbook when the failing thing is AWS infrastructure itself (a whole service down, ALB returning 5xx, RDS failing over, the proxy saturated) rather than an application bug.

Full topology and sizing: docs/ops/ecs-prod-deployment-spec.md. Alert stack reference: observability.md § AWS-Side Alerts.

How to Confirm It's AWS (not an app bug)​

  1. Check the AWS Health Dashboard for us-west-2 (prod region — ADR/spec §15 Q6). A regional ECS/RDS/ELB event there means the blast radius is AWS-wide, not ours.
  2. Confirm multiple surfaces are degraded at once (API + agent, or every request 5xx) rather than one endpoint — a single failing route is an app bug; route to incident-response.md.
  3. Check #ops-alerts in Slack — the CloudWatch→SNS→Slack subscriber (humanwork-prod-alerts) fires for ECS task-count drops, ALB 5xx surges, CPU/memory pressure, and OOMKills. A cluster of these is the AWS-substrate signal.
  4. Confirm the env you're paged on is ECS. Prod API/agent are on ECS; dev/staging are Railway. If a Railway env is the failing substrate, use railway-outage.md instead.

Convention: commands below use humanwork-<env> — substitute humanwork-prod (or humanwork-staging). Region is us-west-2. The custom domain is api.h.work; if it's down, hit the ALB DNS directly (aws elbv2 describe-load-balancers --query 'LoadBalancers[].DNSName').

First Response (< 5 minutes)​

  1. Post in #ops-alerts: "AWS/ECS prod incident in progress — investigating ." Severity per incident-response.md § Severity Levels.
  2. Snapshot the live control plane: 
    aws ecs describe-services --cluster humanwork-<env>-cluster \
      --services humanwork-<env>-api humanwork-<env>-agent \
      --query 'services[].{Name:serviceName,Desired:desiredCount,Running:runningCount,Pending:pendingCount,Status:status}'
  3. Hit health endpoints (custom domain, then ALB DNS as fallback): 
    curl -sS https://api.h.work/health   # liveness
    curl -sS https://api.h.work/ready    # readiness — fails if PG/agent/Redis unreachable
  4. Pull up the CloudWatch dashboard and the firing alarms: 
    aws cloudwatch describe-alarms --state-value ALARM \
      --alarm-name-prefix humanwork-<env> \
      --query 'MetricAlarms[].{Name:AlarmName,Reason:StateReason}'
  5. Decide the failing layer from the table below and jump to that section.
SymptomLikely layerSection
Service runningCount < desiredCount; tasks crash-loopingECS tasksECS Task Health
5xx-rate-critical alarm; /health itself 502/503 from ALBALB / targetsALB Target Health
/ready fails with DB errors; RDS event/failover in progressRDSRDS Multi-AZ Failover
/ready flaps; ECONNREFUSED/too many connections; proxy saturatedRDS ProxyRDS Proxy Degradation
Billing counters/locks/Socket.io degrade; leader lostElastiCache RedisElastiCache Redis Degradation
A humanwork-<env>-* alarm fired and you need the playbookanyCloudWatch Alarm Response

ECS Task Health​

Symptoms: runningCount < desiredCount; the humanwork-<env>-api-running-below-desired (or -agent-) alarm is in ALARM; tasks restarting.

  1. Read the service events — they explain why the scheduler can't keep tasks up: 
    aws ecs describe-services --cluster humanwork-<env>-cluster --services humanwork-<env>-api \
      --query 'services[0].{Desired:desiredCount,Running:runningCount,Pending:pendingCount,Events:events[0:5]}'
  2. Inspect the most recent stopped tasks for the stop reason: 
    aws ecs list-tasks --cluster humanwork-<env>-cluster --service-name humanwork-<env>-api --desired-status STOPPED
    aws ecs describe-tasks --cluster humanwork-<env>-cluster --tasks <task-arn> \
      --query 'tasks[].{Stopped:stoppedReason,Containers:containers[].{Name:name,Reason:reason,Exit:exitCode}}'
  3. Tail logs for the crash: 
    aws logs tail /ecs/humanwork-<env>-api   --follow --since 15m
    aws logs tail /ecs/humanwork-<env>-agent --follow --since 15m
  4. Match the cause:
    • OOMKilled (stopCode=OutOfMemory, and the humanwork-<env>-oom-killed EventBridge alert fired in Slack — the string never reaches the log group because SIGKILL skips the flush). Bump memory/memoryReservation in infra/terraform/modules/ecs-service and re-apply, or scale out for immediate relief: aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --desired-count <n>.
    • Startup invariant exit(1) — the API refuses to boot if a prod invariant is violated (DATABASE_URL not RDS-Proxy, JWT_SECRET < 64 chars, SENTRY_DSN/MASTER_ENCRYPTION_KEY/REDIS_URL/TAVUS_API_KEY unset, DEMO_MODE=true). The log line names the offending var. Fix in Secrets Manager / the task-def environment[], then --force-new-deployment.
    • Migration failure on boot — run-migrations.sh runs on API start. Check DATABASE_URL (RDS Proxy endpoint) + DATABASE_SSL=true. Migrations are idempotent and expand-contract-safe; if a single bad migration is wedging boot, see Rollback below.
    • Missing secret / image pull failure — the service event says unable to pull or ResourceNotFoundException on a secret ARN. Verify the secret exists and the task execution role can read it; re-apply Terraform if an ARN drifted.
  5. Force a fresh rollout once config is fixed: 
    aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --force-new-deployment
    aws ecs wait services-stable --cluster humanwork-<env>-cluster --services humanwork-<env>-api
  6. Rollback if a recent deploy is the cause — re-point at the prior task-definition revision (ECS keeps history indefinitely): 
    aws ecs list-task-definitions --family-prefix humanwork-<env>-api --sort DESC --max-items 10
    aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api \
      --task-definition humanwork-<env>-api:<prev-revision>
    This is the same procedure as incident-response.md § Rollback. Production deploys/rollbacks are gated on Eusden.

ALB Target Health​

Symptoms: humanwork-<env>-alb-5xx-rate-critical in ALARM. That alarm is defined on HTTPCode_Target_5XX_Count — it fires when the tasks themselves answer with 5xx (they're up and reachable behind the ALB, but erroring), which in an infra incident most often means a failing downstream dependency (PG/Redis/agent). A different failure mode — clients getting 502/503 because there's no healthy target behind the ALB — surfaces as HTTPCode_ELB_5XX_Count (load-balancer-generated) and is not counted by this alarm; you catch that via target health (step 1) and the ELB-5xx metric (step 3).

  1. Check target-group health. The ALB API target group health-checks /health/db (a DB-only readiness probe), not /ready — deliberately, so that an agent/Redis blip can't unhealth every API task at once (the app is fail-open on agent/Redis; /ready 503s on those and is reserved as the deploy-smoke gate). So a failing ALB health check means the database is wedged or the task is down, nothing else. 
    # Find the target group ARN — scope to THIS env's TG by exact name
    # (staging + prod share the account, so a contains-`api` filter returns both).
    aws elbv2 describe-target-groups --names humanwork-<env>-api-tg \
      --query 'TargetGroups[0].TargetGroupArn' --output text
    # Health of each registered target
    aws elbv2 describe-target-health --target-group-arn <tg-arn> \
      --query 'TargetHealthDescriptions[].{Target:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason}'
  2. Interpret target state:
    • unhealthy / Target.FailedHealthChecks — /health/db is failing, so the task can't reach Postgres (wedged PG pool, RDS/proxy down) or the task itself is crashing. Jump to RDS Multi-AZ Failover / RDS Proxy Degradation, or ECS Task Health if the container is restarting. Note: because the probe is DB-only, Redis or agent degradation does NOT show up as unhealthy targets — the tasks stay healthy and keep serving (fail-open). You find those via the deep curl /ready (step 3) and the symptoms in their own sections, not here.
    • draining during a deploy — expected; ECS drains old tasks (deregistration delay ≥ 120s so BullMQ drains). Wait for the rollout.
    • No registered targets — the ECS service has zero running tasks; this is really an ECS Task Health problem.
  3. Separate the target 5xx that paged you (this alarm) from ALB-level 5xx, which it does not cover: 
    # ELB-generated 5xx — LB had no healthy target / timed out. NOT counted by alb-5xx-rate-critical.
    aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB \
      --metric-name HTTPCode_ELB_5XX_Count --statistics Sum --period 300 \
      --start-time $(date -u -d '30 min ago' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
      --dimensions Name=LoadBalancer,Value=<alb-arn-suffix>
    • HTTPCode_Target_5XX_Count rising with healthy targets (what fired this alarm) = the tasks are up and answering, but returning 5xx. Fork on the cause: a failing dependency (PG / RDS Proxy / Redis / agent) — stay in this runbook, jump to the relevant section — vs an application bug with all dependencies healthy — route to incident-response.md. The /ready body (which dependency it reports unhealthy) and the API logs tell you which.
    • HTTPCode_ELB_5XX_Count rising with unhealthy / no targets = the ALB has nothing to route to, so it's emitting the 5xx itself. That's a target/task problem, not an app error — go to ECS Task Health (and step 1 above).
  4. WebSocket note: Socket.io rides the same ALB listener. If real-time queue notifications stop but REST works, confirm the ALB idle timeout is still ≥ 120s and WebSocket upgrade is allowed — but this is non-fatal: the frontend's 8s polling fallback compensates (not a P0). See incident-response.md § Expert Queue Not Delivering Notifications.
  5. TLS/cert: the ACM cert is operator-provided and DNS-validated via Cloudflare. A 525/526 from Cloudflare means the ALB cert is invalid/expired — check aws acm describe-certificate and the Cloudflare CNAME → ALB mapping (Terraform creates no DNS records; DNS lives in Cloudflare).

RDS Multi-AZ Failover​

Symptoms: /ready fails with DB connection errors; RDS console shows a failover event; brief write unavailability.

RDS PostgreSQL 16 runs Multi-AZ — on primary failure (or a forced reboot), AWS promotes the standby and re-points the RDS endpoint. The app reaches PG through RDS Proxy, which holds the client side steady across the swap, so a healthy failover is typically a ~30s blip, not an outage.

  1. Check instance + recent events: 
    aws rds describe-db-instances --db-instance-identifier humanwork-<env>-pg \
      --query 'DBInstances[0].{Status:DBInstanceStatus,AZ:AvailabilityZone,MultiAZ:MultiAZ,Endpoint:Endpoint.Address}'
    aws rds describe-events --source-identifier humanwork-<env>-pg --source-type db-instance --duration 120
  2. If a failover is already in progress (status failing-over / rebooting): do not intervene — let it complete. RDS Proxy pins the connections; the API's /ready will flap then recover as the standby accepts writes. Watch ALB target health recover within ~30–60s (same shape as the quarterly drill, staging-smoke-runbook.md Item 7).
  3. If the primary is wedged and no failover triggered, force one (Multi-AZ only): 
    aws rds reboot-db-instance --db-instance-identifier humanwork-<env>-pg --force-failover
    This promotes the standby. Expect a short write-unavailability window; reads from cached connections may also error briefly.
  4. After recovery, verify the database is whole: 
    # From inside an API task:
    aws ecs execute-command --cluster humanwork-<env>-cluster --task <task-arn> \
      --container api --interactive --command "/bin/sh"
    psql "$DATABASE_URL" -c "\dx"   # expect: vector, pg_trgm present
    psql "$DATABASE_URL" -c "SELECT 1"
  5. Storage-exhaustion-induced failure (status storage-full) is a different problem — see db-full.md. Free space / scale storage there; a failover won't fix a full disk.
  6. Confirm the connection-pool/proxy side recovered cleanly — proceed to RDS Proxy Degradation if /ready keeps flapping after the instance reports available.

RDS Proxy Degradation​

Symptoms: /ready flaps even though the RDS instance is available; logs show ECONNREFUSED, connection timeouts, or remaining connection slots are reserved; latency spikes on every DB call.

RDS Proxy is load-bearing, not optional: the prod startup invariant rejects a non-proxy DATABASE_URL, and the proxy multiplexes the many per-task PG pools (DATABASE_POOL_MAX, default 20) so Fargate scale-out doesn't blow max_connections. When the proxy saturates, every service that talks to PG degrades at once.

  1. Check proxy + its target health: 
    aws rds describe-db-proxies --db-proxy-name humanwork-<env>-pg-proxy \
      --query 'DBProxies[0].{Status:Status,Endpoint:Endpoint}'
    aws rds describe-db-proxy-targets --db-proxy-name humanwork-<env>-pg-proxy \
      --query 'Targets[].{Type:Type,State:TargetHealth.State,Reason:TargetHealth.Reason}'
  2. Look at the saturation metrics in CloudWatch (AWS/RDS namespace, DBProxyName dimension):
    • DatabaseConnections / MaxDatabaseConnectionsAllowed — proxy-to-DB side near the ceiling = the backend is the bottleneck (a slow query or a leak holding connections).
    • ClientConnections — many idle/borrowed client connections = the app is leaking or not releasing pool connections.
    • DatabaseConnectionsBorrowLatency rising = clients waiting on the proxy to hand out a backend connection.
  3. Connection leak / pool exhaustion is the common cause:
    • Find the hogs on the DB: 
      psql "$DATABASE_URL" -c \
        "SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY 2 DESC;"
      psql "$DATABASE_URL" -c \
        "SELECT pid, state, now()-query_start AS age, left(query,80) FROM pg_stat_activity \
         WHERE state <> 'idle' ORDER BY age DESC LIMIT 10;"
    • Immediate relief: roll the API to drop and re-establish pools — aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --force-new-deployment. The proxy's IdleClientTimeout also reaps stale client connections over time.
    • If a single runaway query is pinning connections, terminate it: SELECT pg_terminate_backend(<pid>);.
  4. Don't bypass the proxy. Temporarily pointing DATABASE_URL at the raw RDS endpoint will trip the production startup invariant (process.exit(1)) and the task won't even boot. Fix the proxy or the leak, not the wiring.
  5. If the proxy itself is unavailable (not just saturated): check its Secrets Manager auth secret and subnet/SG reachability (rds-proxy-sg allows 5432 from api-sg); re-apply the data Terraform module if config drifted.

ElastiCache Redis Degradation​

Symptoms: leader lost (/ops/leader-status shows no leader > 60s); billing counters / circuit-breaker / Socket.io cross-pod emits degrade; rate-limit shadow-mode Sentry alerts (#839).

Redis backs BullMQ, the Socket.io adapter, the cross-pod rate limiter, circuit breaker, and leader election. The deep /ready probe does 503 when Redis is down, but the ALB health-checks /health/db (DB-only), so a Redis outage does not unhealth API targets — the tasks keep serving and Redis trouble degrades features rather than taking the platform down. A lost leader, though, means scheduled work stalls.

  1. Check the cluster: 
    aws elasticache describe-replication-groups --replication-group-id humanwork-<env>-redis \
      --query 'ReplicationGroups[0].{Status:Status,Failover:AutomaticFailover,Nodes:NodeGroups[0].NodeGroupMembers[].{Id:CacheClusterId,Role:CurrentRole}}'
    Multi-AZ with automatic failover is enabled; a primary loss promotes the replica automatically (brief connection reset).
  2. Check leader health — no leader > 60s means cron sweeps, Haystack reconciliation, and billing rollups aren't running: 
    curl -H "Authorization: Bearer ${SUPERADMIN_TOKEN}" https://api.h.work/ops/leader-status
    See observability.md § Leader-Status. Leader election re-acquires automatically once Redis is reachable; a rolling --force-new-deployment of the API forces fresh lease attempts if it's wedged.
  3. Connectivity: confirm REDIS_URL is rediss:// (TLS) with the AUTH token, and redis-sg allows 6379 from api-sg. A cache.r6g.large near 100% CPU or with high evictions points at memory pressure — check CPUUtilization / Evictions / DatabaseMemoryUsagePercentage in CloudWatch (AWS/ElastiCache).
  4. Blast radius is bounded by design (see railway-outage.md for the same reasoning): idempotency caches fall back to in-memory 24h TTL; rate limiting runs in shadow mode (logs, doesn't block); the 8s frontend polling fallback covers dropped Socket.io emits. Redis-down is degraded-but-serving, usually P1/P2 not P0 — confirm PG is healthy and keep the platform answering.

CloudWatch Alarm Response​

The alert stack (infra/terraform/modules/monitoring, wired per env) fans every alarm into one SNS topic (humanwork-<env>-alerts) → Slack subscriber Lambda. Full reference: observability.md § AWS-Side Alerts. When a humanwork-<env>-* alarm pages:

AlarmMeansGo to
api-running-below-desired / agent-running-below-desiredService below capacity 10m — bad deploy or crash loopECS Task Health
api-cpu-high / agent-cpu-high (CPU > 85%, 10m)Sustained load or hot loopCheck autoscaling caught up (describe-services desired vs running); scale max if pinned; profile if it's a hot path
api-memory-high / agent-memory-high (Mem > 90%, 5m)OOM precursorBump task memory (Terraform) before it OOMKills; see ECS Task Health
alb-5xx-rate-critical (target 5xx > 1%, 5m)Clients seeing errorsALB Target Health
oom-killed (EventBridge stopCode=OutOfMemory)A container was SIGKILLed by the kernelBump memory or fix the leak; ECS Task Health
specialist-mismatch-spike (> 5/min, 5m)Runtime manifest/claim mis-routing — isolation-sensitivePage Platform Lead before muting. See observability.md; an Expert may be testing an unbound runtime, so triage is not automatic
agent-backend-failed-spike (> 10/min, 5m)Agent → platform API roundtrip failingCheck API health + the agent→API network path; incident-response.md § Agent Errors Spiking

General alarm hygiene:

  • An alarm in INSUFFICIENT_DATA for the ECS task-count alarms is treat_missing_data = breaching by design — no metric usually means no running tasks. Treat it as real.
  • Don't disable an alarm to silence a page. If it's a known false positive, note it in #ops-alerts and fix the threshold in modules/monitoring via PR — silently muting hides the next real event.
  • Resource/log-filter alarms send an OK action too, so the channel self-clears when the condition resolves; confirm the OK arrived before declaring recovery.

Stop-the-bleeding levers (while you diagnose)​

These don't require a deploy and contain blast radius fast:

  • Feature-flag kill-switches (/ops/feature-flags, ~15s propagation) — flip auto_reply_enabled OFF to force all drafts to Expert review, or tool_calling_enabled/agentic_mode_enabled/kb_retrieval_enabled OFF if one capability is the culprit. Full table in incident-response.md § Feature Flags / Kill-Switches.
  • Disable auto-send platform-wide — set CONFIDENCE_THRESHOLD_OVERRIDE=200 in Secrets Manager / the API task-def and --force-new-deployment. Routes everything to the Expert queue regardless of confidence (the platform is always-HITL by default anyway). Remove when healthy.
  • Scale out — aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --desired-count <n> buys headroom during a CPU/memory squeeze faster than re-sizing the task.

The platform fails open on agent errors (unreachable agent → safe escalation message + Expert queue), so a degraded agent never surfaces a 500 to clients — prioritize the API/PG/ALB path.

Recovery & Post-Incident​

  1. Confirm all services are back to desiredCount, /ready is 200, and every humanwork-<env>-* alarm has cleared to OK: 
    aws ecs describe-services --cluster humanwork-<env>-cluster --services humanwork-<env>-api humanwork-<env>-agent \
      --query 'services[].{Name:serviceName,Desired:desiredCount,Running:runningCount}'
    aws cloudwatch describe-alarms --alarm-name-prefix humanwork-<env> --state-value ALARM \
      --query 'MetricAlarms[].AlarmName'
    curl -sS https://api.h.work/ready
  2. Run a full conversation flow manually (or the staging smoke helper for staging). Verify Expert queue items created during the incident appear on next queue load (Socket.io emits during an outage may have dropped; the queue is durable in PG).
  3. Reverse any temporary levers — remove CONFIDENCE_THRESHOLD_OVERRIDE, flip feature flags back ON (with a reason), scale desired-count back to baseline.
  4. Reconstruct the timeline from /ops/audit (SuperAdmin) — see incident-response.md § Forensics. Audit writes are eventual/fire-and-forget (ADR-019), so treat the log as a strong record, not a guaranteed-complete ledger.
  5. Write the post-mortem (timeline, root cause, fix, prevention), file GitHub issues for systemic gaps, and update this runbook if the incident revealed a missing procedure.
Last updated on by Degen