Runbook: Incident Response

Who to Contact

See OWNERS.md for full contact list.

Role	Responsibility
Platform Lead	Overall incident ownership, escalation to Eusden
On-call Engineer	First responder, initial triage
Eusden	Production deploy authority, major architectural decisions

Production deploys are restricted to Eusden. For incidents requiring a production rollback or hotfix deploy, Eusden must be looped in.

---

Severity Levels

Severity	Definition	Response Time
P0	All clients unable to send/receive messages; auth down	Immediate (wake up)
P1	Major feature broken for all orgs (queue down, billing broken)	30 minutes
P2	Feature broken for some orgs; degraded performance	2 hours
P3	Minor issue; workaround available	Next business day

Signal → severity mappings (#3990)

Explicit classifications for alerts that previously had no place in the matrix (the #3444 30k-row DLQ pileup sat unclassified for a week — never again):

Signal	Severity	Rationale
DLQ pending (queued+retrying) over DLQ_ALERT_THRESHOLD for 30+ minutes	P1	Expert-approved replies may not be reaching customers — treat as an outbound delivery incident. Runbook: dlq-handling.md.
DLQ dead-growth alert fired (DLQ_DEAD_GROWTH_ALERT, #3986)	P2	Deliveries are terminally failing for some channel/org; retention + replay tooling bound the blast radius. Runbook: dlq-handling.md.
Per-channel canary failure (#3989, once live)	P1	A whole outbound channel is broken — the #3506 class.

---

First Response Checklist

Where prod runs (2026-06-02): Production API + agent run on AWS ECS Fargate (#1101, Terraform modules under infra/terraform/modules/{ecs-cluster,ecs-service,alb,monitoring}). Frontend stays on Vercel. Legacy Railway services remain reachable for dev/staging during the migration window. Each section below leads with the ECS / AWS path; the Railway block beneath is the legacy / staging fallback — copy from the correct block for the env you're paged on. When the failing thing is AWS infrastructure itself (a whole ECS service down, ALB 5xx surge, RDS failover, RDS Proxy saturation, a CloudWatch alarm), use the deep-dive companion aws-outage.md.

1. Check the live control plane.

   ECS / AWS (primary prod path): commands use humanwork-<env> — substitute humanwork-prod or humanwork-staging (the Terraform name_prefix). The ECS cluster is humanwork-<env>-cluster, services/task-def families are humanwork-<env>-api / humanwork-<env>-agent, and log groups are /ecs/humanwork-<env>-api / /ecs/humanwork-<env>-agent.

   aws ecs describe-services --cluster humanwork-<env>-cluster \
     --services humanwork-<env>-api humanwork-<env>-agent \
     --query 'services[].{Name:serviceName,Desired:desiredCount,Running:runningCount,Status:status}'
   aws logs tail /ecs/humanwork-<env>-api   --follow --since 15m
   aws logs tail /ecs/humanwork-<env>-agent --follow --since 15m

   Also: CloudWatch dashboards (humanwork-<env>-overview) and the ALB target group health.

   Railway (legacy / staging fallback):

   railway logs --service humanwork-api   | tail -50
   railway logs --service humanwork-agent | tail -50

2. Check health endpoints:

   curl https://api.h.work/health
   curl https://api.h.work/ready
   # If the custom domain is down, hit the ALB DNS for the ECS service directly
   # (look it up in the ALB console or `aws elbv2 describe-load-balancers --query 'LoadBalancers[].DNSName'`):
   curl https://<alb-dns-name>/health

3. Check Sentry for a spike in errors (#845). Filter by environment:production and the last 30 minutes. Both humanwork-api and humanwork-agent projects carry org_id + osa_id tags — group by org_id to see if the impact is one tenant or platform-wide.

4. Check leader-status (#865/#1007) — multi-pod deployments rely on a single pod holding the Redis leader lease for scheduled jobs:

   curl -H "Authorization: Bearer ${SUPERADMIN_TOKEN}" \
     https://api.h.work/ops/leader-status

   No leader for > 60s means cron sweeps + Haystack reconciliation + billing rollups are not running. See docs/runbooks/observability.md § Leader-Status.

5. Check Vercel dashboard for frontend deploy status

6. Check PostgreSQL connectivity: GET /ready will fail if DB is unreachable

7. Check ThrottlerModule shadow-mode alerts in Sentry (#812/#839) — sustained throttler.shadow_block spikes may indicate an abuse storm even though no requests are being blocked yet. See docs/runbooks/observability.md § Throttler Shadow Mode.

8. Post status update in team Slack channel within 15 minutes of incident start

---

API / Agent Service Down

Symptoms: GET /health returns non-200; the live control plane shows tasks crashed or restarting.

ECS / AWS path (primary prod)

1. Check service health + recent task stops:

   aws ecs describe-services --cluster humanwork-<env>-cluster --services humanwork-<env>-api \
     --query 'services[0].{Desired:desiredCount,Running:runningCount,Pending:pendingCount,Events:events[0:5]}'
   aws ecs list-tasks --cluster humanwork-<env>-cluster --service-name humanwork-<env>-api --desired-status STOPPED
   aws ecs describe-tasks --cluster humanwork-<env>-cluster --tasks <task-arn>
2. Tail logs:

   aws logs tail /ecs/humanwork-<env>-api --follow --since 15m
3. Look for: OOM (Out of Memory — also see OOMKilled EventBridge rule in observability.md), migration failure, missing env var/secret, crash on startup.
4. OOM: increase memory / memoryReservation in the task definition (infra/terraform/modules/ecs-service) and re-apply; or temporarily scale out (aws ecs update-service --desired-count).
5. Migration failure: check DATABASE_URL (RDS Proxy endpoint) and DATABASE_SSL=true in the task definition / AWS Secrets Manager; run the migration manually via aws ecs execute-command into a running task.
6. Env var / secret missing: update Secrets Manager or the task-definition environment block; force a new deployment (aws ecs update-service --force-new-deployment).
7. Force a restart: aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --force-new-deployment.
8. Rollback: see ## Rollback Procedure below.

Railway path (legacy / staging fallback)

1. Check logs: railway logs --service humanwork-api --lines 100
2. Look for: OOM, migration failure, missing env var, crash on startup
3. If OOM: scale up memory in Railway dashboard (Settings → Resources)
4. If migration failure: check DATABASE_URL and DATABASE_SSL=true are set; run migration manually
5. If env var missing: check Railway Variables for the service; compare against .env.example
6. Restart service: Railway dashboard → Service → Deployments → Restart
7. If restart fails: rollback to previous deploy

   railway rollback --service humanwork-api --deployment <previous-deploy-id>
   Railway keeps the last 5 deployments.

---

Vercel Deploy Failed / Frontend Down

Symptoms: https://app.h.work (production) or https://humanwork-frontend.vercel.app returns 500 or shows stale build.

Steps:

1. Check Vercel dashboard → Deployments → find failing deploy → view build logs
2. Common causes: TypeScript build error, missing env var, Next.js build timeout
3. To rollback: Vercel dashboard → Deployments → select previous successful deploy → "Promote to Production"
4. To redeploy from CLI:

   cd frontend && vercel --prod

---

Database Connection Errors

Symptoms: GET /ready fails; API logs show ECONNREFUSED or connection timeout.

ECS / RDS path (primary prod)

1. Check RDS instance + RDS Proxy status:

   aws rds describe-db-instances --db-instance-identifier humanwork-<env>-pg \
     --query 'DBInstances[0].{Status:DBInstanceStatus,Endpoint:Endpoint.Address}'
   aws rds describe-db-proxies --db-proxy-name humanwork-<env>-pg-proxy
2. Verify the task definition's DATABASE_URL (Secrets Manager) points at the RDS Proxy endpoint, not the raw instance, and that DATABASE_SSL=true is set.
3. Connection pool exhausted? Check Database/Postgres/DatabaseConnections CloudWatch metric and the RDS Proxy ClientConnections / DatabaseConnections metrics. The proxy is what fans out client connections — if it's saturated, scale or look for a leak.
4. If the DB itself is degraded: AWS Console → RDS → Events; consider failover (aws rds reboot-db-instance --force-failover for Multi-AZ). For the full Multi-AZ failover and RDS Proxy degradation playbooks, see aws-outage.md § RDS Multi-AZ Failover and § RDS Proxy Degradation.
5. After a restore, verify extensions:

   psql "$DATABASE_URL" -c "\dx"
   # Expected: vector, pg_trgm extensions present.

Railway path (dev only)

Dev only — staging/prod are on AWS ECS Fargate (RDS Proxy). See docs/ops/deploy.md.

1. Check Railway Postgres service status in dashboard
2. Verify DATABASE_URL env var is set on the API service
3. Verify DATABASE_SSL=true is set (Railway Postgres requires SSL)
4. Check if connection pool is exhausted (Railway Postgres has a connection limit per plan)
   • Dev runs direct to Postgres (no pooler — decision #1233, 2026-06-05). If pool exhaustion shows up in real usage on dev, restart the API service to release connections. Don't reflexively reintroduce PgBouncer — RDS Proxy already handles pooling on staging/prod, and dev's load profile typically doesn't justify the extra moving part. If saturation becomes chronic on dev, the alternatives are (a) bump DATABASE_POOL_MAX (default 20, see railway-env-vars.md) and the Postgres max_connections ceiling, (b) re-wire the unused Railway PgBouncer plugin, or (c) move dev to AWS.
5. If Postgres service is crashed: Railway dashboard → humanwork-postgres → Restart
6. Verify extensions are installed after restore:

   railway run --service humanwork-postgres \
     psql "$DATABASE_URL" -c "\dx"
   Expected: vector, pg_trgm extensions present.

---

Agent Errors Spiking

Symptoms: Expert queue filling up faster than usual; POST /chat or POST /v1/chat returning errors; API logs show agent timeouts.

ECS / AWS path (primary prod)

1. Tail agent logs:

   aws logs tail /ecs/humanwork-<env>-agent --follow --since 15m
2. Look for: OpenAI API rate limits (429), model errors (500), OOM.
3. If OpenAI rate limit: check usage at platform.openai.com; reduce throughput or upgrade tier.
4. If agent is OOM: bump task memory in the agent's ECS task definition (Terraform) and re-apply.
5. To disable auto-send temporarily (route everything to Expert queue):
   • Update the API task definition's env / Secrets Manager value: set CONFIDENCE_THRESHOLD_OVERRIDE=200. (For one-shot incident control, AWS SSM Parameter Store also works if the task is wired to read from there.)
   • Force a new deployment: aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-api --force-new-deployment.
   • This must be set higher than any per-org autoRespondThreshold (default is 101 → "never auto-send"; an override of 200 ensures no org can ever auto-send during the incident).
   • All responses will route to Expert queue regardless of agent confidence.
   • Remove the override when agent is healthy.
6. If agent tasks are repeatedly crashing, force a new deployment after fixing config:

   aws ecs update-service --cluster humanwork-<env>-cluster --service humanwork-<env>-agent --force-new-deployment

Railway path (legacy / staging fallback)

1. Check agent logs: railway logs --service humanwork-agent | tail -100
2. Look for: OpenAI API rate limits (429), model errors (500), OOM
3. If OpenAI rate limit: check usage at platform.openai.com; reduce throughput or upgrade tier
4. If agent is OOM: scale memory in Railway dashboard
5. To disable auto-send temporarily (route everything to Expert queue):
   • In Railway Variables for humanwork-api, set CONFIDENCE_THRESHOLD_OVERRIDE=200
   • This must be set higher than any per-org autoRespondThreshold (default is 101 → "never auto-send"; an override of 200 ensures no org can ever auto-send during the incident).
   • All responses will route to Expert queue regardless of agent confidence.
   • Remove the override when agent is healthy.
6. If agent service is repeatedly crashing, restart it:

   railway up --service humanwork-agent

---

WhatsApp (Twilio) Not Receiving Messages

Symptoms: WhatsApp messages not arriving in API; clients reporting messages to Specialist not getting responses.

Steps:

1. Check API logs for Twilio webhook activity (aws logs tail /ecs/humanwork-<env>-api --follow --since 30m | grep whatsapp on ECS; railway logs --service humanwork-api | grep whatsapp | tail -50 on Railway-legacy).
2. Check Twilio Console → Monitor → Errors for webhook delivery failures.
3. Verify webhook URL in Twilio Console matches https://api.h.work/channels/whatsapp/inbound (prod) or the dev URL for the env.
4. Verify TWILIO_AUTH_TOKEN (main account) — on ECS this lives in AWS Secrets Manager; on Railway-legacy in Railway Variables. For subaccount-owned senders, also check TWILIO_SUBACCOUNT_TOKEN.
5. If webhook is being rejected with 403: TWILIO_AUTH_TOKEN or TWILIO_WEBHOOK_BASE_URL is wrong — update the corresponding secret store and force a new deployment.
6. Twilio retries failed webhooks for up to 4 hours — messages are not lost unless the outage exceeds that window.

---

Expert Queue Not Delivering Notifications

Symptoms: New queue items not appearing in real-time; experts must refresh to see new items.

Steps:

1. Check if WebSocket connection is established: browser DevTools → Network → WS tab
2. Check API logs for Socket.io errors
3. Check NEXT_PUBLIC_WS_URL env var is set correctly in Vercel (should be wss://api.h.work for production, or the ALB DNS / staging hostname if the custom domain is down).
4. Verify CORS: ALLOWED_ORIGINS env var on API must include the Vercel frontend domain (e.g. https://humanwork-frontend.vercel.app or https://app.h.work)
5. If Socket.io failing, queue still functions — experts can poll manually. Not a P0.

---

Feature Flags / Kill-Switches

When to use: a Specialist capability is actively causing harm (bad KB results, runaway tool calls, an agentic loop, or auto-replies going out that shouldn't) and you need to stop the bleeding faster than a deploy or rollback. Flipping a flag takes effect across the fleet within ~15 s (the in-process resolve cache TTL) — no redeploy required.

Full system reference: docs/features/feature-flags.md.

Surface: SuperAdmin → /ops/feature-flags. Pick the narrowest scope that contains the blast radius — a single (Org × Specialist) override first, then the whole org, and only global when the problem is platform-wide.

Symptom	Flag to flip OFF	Effect
KB returning wrong/leaked context, or Haystack misbehaving	kb_retrieval_enabled	Specialist drafts with no KB context (empty retrieval), keeps responding.
Runaway / erroring tool calls (Shopify/Amazon, etc.)	tool_calling_enabled	Tool calls blocked at the gateway (status=blocked), drafting continues without tools.
Agentic task loop or bad open-ended drafting	agentic_mode_enabled	Agentic task creation returns 403; pipeline falls back to non-agentic drafting.
AI drafts being auto-delivered to clients that shouldn't be	auto_reply_enabled	All drafts queue for Expert review (always-HITL); nothing auto-sends.

Procedure:

1. Open /ops/feature-flags (SuperAdmin). If you only need to contain one client, use Add override in Per-org overrides (Org ID) or Per-org-specialist overrides (OSA id) instead of touching the global row.
2. Toggle the flag OFF. A reason is mandatory — write the incident id / short cause; it is recorded in the per-flag audit trail and the platform audit log.
3. Confirm propagation: allow up to ~15 s, then verify the symptom stops. A narrower override beats a broader one, so an org = ON row will keep a client live even while global = OFF (and vice-versa) — check you flipped the scope that actually wins.
4. Recovery: once the underlying fix is deployed, flip the flag back ON (or delete the override) with a reason. Use the History (clock) icon to see who changed what and confirm the live state.

Flipping a flag is a mitigation, not a fix — still file the post-incident issue and roll out the real change. Remember auto_reply_enabled defaults OFF and most others default ON; re-read the live row rather than assuming.

---

Rollback Procedure

ECS / AWS path (primary prod)

ECS rollback = re-point the service at the previous task-definition revision. ECS keeps task-definition history indefinitely; the service-events list shows the last several deploys.

# List recent task-definition revisions for the API
aws ecs list-task-definitions --family-prefix humanwork-<env>-api --sort DESC --max-items 10

# Roll the service back to the prior revision
aws ecs update-service \
  --cluster humanwork-<env>-cluster \
  --service humanwork-<env>-api \
  --task-definition humanwork-<env>-api:<prev-revision-number>

# Watch the rollout
aws ecs describe-services --cluster humanwork-<env>-cluster --services humanwork-<env>-api \
  --query 'services[0].deployments'

Frontend rollback is unchanged: Vercel dashboard → Deployments → previous successful deploy → "Promote to Production".

Railway path (legacy / staging fallback)

# Railway: list recent deployments
railway deployments --service humanwork-api

# Rollback to a specific deployment
railway rollback --service humanwork-api --deployment <deploy-id>

# Vercel: promote a previous deployment
# Go to: vercel.com → Project → Deployments → click previous deploy → "Promote to Production"

---

Forensics — reconstructing what happened

Use the platform audit log at /ops/audit (SuperAdmin only) to reconstruct the action timeline during and after an incident: who did what, to which resource, in which client, and when. Filter by Client + time window, then narrow by Action / Entity, and export to CSV to attach to the post-mortem. Impersonation sessions also have a dedicated trail at /ops/impersonation, and feature-flag flips are recorded in the per-flag history (clock icon on /ops/feature-flags) as well as the platform audit log.

Caveat: audit writes are eventual / fire-and-forget (ADR-019) — they happen after the business commit and failures are swallowed, so treat the log as a strong operational record, not a guaranteed-complete ledger. See docs/features/audit-log.md for the event model, scope, and retention notes.

Post-Incident

1. Write a brief post-mortem (timeline, root cause, fix, prevention) — pull the action timeline from /ops/audit (see Forensics)
2. Create GitHub issues for any systemic gaps found
3. Update this runbook if the incident revealed a missing procedure