Skip to main content

Runbook: Railway-Backed Service Outage (legacy / staging-fallback)

Status (2026-06-02): Production has been migrated to AWS ECS Fargate for API + agent (#1101, Terraform under infra/terraform/modules/{ecs-cluster,ecs-service,alb,monitoring}, staging boot validated in #1303). Railway remains the substrate for dev and staging during the migration window and as a legacy fallback. If prod is degraded and the failing surface is ECS/RDS/ALB, the right runbooks are docs/runbooks/incident-response.md (first response, per-surface remediation) and its AWS-substrate companion docs/runbooks/aws-outage.md (ECS task health, ALB target health, RDS Multi-AZ failover, RDS Proxy degradation, CloudWatch alarm response). Use this doc only for Railway-as-the-failing-substrate scenarios.

Use this runbook when Railway itself is experiencing an outage (not a single service crash) AND the affected env still runs on Railway.

How to Confirm It's Railway​

  1. Check https://status.railway.app for active incidents
  2. Verify multiple services are affected simultaneously (not just one crashed service)
  3. Check team Slack — others will likely be reporting the same thing
  4. Confirm the env you're paged on is still Railway-hosted. Prod API/agent are on ECS — see docs/ops/ecs-prod-deployment-spec.md. If prod is impaired, route to incident-response.md.

During a Railway Outage (dev/staging)​

Immediate (< 5 minutes)​

  1. Post in team Slack: "Railway outage in progress — monitoring status.railway.app (impact: dev/staging only)"
  2. Check if Vercel frontend is still serving (it's independent of Railway)
    • If frontend is up: clients can still load the app but API calls against Railway-backed envs will fail
    • Post a status note if you have a client-facing status page
  3. Do not attempt to redeploy services during an outage — Railway deployments will also fail

If Outage Extends > 30 Minutes​

  1. Assess which functionality is affected (on Railway-backed envs):

    • API down → no new conversations, no queue processing, no auth
    • Postgres down → API will also be down (DB dependency)
    • Redis down → per-Specialist billing counters (Redis-backed) degrade; some webhook idempotency caches fall back to in-memory (24h TTL). ThrottlerModule was re-registered in shadow mode (#812 closes #803) and emits Sentry alerts on Redis disconnect (#839); shadow mode logs but does not block requests, so Redis-down does not currently affect rate limiting in production. Minimal user impact.
    • Agent down → new messages auto-escalate to Expert queue (fail-open behavior)
    • WhatsApp (Twilio) → Twilio delivers to API; if API is down, Twilio will retry for up to 4 hours

  2. Communicate to affected clients if SLA breach is imminent

  3. For extended Postgres outage on Railway (> 2 hours), assess data loss risk:

    • Railway Postgres has automatic backups — check Railway dashboard for last backup time
    • No action needed unless DB was corrupted before the outage

Recovery​

When Railway recovers:

  1. Check all service health: curl https://api.h.work/health and /ready (for a Railway-backed staging env, use the ALB DNS for the ECS prod path or the env's Railway public URL — never paper over a Railway outage by pointing prod traffic at a stale Railway URL).
  2. Check Railway logs for each service for startup errors
  3. Verify migrations ran successfully (check API logs for migration output on startup)
  4. Verify pgvector extension is still present: 
    railway run --service humanwork-postgres \
      psql "$DATABASE_URL" -c "SELECT extname FROM pg_extension WHERE extname = 'vector';"
  5. Test a full conversation flow manually
  6. Check Expert queue for any items that were created during the outage but not delivered via WebSocket — they will appear on next queue load

Failover Posture (2026-06-02)​

Production is no longer Railway-only. The de-facto DR posture today:

  • AWS ECS Fargate (live for prod): API + agent run on ECS in their own VPC/subnets. Terraform modules (infra/terraform/modules/stack) implement the full stack — cluster, services, ALB, monitoring, OIDC GitHub Actions deploy roles. Staging boot is validated (#1303). Prod cutover is in flight; the ECS path is reachable today.
  • RDS Postgres (live for prod): Multi-AZ RDS behind RDS Proxy is the prod database (see #1303, #917). Quarterly Multi-AZ failover drill is in docs/runbooks/staging-smoke-runbook.md Item 7.
  • Railway (dev/staging substrate): Where this runbook applies. No hot standby on Railway itself.
  • Manual failover to Render/Fly.io: still hypothetical for the Railway tier; not part of the active DR plan now that ECS is live for prod.

Formal cross-region DR planning for the AWS stack is a future initiative. For Railway specifically, the mitigation remains Railway's own redundancy and fast recovery (typical Railway incidents resolve in < 2 hours), now bounded to dev/staging blast radius.

Last updated on by Degen