Skip to main content

ADR-038: Railway deprecation and ECS-canonical production

Status: Accepted (2026-07-13) Authors: Degen (formal write-up); decision confirmed by Eusden/Emanuele per ADR-003 addenda and the #701 Kaito-launch P0 cutover. Supersedes: ADR-003 β€” Railway + Vercel Split Deployment (partially, per below)

Context​

ADR-003 (2026-04-22) established Railway for all backend services (api, agent, Postgres, Redis) plus Vercel for the frontend, as the platform's canonical deployment target. That was correct at the time β€” Railway's managed Postgres/Redis and simple deploy flow suited an early-stage single-region product.

As the platform moved toward its first paying client (Kaito) and multi-region ambitions, the Railway posture became insufficient:

  • Railway is single-region (US-West); Asia-Pacific latency for the Kaito launch was a known trade-off, tracked as BLK-037 and folded into the ECS cutover.
  • #701 (P0, "Migrate humanwork production from Railway to ECS β€” Kaito-launch P0") framed the decision plainly: "Railway is fine for vibe-coded dev iteration; it is not the production environment we hand a real client a WhatsApp number on."
  • ADR-028 (2026-06-03) had already retired the per-Org Railway agent-runtime provisioner (railway_per_org) in favor of ECS-only per-Org isolation β€” a narrower, code-level decommission that anticipated this broader infrastructure move.

#701 closed on 2026-06-15. Its acceptance criteria β€” ECS infrastructure provisioned, API+Agent+Frontend on ECS (not Railway) for production, secrets in AWS Secrets Manager, RDS migrated, Twilio/Cloudflare/Slack webhooks repointed β€” were all met and smoke-tested (WhatsApp round-trip via Kaito's number, end to end on ECS).

This ADR is the formal supersession that ADR-003's addendum said would land "once prod also cuts over."

Decision​

ADR-003 is superseded for staging and production. The current and going-forward canonical posture, per docs/ops/deploy.md:

EnvironmentRuntimeFrontendDatabaseConnection pooler
dev (api.h852.work)RailwayVercel (preview)Railway PostgresNone β€” direct connection
staging (api.h853.work)AWS ECS FargateAWS ECS Fargate (app.h853.work)RDS PostgreSQLAWS RDS Proxy
production (api.h.work)AWS ECS FargateAWS ECS Fargate (app.h.work)RDS PostgreSQLAWS RDS Proxy
  • Railway remains canonical for dev only β€” this is a deliberate choice (see #1233, 2026-06-05), not an oversight or a lagging migration. Dev's low load profile doesn't justify AWS's operational overhead; the Railway PgBouncer plugin remains available unwired if that ever changes.
  • Staging and production are AWS ECS Fargate, with RDS PostgreSQL fronted by RDS Proxy, Terraform-managed (infra/terraform/modules/{ecs-cluster,ecs-service,alb,monitoring,data}), and deployed via the dev β†’ staging β†’ main CI-enforced promotion flow (#1559).
  • Frontend also deploys to ECS for staging/production β€” .github/workflows/frontend-deploy.yml and Terraform provision an frontend_service ECS task definition with ALB host rules for staging (app.h853.work) and production (app.h.work). Dev retains Vercel for fast preview deployments (APP_ENV=development gates programmatic Vercel domain attach/detach in api/src/vercel/vercel-domain.service.ts). See ADR-015 for the wildcard ACM certificate strategy that made the AWS frontend target possible.

What ADR-003 got right (preserved)​

  • The original Vercel-for-frontend decision (now superseded for staging/production by ECS; dev retains Vercel).
  • The reasoning for rejecting Render and Fly.io at the time (limited WebSocket support, no managed Postgres/Redis) β€” still valid; those options were never revisited because ECS solved the actual problem (multi-region readiness, AWS-native secrets/observability, no per-instance connection-limit surprises).
  • The env-driven DB pool sizing addendum (DATABASE_POOL_MAX etc., #917) β€” the mechanism is unchanged; only the pooler in front of it differs by environment (RDS Proxy on AWS, none on Railway dev).

What changed​

  • Railway is no longer a production concern. Production incidents, on-call runbooks, and rollback procedures should default to the ECS/AWS path first (see docs/runbooks/incident-response.md, which already leads with ECS/AWS and treats Railway as "dev only").
  • The per-Org Railway agent-runtime provisioner is deleted (ADR-028) β€” there is no Railway per-Org isolation escape hatch anymore. shared_railway remains as the dev-only shared runtime path.
  • Deploy credentials for staging/prod are AWS IAM + Terraform state, not Railway tokens. Railway tokens remain only for dev.

Consequences​

Positive:

  • Multi-region readiness (AWS AP-Southeast is now a real option for the Asia-Pacific latency concern BLK-037 raised) β€” not yet exercised, but no longer architecturally blocked by Railway's single-region limitation.
  • One production incident-response playbook (ECS-first), not two parallel paths to reason about for staging+prod.
  • AWS Secrets Manager for production secrets, closing the "plain Railway env vars" gap #701 flagged as unacceptable for a paying client's credentials.

Negative / trade-offs:

  • Two infrastructure stacks to maintain (Railway for dev, AWS for staging+prod) rather than one. Accepted deliberately β€” dev's low load profile and fast-iteration needs don't match production's isolation/compliance needs, and unifying them would either over-provision dev or under-provision prod.
  • Contributors need to know which environment they're debugging before reaching for railway logs vs. aws ecs/aws logs. Mitigated by the ECS-first framing in incident-response.md and the explicit "Railway path (dev only)" labeling introduced there.

Rollback​

There is no rollback to Railway for production. If ECS provisioning breaks, the mitigation is ECS-native (task rollback to a prior revision, scale-out, or β€” per ADR-028 β€” falling back to shared_railway for the agent-runtime path specifically, which is a dev/shared-infra fallback, not a production Railway reinstatement).

References​

Last updated on by Degen