Skip to main content

ECS Production Deployment Spec

Status: Draft · Owner: Platform · Date: 2026-05-29 Scope: Production only (dev + staging remain on Railway/Vercel). IaC: Terraform. Launch type: Fargate. Supersedes/extends: ADR-003 (Railway deployment) for the prod environment; reuses the deferred assets in agent/deploy/ecs.deferred/ (tracking issue humanity-org/humanwork#238).

Implemented. The Terraform for this spec lives in infra/terraform/: reusable modules/ composed by modules/stack, instantiated by two thin env roots — envs/staging (h852.work) and envs/prod (h.work) — in one AWS account, S3-native state locking, shared GitHub OIDC provider in bootstrap. DNS stays in Cloudflare (Terraform creates no DNS records; you CNAME to the ALB output), and the ACM cert is operator-provided. Deploy automation: .github/workflows/{terraform-plan,terraform-apply,api-deploy,agent-deploy,reusable-ecs-deploy}.yml (plan both envs on PR; apply staging from staging / prod from main; per-env app deploys). Setup + cutover runbook: infra/terraform/README.md. Validated with terraform validate, tflint (AWS ruleset), and actionlint.

What prod actually runs today (verified against the repo, not assumed): ADR-003 — one Railway project humanwork with managed Postgres 16+pgvector and Redis 7, plus humanwork-api and humanwork-agent services; frontend on Vercel, not Railway. The agent runs as one shared service (AGENT_RUNTIME_PROVISIONER=shared_railway) — there are no per-org agent runners in production. Region: Railway US-West. Deploys are manual (railway up --service <name>); gitflow promotion is --ff-only dev→staging→main. This re-scopes the migration below.

1. Goal & non-goals​

Goal. Stand up the h.work production environment on AWS ECS Fargate: the long-running services (api, shared agent; frontend — see note), the managed data stores they depend on, and a CI/CD path that builds, pushes, and deploys them.

Re-scope based on current prod: the dynamic per-org agent runners are NOT part of the cutover — they don't run in prod today (shared_railway) and the ecs-per-org.provisioner.ts is an unimplemented stub (throws NotImplementedError, "ECS per-org provisioner is M2 work; see #362c"). They are M2/future (§4 kept as forward-looking design). A like-for-like cutover needs only: api + shared agent + data stores + CI/CD.

Frontend is on Vercel today, not Railway. Moving it to ECS is optional and independent of the API cutover. Decide explicitly (§15 Q7): keep it on Vercel (zero work, recommended for v1) or containerize it onto ECS. The frontend service spec below applies only if you choose to move it.

Non-goals.

  • Migrating dev/staging off Railway/Vercel (explicitly out of scope — those stay).
  • Re-architecting the app. This is a hosting move; the app already obeys the active-active constraints in CLAUDE.md (no in-process state, BullMQ for cron/background, Redis/PG for shared state).
  • A blue/green or canary rollout system on day one (ECS rolling deploy with circuit breaker is enough to start).

Hard constraints carried from CLAUDE.md:

  • Multi-instance active-active — every service runs ≥2 tasks; no sticky sessions; ALB stickiness off.
  • stopTimeout ≥ 120s on the API task (2× the 60s max LLM call) so BullMQ workers drain.
  • Production startup invariants in api/src/main.ts will process.exit(1) unless: DEMO_MODE!='true', JWT_SECRET ≥ 64 chars, SENTRY_DSN set, MASTER_ENCRYPTION_KEY set, DATABASE_URL points at RDS Proxy. Provision accordingly.

2. Target topology​

                  Cloudflare DNS (CNAME api.h.work → ALB; external)
                              │
                      ┌───────┴────────┐
                      │ ACM cert (provided) │
                      └───────┬────────┘
                              │
                   ┌──────────▼───────────┐         ┌──────────────────────┐
   internet  ───►  │  ALB (public)        │         │ ALB (internal)        │
                   │  :443                │         │  :443  agent only     │
                   └───┬───────────┬──────┘         └──────────┬───────────┘
                       │ /api,/ws  │ / (default)               │
              ┌────────▼───┐  ┌────▼─────────┐        ┌─────────▼──────────┐
              │ api service│  │frontend svc  │        │ agent service      │
              │ Fargate ≥2 │  │ Fargate ≥2   │        │ Fargate ≥2         │
              │ :3000      │  │ :3001        │        │ :8000              │
              └─┬───┬───┬──┘  └──────────────┘        └────────────────────┘
                │   │   └──────────────► (calls agent service via internal ALB / Cloud Map)
                │   │
                │   └──► ElastiCache Redis (BullMQ, Socket.io adapter, rate limit, CB)
                │
                └──► RDS Proxy ──► RDS Postgres 16 + pgvector (Multi-AZ)

   Per-org agent runners (on demand, NOT behind ALB):
              ┌────────────────────────────────────────────┐
              │ ECS RunTask: humanwork-agent-<org>-<role>   │
              │ Fargate, EFS workspace mount /app/workspace │
              │ provisioned by api/src/agent-runtime/        │
              │   provisioners/ecs-per-org.provisioner.ts    │
              └────────────────────────────────────────────┘

   Shared external (not in VPC): Cloudflare R2, OpenAI/OpenRouter, Resend,
   Twilio, Slack, Stripe/Lago. KB/RAG is **in-stack**: Haystack 2.x + Hayhooks
   on the `humanwork-rag` ECS service (Cloud Map `humanwork-rag.humanwork.internal:1416`),
   per ADR-024 + #1527.

Why an internal ALB (or Cloud Map) for the agent: the shared agent service must never be internet-reachable — only the api service calls it (AGENT_SERVICE_URL). Put it behind an internal ALB or register it in Cloud Map (agent.humanwork.internal) and let the API resolve it. Cloud Map is cheaper and sufficient since it's purely service-to-service; ALB is only needed if you want per-target health-based routing. Recommendation: Cloud Map for the shared agent, ALB only for api + frontend.

3. Services (long-running) — task sizing​

All Fargate, awsvpc, X86_64, in private subnets, egress via NAT.

ServicePortvCPU / MemDesired / Min / MaxHealth (ALB / container)stopTimeoutNotes
api30001 vCPU / 2 GB2 / 2 / 6GET /ready (ALB), GET /health (container)120sRuns migrations on boot; hosts Socket.io on same port. Scale on CPU + ALB req count.
agent (shared)80001 vCPU / 2 GB2 / 2 / 4GET /health120sHermes shells out per request; ≥ HERMES_TIMEOUT_SEC so in-flight model calls finish before SIGKILL. Cloud Map, no public ALB.
frontend30010.5 vCPU / 1 GB2 / 2 / 4GET / (200)30sNext.js standalone (node server.js). Mostly static + SSR.

ALB routing (public ALB):

  • Listener :443 (ACM cert for h.work, *.h.work).
  • Rule: host api.h.work or path /api/*, /socket.io/*, /ws/* → api target group (sticky off, but Socket.io needs WebSocket upgrade — ALB supports it natively; ensure idle timeout ≥ 60s, set to 120s).
  • Default rule → frontend target group.
  • Target group health checks: api → /ready (matcher 200), frontend → / (matcher 200,304).

WebSocket note: Socket.io over ALB works without stickiness because the app already uses the Redis Socket.io adapter (per CLAUDE.md). Confirm @socket.io/redis-adapter is wired to ElastiCache before cutover — otherwise cross-pod emits drop and the 8s polling fallback masks it.

Autoscaling: target-tracking on ECSServiceAverageCPUUtilization (60%) + ALBRequestCountPerTarget for api/frontend. Per-org runners are not autoscaled — they're RunTask-on-demand.

4. Per-org agent runners (dynamic Fargate) — M2 / NOT in cutover​

Status: deferred. Prod runs the shared agent (shared_railway), so this whole section is forward-looking. api/src/agent-runtime/provisioners/ecs-per-org.provisioner.ts exists but every method throws NotImplementedError ("M2 work; see #362c"). The reference implementation railway-per-org.provisioner.ts was decommissioned per ADR-028 / #1584 (PR merged) — when picking up the M2 work, consult the pre-#1584 git history for its config surface and lifecycle shape rather than expecting the file in-tree. Do not block the cutover on this.

When M2 starts, the assets are already templated in agent/deploy/ecs.deferred/ — promote to agent/deploy/ecs/ and wire it.

  • Launch: ECS RunTask (not a Service), one task per (org_id, agent_role), family humanwork-agent-${ORG_ID}-${ROLE} (template exists: task-definition.json).
  • Provisioner: api/src/agent-runtime/provisioners/ecs-per-org.provisioner.ts (already in repo). Set AGENT_RUNTIME_PROVISIONER=ecs_per_org in prod and supply the cluster/subnets/security-group/role ARNs it needs (see §7, §8).
  • Workspace: EFS, transit-encrypted, IAM-authorized, mounted at /app/workspace. Access point per ${OrgId}/${AgentRole} per efs-access-point.json (CloudFormation snippet — port to Terraform aws_efs_access_point, or keep creating access points dynamically at provision time).
  • Runner token: injected as a Secrets Manager secrets[] entry (HUMANWORK_RUNNER_TOKEN ← ${HUMANWORK_RUNNER_TOKEN_SECRET_ARN}). One secret per runner registration.
  • Lifecycle: runner self-terminates on idle (TERMINAL_LIFETIME_SECONDS, currently 300s) — confirm the provisioner also has a reaper/StopTask path + a BullMQ sweep for orphaned tasks (don't rely on a per-pod setInterval).
  • Sizing per the template: 1 vCPU / 2 GB. Revisit once real Hermes workloads are measured; this is the most likely cost driver.

Config surface to build (mirror the pre-#1584 Railway provisioner shape). The ECS provisioner currently reads nothing (stub). The Railway one (railway-per-org.provisioner.ts) was removed in #1584 / ADR-028, but historically it read, via the factory selected by AGENT_RUNTIME_PROVISIONER:

  • Required: AGENT_RUNTIME_RAILWAY_PROJECT_ID (fallback RAILWAY_PROJECT_ID), AGENT_RUNTIME_RAILWAY_ENVIRONMENT_ID (fallback RAILWAY_ENVIRONMENT_ID), RAILWAY_API_TOKEN, AGENT_SERVICE_SECRET.
  • Optional: RAILWAY_AGENT_IMAGE (default ghcr.io/humanity-org/humanwork-agent:dev), GHCR_PULL_USERNAME/GHCR_PULL_TOKEN, RAILWAY_DEPLOY_TIMEOUT_MS, RAILWAY_DEPLOY_POLL_MS.
  • Always injects into each runner: HUMANWORK_ORG_ID/_SLUG, HUMANWORK_AGENT_NAME/_ROLE, PLATFORM_API_URL, PLATFORM_API_TOKEN, AGENT_SERVICE_SECRET, PORT=8000, the HERMES_* set, workspace paths, R2_KEY_PREFIX=humanwork/${orgId}. It explicitly deletes DATABASE_URL — runners must never touch PG directly.

The ECS equivalent needs instead: cluster ARN, private subnets, runner SG, exec/task role ARNs, EFS filesystem id, AWS region, task-def family. Add those as AGENT_RUNTIME_ECS_* env on the api task def + .env.example when M2 lands.

Reaper: there is no idle reaper today on any provisioner — the pre-#1584 Railway teardownForOrg only fired on explicit org deactivation (serviceDelete). For ECS, runners self-expire via TERMINAL_LIFETIME_SECONDS=300, but a BullMQ sweep (StopTask on orphans + EFS access-point GC) must be built — do not add a per-pod setInterval (CLAUDE.md active-active rule).

5. Managed data stores​

StoreAWS resourceSizing (start)Notes
Postgres 16 + pgvectorRDS PostgreSQL 16, Multi-AZdb.r6g.large, 100 GB gp3, autoscale to 500 GBEnable vector extension. Required: front with RDS Proxy (prod invariant checks DATABASE_URL targets the proxy).
RedisElastiCache Redis 7, cluster-mode disabled, 1 primary + 1 replica, Multi-AZ + automatic failovercache.r6g.largeUsed for BullMQ, Socket.io adapter, rate limiter, circuit breaker, leader election. AUTH token + in-transit TLS → REDIS_URL=rediss://....
EFSEFS (per-org agent workspaces)Elastic throughput, lifecycle to IA at 30dMount targets in each AZ; SG allows 2049 from agent-runner SG only.
Object storageCloudflare R2 (existing, external)n/aKeep R2 — no S3 migration. Buckets: R2_BUCKET_NAME, R2_CLIENT_BUCKET, KB_R2_BUCKET, RUNTIME_ARTIFACTS_R2_BUCKET.
KB / RAGHaystack 2.x + Hayhooks (humanwork-rag ECS service, per #1527 / ADR-024)small Fargate task; pgvector on the existing humanwork RDS (ADR-025); embed/chat/rerank via OpenRouter (ADR-026)Replaces RAGflow per ADR-024/025/026. Cloud Map at humanwork-rag.humanwork.internal:1416. Budget target ≈ $440/env-month. Deploy via .github/workflows/rag-deploy.yml (#1541). See docs/runbooks/haystack-deploy.md.

RDS Proxy is load-bearing, not optional: the prod startup invariant in main.ts rejects a non-proxy DATABASE_URL. It also matters because Fargate tasks scale and each opens a PG pool (DATABASE_POOL_MAX, default 20 since #2540) — the proxy multiplexes so you don't exhaust max_connections.

6. Secrets & config​

Pattern (matches the existing task-def secrets[] block): sensitive values come from AWS Secrets Manager via task-definition secrets[] (injected as env at start, never baked into the image); non-sensitive config via environment[].

Secrets Manager entries (one secret, JSON keys, referenced by valueFrom per key): JWT_SECRET, RUNNER_JWT_SECRET, RUN_CONTRACT_SIGNING_SECRET, MASTER_ENCRYPTION_KEY, DATABASE_URL (RDS Proxy + creds — or compose from a separate RDS secret), REDIS_URL/REDIS_PASSWORD, OPENAI_API_KEY, OPENROUTER_API_KEY, RESEND_API_KEY, TWILIO_AUTH_TOKEN, TWILIO_SUBACCOUNT_TOKEN, SLACK_SIGNING_SECRET, SLACK_BOT_TOKEN, TELEGRAM_BOT_TOKEN, CLOUDFLARE_EMAIL_WEBHOOK_SECRET, STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, LAGO_API_KEY/LAGO_WEBHOOK_SECRET, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, HAYSTACK_API_KEY (per #1588 — sent as X-API-Key to humanwork-rag), PLATFORM_API_TOKEN, SANDBOX_WORKER_TOKEN, SENTRY_DSN*, plus per-runner HUMANWORK_RUNNER_TOKEN.

Plain environment[] (config): APP_ENV=production, HWORK_DOMAIN=h.work, NEXT_PUBLIC_HWORK_DOMAIN=h.work, AGENT_SERVICE_URL=http://agent.humanwork.internal:8000, HAYSTACK_BASE_URL=http://humanwork-rag.humanwork.internal:1416 (per ADR-024 + #1527; downstream sends X-API-Key per #1588), HAYSTACK_TIMEOUT_MS, HAYSTACK_CIRCUIT_*, HAYSTACK_TOP_K, HAYSTACK_MAX_UPLOAD_MB, PLATFORM_API_URL, DATABASE_SSL=true, DATABASE_POOL_MAX, AGENT_CHAT_TIMEOUT_MS=60000, AUTORESPOND threshold defaults, the frontend NEXT_PUBLIC_* (baked at build time for the frontend — see §9), and AGENT_RUNTIME_PROVISIONER=shared_railway (prod runs the shared agent; ecs_per_org + its ECS runner config — cluster/subnets/SG/role ARNs/EFS id — is M2, not the cutover; see §4, §15 Q3). RAGflow envs (RAGFLOW_*) are removed post-ADR-024 — Hayhooks is the replacement.

On the agent task: HW_IMAGE_GEN_ENABLED / HW_WEB_RESEARCH_ENABLED (#4022/#4028) — kill switches, fail-safe OFF default (the agent image bakes false), set per env via agent_environment in infra/terraform/modules/stack (prod false until sign-off; staging true). Plus their model/base-url config: HW_IMAGE_GEN_MODEL, HW_IMAGE_GEN_BASE_URL, HW_WEB_RESEARCH_MODEL, HW_WEB_RESEARCH_ROUTER_MODEL. These are plain config, not secrets. See railway-env-vars.md §Agent.

Frontend caveat: NEXT_PUBLIC_* vars are inlined at next build, not at runtime. The frontend image must be built per environment with prod values (NEXT_PUBLIC_API_URL=https://api.h.work, NEXT_PUBLIC_WS_URL=wss://api.h.work, NEXT_PUBLIC_APP_ENV=production, NEXT_PUBLIC_HWORK_DOMAIN=h.work, Sentry DSN). Don't expect to set these in the task definition.

Secret rotation: Secrets Manager rotation for DB creds (works with RDS Proxy). Rotating MASTER_ENCRYPTION_KEY requires a re-encrypt migration of integration_credentials — document, don't automate.

7. IAM roles​

Three role types (least privilege):

  1. Task execution role (shared by all task defs): AmazonECSTaskExecutionRolePolicy + secretsmanager:GetSecretValue on the specific secret ARNs + ECR pull + CloudWatch Logs CreateLogStream/PutLogEvents. (The template's ${ECS_EXECUTION_ROLE_ARN}.)

  2. Task roles (per service, app-runtime perms — the template's ${ECS_TASK_ROLE_ARN}):

    • api task role: ecs:RunTask/ecs:StopTask/ecs:DescribeTasks + iam:PassRole (for the runner exec/task roles) scoped to the agent-runner task-def family — this is what ecs-per-org.provisioner.ts uses to spawn runners. Plus elasticfilesystem:*AccessPoint if it creates EFS access points dynamically. No R2 (R2 is keyed creds, not IAM).
    • agent task role + per-org runner task role: elasticfilesystem:ClientMount/ClientWrite on the EFS fs (IAM auth is ENABLED in the template). R2 via env keys.
    • frontend task role: minimal (logs only).

  3. CI/CD deploy role (GitHub OIDC, no long-lived keys): ecr:* (push), ecs:RegisterTaskDefinition/UpdateService/DescribeServices, iam:PassRole for the exec/task roles. Mirror the existing agent-ecs-deploy.yml OIDC pattern.

8. Networking​

  • VPC: 2–3 AZs. Public subnets (ALB, NAT GW), private subnets (all Fargate tasks, RDS, ElastiCache, EFS mount targets).
  • Security groups:
    • alb-public-sg: 443 from 0.0.0.0/0.
    • api-sg: 3000 from alb-public-sg; egress to redis-sg:6379, rds-proxy-sg:5432, agent-sg:8000, NAT (R2/OpenAI/etc.), ecs:RunTask (AWS API via NAT/VPC endpoint).
    • agent-sg: 8000 from api-sg only.
    • agent-runner-sg: egress to efs-sg:2049, NAT (R2/model gateway/PLATFORM_API_URL→api). Inbound: none needed (RunTask, not behind LB) — but allow 8000 from api-sg if the API health-checks runners directly.
    • frontend-sg: 3001 from alb-public-sg.
    • rds-proxy-sg: 5432 from api-sg. redis-sg: 6379 from api-sg. efs-sg: 2049 from agent-runner-sg + agent-sg.
  • VPC endpoints (cut NAT cost + keep AWS traffic private): ECR (api + dkr), S3 gateway (ECR layers), Secrets Manager, CloudWatch Logs, ECS.
  • DNS/TLS: DNS stays in Cloudflare (Terraform creates no DNS records) — after apply, CNAME api.h.work → the ALB DNS name. The ACM cert is operator-provided (you DNS-validate it via a Cloudflare CNAME and pass the ARN). Cloud Map private namespace humanwork.internal for service-to-service.

9. CI/CD pipeline​

Build → ECR → deploy, prod-only, triggered on push to main (current branch strategy: dev → staging → main). Reuse the OIDC + render-task-def + aws ecs update-service shape already in .github/workflows/agent-ecs-deploy.yml.

Per service:

  1. Build & push (*-ecr-push.yml): docker build each of api/, agent/, frontend/, tag with the git SHA + latest-prod, push to ECR. Frontend build passes prod NEXT_PUBLIC_* as --build-arg (see §6 caveat). Three ECR repos: humanwork-api, humanwork-agent, humanwork-frontend.
  2. Deploy (*-ecs-deploy.yml): envsubst the task-def template with the new image URI, RegisterTaskDefinition, UpdateService, wait for stability (the agent workflow uses a 15-min stability + /health smoke loop — copy it).
  3. Migrations: the api container runs run-migrations.sh on boot (idempotent, TypeORM-tracked). Under expand-contract (CLAUDE.md) this is safe during rolling deploys. Enable ECS deployment circuit breaker with rollback so a migration/boot failure auto-reverts.

Promotion: dev/staging keep deploying to Railway as today; only main fans out to ECS. No auto-promotion (matches gitflow-deploy.yml).

Smoke after deploy: hit https://api.h.work/ready (checks PG, agent, Redis, throttler) and https://h.work/ through the ALB; fail the job on non-200.

10. Observability​

  • Logs: awslogs driver → CloudWatch. The ecs-service module names each group /ecs/${name_prefix}-${service_name}, so with name_prefix = humanwork-<env> (humanwork-prod / humanwork-staging) that is /ecs/humanwork-<env>-api, /ecs/humanwork-<env>-agent, /ecs/humanwork-<env>-frontend. The M2 per-org runner fleet logs to its own group (/ecs/humanwork-agent in agent/deploy/ecs.deferred/task-definition.json) with a per-org/role stream prefix. Consider a Firelens→OTel/Datadog sink later.
  • Metrics: Container Insights on the cluster. Alarms: ALB 5xx rate, target unhealthy count, api /ready failures, RDS CPU/connections/free-storage, ElastiCache evictions/CPU, BullMQ depth via the existing GET /health/queues (scrape it into a CloudWatch custom metric or alert from a synthetic).
  • Errors: Sentry already wired (SENTRY_DSN, SENTRY_DSN_AGENT, SENTRY_DSN_FRONTEND) — prod invariant requires SENTRY_DSN. Set SENTRY_RELEASE to the git SHA in CI.
  • Tracing: optional ADOT sidecar later; not day-one.

11. Terraform module layout​

infra/terraform/
├── bootstrap/                 # run-once per account: S3 state bucket + GitHub OIDC provider
├── envs/
│   ├── staging/               # h852.work, smaller sizing, 2 AZs, deploys from `staging`
│   └── prod/                  # h.work, prod sizing, 3 AZs, deploys from `main`
│       ├── main.tf            # module "stack" with prod literals
│       ├── variables.tf       # operator pass-through (acm cert arn, oidc arn, state bucket arn)
│       ├── outputs.tf
│       ├── backend.tf         # S3 backend (S3-native lockfile; no DynamoDB)
│       └── *.example          # backend.hcl.example, terraform.tfvars.example
└── modules/
    ├── network/               # VPC, subnets, NAT, SGs, VPC endpoints
    ├── data/                  # RDS + RDS Proxy, ElastiCache + DATABASE_URL/REDIS_URL secrets
    ├── secrets/               # Secrets Manager containers (generated + external placeholders)
    ├── iam/                   # exec/task roles + GitHub OIDC deploy/terraform roles
    ├── ecs-cluster/           # cluster, Cloud Map namespace, Container Insights, ECR repos
    ├── alb/                   # public ALB + api target group (operator-provided ACM cert)
    ├── ecs-service/           # reusable task-def + service + autoscaling (instantiated for api, agent)
    └── stack/                 # composition: wires all of the above for one environment
  • State: S3 backend with S3-native locking (use_lockfile, no DynamoDB); one bucket, per-env keys (staging/, prod/). The shared GitHub OIDC provider lives in bootstrap/.
  • Secrets: TF creates the Secrets Manager containers — generated internal values are stored, external ones hold REPLACE_ME with ignore_changes so the real values are injected out-of-band and never enter TF-managed drift. DATABASE_URL/REDIS_URL are owned by the data module. Reference by ARN in task defs.
  • Task defs: the ecs-service module authors the api + agent task defs from one shape. Per-org runner assets (the deferred agent/deploy/ecs.deferred/ task-def + EFS access point) are M2 — no agent-runner module exists yet (see §4).

12. Sizing & rough monthly cost (us-west-2, order-of-magnitude)​

ItemConfig~$/mo
api Fargate2× (1 vCPU/2 GB) avg~$70
agent Fargate (shared)2× (1 vCPU/2 GB) avg~$70
frontendstays on Vercel (separate bill); ~$35 only if moved to ECS$0 here
per-org runnersM2 — not in cutover. bursty, org count × session minutesn/a until M2
RDS r6g.large Multi-AZ + Proxy~$450
ElastiCache r6g.large ×2~$300
ALB ×1 (+ internal if used)~$25–50
NAT GW + data2 AZ~$70+
EFSelastic, low~$10–50
Baseline (excl. runners + R2/LLM)~$1.1k–1.3k/mo

R2, OpenAI/OpenRouter, Twilio, Resend billed separately/externally. Haystack/Hayhooks is in-stack as the humanwork-rag ECS Fargate service (≈ $440/env-month target per ADR-024). The per-org runner fleet is the cost variable to watch — instrument session-minutes early.

13. Gotchas / prod invariants checklist (don't skip)​

  • DATABASE_URL → RDS Proxy endpoint (else main.ts exits 1).
  • JWT_SECRET ≥ 64 chars, MASTER_ENCRYPTION_KEY set (generate 32 bytes / 64 hex; the startup invariant only checks presence, not length), SENTRY_DSN set, DEMO_MODE unset/false (CI already greps .env.production* for DEMO_MODE=true).
  • api task stopTimeout = 120; ECS service deregistration delay ≥ 120s so BullMQ drains.
  • ALB stickiness off; ALB idle timeout ≥ 120s for WebSocket; WebSocket upgrade allowed.
  • Socket.io Redis adapter pointed at ElastiCache before cutover (else cross-pod emits silently drop).
  • No setInterval/setImmediate business work — verified, but re-check any runner-reaper code uses BullMQ.
  • rediss:// (TLS) + AUTH token for ElastiCache in transit.
  • Webhook signature verification still works behind ALB — confirm rawBody:true survives (ALB doesn't mangle the body; just verify the inbound URLs are routed to api).
  • Channel webhook public URLs updated to api.h.work in Twilio/Slack/Cloudflare/Tavus configs at cutover.
  • ECS deployment circuit breaker + rollback enabled (catches bad migrations/boots).
  • EFS access point cleanup for deprovisioned orgs (avoid unbounded growth).

14. Phased rollout​

Cutover path (like-for-like with current prod): api + shared agent + data stores. Frontend stays on Vercel; per-org runners are M2.

  1. Foundation — Terraform network + data (RDS + RDS Proxy — no pooler exists today, this is new — ElastiCache, no EFS needed until M2) + iam + ECR repos (humanwork-api, humanwork-agent) + Secrets Manager containers. Populate secrets. CREATE EXTENSION vector on RDS.
  2. DB migration dry-run — pg_dump from Railway PG → pg_restore into RDS; verify row counts + pgvector indexes. Confirms the cutover window size.
  3. Services — ecs-cluster + ecs-service ×2 (api, shared agent), Cloud Map for agent, public ALB with the operator-provided ACM cert for api; CNAME a staging hostname in Cloudflare to the ALB, run /ready + smoke while Railway still serves prod. Confirm Socket.io Redis adapter on ElastiCache.
  4. Cutover — final pg_dump/restore in a short read-only window, flip the api.h.work CNAME in Cloudflare to the ECS ALB, repoint Vercel NEXT_PUBLIC_API_URL/WS_URL → api.h.work, update channel webhook URLs (Twilio/Slack/Cloudflare/Tavus), point AGENT_SERVICE_URL at Cloud Map. Watch dashboards; keep Railway warm for DNS-flip rollback.
  5. Decommission — after a soak window, retire the Railway prod project (dev/staging stay on Railway).
  6. (M2, later) — build the ECS per-org provisioner + EFS + runner reaper; promote ecs.deferred → ecs/; set AGENT_RUNTIME_PROVISIONER=ecs_per_org; validate one org end-to-end (spawn → workspace → R2 round-trip → reap).

15. Questions — answered from the current Railway deployment​

  1. Internal agent transport. Resolved → Cloud Map. Today the shared agent is reached purely service-to-service via AGENT_SERVICE_URL (a Railway internal/public URL); nothing internet-facing hits it directly. No need for an internal ALB — register the ECS agent service in Cloud Map (agent.humanwork.internal:8000) and set AGENT_SERVICE_URL to it. Internal ALB only if you later want health-based weighted routing.

  2. KB/RAG hosting. Resolved → in-stack Haystack/Hayhooks, per ADR-024 (Accepted 2026-06-02). ADR-024 supersedes ADR-004 + ADR-009; RAGflow is being replaced by self-hosted Haystack 2.x + Hayhooks. The new RAG service is humanwork-rag (Hayhooks) on its own ECS Fargate task in the same cluster, registered in Cloud Map at humanwork-rag.humanwork.internal:1416, using pgvector on the existing humanwork Postgres (ADR-025) and OpenRouter for embed/chat/rerank (ADR-026). Deploy via the manual Deploy Rag workflow (.github/workflows/rag-deploy.yml, #1541). Downstream callers send X-API-Key (#1588). See docs/runbooks/haystack-deploy.md and PRs #1527 (RAG ECS service + RAGFlow cleanup in infra/terraform/), #1541, #1586 (curl-based container health check), #1588.

  3. ecs-per-org.provisioner.ts config surface. Resolved → it's a stub, deferred to M2. It reads no config and throws NotImplementedError. Prod is shared_railway. → Drop per-org runners from the cutover Terraform. Build the ECS provisioner + its AGENT_RUNTIME_ECS_* config (cluster/subnets/SG/roles/EFS — see §4) as M2. The Railway reference (railway-per-org.provisioner.ts) was removed in #1584 / ADR-028 — consult the pre-#1584 git history (e.g. the commit immediately before #1584 merged) for its config surface and lifecycle shape rather than expecting the file in-tree.

  4. DB migration method. Resolved → logical pg_dump/pg_restore. Today PG is Railway-managed (PG16 + pgvector); there are no existing dump/restore scripts and no pooler/pgbouncer in front of it. Migration: pg_dump Railway → pg_restore into RDS (ensure CREATE EXTENSION vector first). App migrations are idempotent on boot (api/scripts/run-migrations.sh, waits ≤60s for DB). Cutover downtime = a short read-only window during the final dump/restore; size it once you know the DB volume (likely minutes at MVP scale). DMS only if the DB is too large for a stop-the-world dump.

  5. Per-org runner cost ceiling. N/A for cutover (no runners in prod). When M2 lands: a concurrency cap + budget alarm is required before opening it up; MAX_RUNTIME_HEALTH_PER_TICK=10 already bounds health-check fan-out, but nothing caps concurrent runner count yet — add one.

  6. Region + DR. Resolved → single region, Multi-AZ only, no cross-region DR. Railway prod is US-West today; to minimize latency change pick us-west-2 for ECS (the ecs.deferred examples used us-east-1/staging.humanwork.ai placeholders, not a committed region). Multi-region (APAC) is tracked as BLK-037 and explicitly deferred beyond MVP — so Multi-AZ within one region is the target; no active DR.

  7. Frontend: Vercel vs ECS. Decision needed. Frontend is on Vercel today, not Railway. Recommendation: keep it on Vercel for v1 — it's working, decoupled from the API cutover, and avoids the NEXT_PUBLIC_* build-per-env packaging (§6). Only containerize onto ECS if you want a single platform/VPC-internal frontend. If you keep Vercel, just repoint its NEXT_PUBLIC_API_URL/NEXT_PUBLIC_WS_URL at api.h.work at cutover and skip the frontend service + its ALB default rule.

  8. Agent cold starts. Note from ADR-003: the Python agent has ~2 min cold starts (mitigated by always-on). On ECS keep agent min tasks ≥2 and disable scale-to-zero so a cold Fargate task never gates a live chat turn.

Last updated on by Degen