Skip to main content

h.work — Deployment Guide

Today (2026-06-04): Dev runs on Railway; staging + prod run on AWS ECS Fargate. Terraform + GitHub Actions landed in #1101; staging boot validated #1303; prod bootstrap #1529; CI promotion enforcement #1559; manual terraform-apply #1577; composite deploy actions #1314/#1327. The Railway-hosted setup documented below remains relevant for dev only. For the prod ECS target architecture see docs/ops/ecs-prod-deployment-spec.md. For runbook material when running on ECS see docs/runbooks/incident-response.md.

Domains by environment​

The webapp is served via app.{HWORK_DOMAIN} in every env (so production is app.h.work, staging is app.h853.work, dev is app.h852.work). Dev historically ran at the bare dev.h852.work subdomain — no app. prefix — and may still need to be in the CORS allowlist for the dev API explicitly until that legacy host is fully retired.

EnvironmentFrontendAPIHWORK_DOMAIN env varSpecialist email patternCORS allowlist (ALLOWED_ORIGINS)
devhttps://app.h852.work (legacy: https://dev.h852.work)https://api.h852.work (or Railway URL)h852.work{name}@h852.workhttps://app.h852.work,https://*.h852.work,https://dev.h852.work
staginghttps://app.h853.workhttps://api.h853.workh853.work{name}@h853.workhttps://app.h853.work,https://*.h853.work
productionhttps://app.h.work + per-org {slug}.h.workhttps://api.h.workh.work{name}@h.workhttps://app.h.work,https://*.h.work

⚠️ The domain humanwork.ai is NOT used. Older docs may reference it from a pre-pivot snapshot — replace with h.work (prod) when found.

Source of truth: api/src/common/env.ts + .env.example — HWORK_DOMAIN and NEXT_PUBLIC_HWORK_DOMAIN control all domain-derived behavior (CORS, Specialist email, client portal subdomain). If you see SPECIALIST_EMAIL_DOMAIN or NEXT_PUBLIC_APP_DOMAIN anywhere, those are deprecated — HWORK_DOMAIN replaced them per CLAUDE.md.

Canonical zone-to-env mapping: see hp-knowledge-base — Domains & Environments.

Hosting + database pooler per environment​

EnvironmentRuntimeDatabaseConnection pooler
dev (api.h852.work)RailwayRailway Postgres (postgres.railway.internal:5432)None — direct connection. Pool size defaults to DATABASE_POOL_MAX=20 (raised from 5 in #2540: the Ops setup wizard step-3 fans out ~8 parallel API calls and a pool of 5 exhausted under burst).
staging (api.h853.work)AWS ECS FargateRDS PostgreSQLAWS RDS Proxy (Terraform: infra/terraform/modules/data/main.tf, resource aws_db_proxy.this)
production (api.h.work)AWS ECS FargateRDS PostgreSQLAWS RDS Proxy (Terraform: infra/terraform/modules/data/main.tf, resource aws_db_proxy.this)

The dev environment is intentionally left without a pooler (decision 2026-06-05, #1233). RDS Proxy handles all production pooling responsibilities on AWS; bringing up a separate PgBouncer on Railway dev added operational surface area for very little benefit at dev's load profile. If multi-pod connection saturation ever becomes a real issue on Railway dev, the path is either (a) put PgBouncer back in — Railway plugin still exists, just unwired — or (b) accelerate the dev-to-AWS migration. Don't reintroduce PgBouncer reflexively; check whether you're actually saturating first.

The startup invariant in api/src/common/startup-invariants.ts enforces in production builds (APP_ENV=production) that DATABASE_URL resolves to a known pooler hostname pattern (*.proxy-*.<region>.rds.amazonaws.com for RDS Proxy, pgbouncer substring for the Railway path). The invariant is dormant on dev because dev runs APP_ENV=development (note: Railway sets NODE_ENV=production on all Railway services — including dev — so NODE_ENV is not the distinguishing variable; APP_ENV is).

Architecture​

Scope below: dev / Railway only. Staging + prod run on AWS ECS Fargate per ADR-028; for those envs use docs/ops/ecs-prod-deployment-spec.md and the Terraform/GitHub Actions wiring in #1101 / #1303 / #1559. The Railway diagram and steps in this section apply to the dev environment only.

Internet
   │
   ├─→ Vercel (CDN)          frontend (Next.js) — staging + prod redirected to AWS ALB after #1720
   │
   ├─→ AWS ALB (staging, prod)
   │      └─ ECS Fargate: humanwork-api, humanwork-agent, humanwork-rag
   │             ↘ RDS PostgreSQL via RDS Proxy
   │             ↘ ElastiCache Redis
   │
   └─→ Railway (dev only)
          ├─ humanwork-api          NestJS  :3000  /health
          ├─ humanwork-agent        FastAPI :8000  /health
          ├─ Postgres               PostgreSQL 16 + pgvector (direct, no pooler)
          └─ redis-dev              Redis 7

Prerequisites​

  • Railway CLI: npm install -g @railway/cli
  • Vercel CLI: npm install -g vercel
  • Railway account with a project created
  • Vercel account with a project created

1. Railway — First-time Setup​

# Authenticate
railway login

# Link to existing project (or create new)
railway link

Create services in Railway dashboard​

Create 4 services manually in the Railway UI:

  • humanwork-postgres — add PostgreSQL plugin
  • humanwork-redis — add Redis plugin
  • humanwork-api — point to api/ directory
  • humanwork-agent — point to agent/ directory

Set environment variables​

For each service, configure via Railway dashboard → Variables, or use the CLI:

# API service
railway variables set \
  DATABASE_URL="$DATABASE_URL" \
  REDIS_URL="$REDIS_URL" \
  NODE_ENV=production \
  JWT_SECRET="$(openssl rand -hex 32)" \
  OPENAI_API_KEY="$OPENAI_API_KEY" \
  AGENT_SERVICE_URL="https://humanwork-agent.up.railway.app" \
  ALLOWED_ORIGINS="https://app.h.work,https://*.h.work" \
  HWORK_DOMAIN=h.work \
  PLATFORM_API_TOKEN="$(openssl rand -hex 32)" \
  DATABASE_SSL=true \
  --service humanwork-api

# Agent service
railway variables set \
  OPENAI_API_KEY="$OPENAI_API_KEY" \
  PLATFORM_API_URL="https://humanwork-api.up.railway.app" \
  PLATFORM_API_TOKEN="$PLATFORM_API_TOKEN" \
  --service humanwork-agent

Dev API Tailscale userspace mode for remote desktop (#3769)​

The remote-desktop PoC is dev-only. The Railway dev API must be joined to the Tailscale tailnet so /v1/remote-desktop can open a TCP connection to the Mac Mini running websockify. This is an operator step, not application code: do not start tailscaled from NestJS and do not commit the auth key.

Required dev API variables:

railway variables set \
  TS_AUTHKEY="$TS_AUTHKEY" \
  --service humanwork-api

The API container/runtime must run userspace tailscaled with that auth key before starting node /app/dist/src/main. Keep the key reusable only if the Tailscale admin policy requires it, tag it for the dev API service, and rotate it after the PoC if it is no longer needed.

The single dev machine registry row is data, not application logic. After the org_machines migration has run on dev, seed the Humanity dev org machine row from api/scripts/dev-org-machine.sql:

cd api
railway run --service humanwork-api psql "$DATABASE_URL" -f scripts/dev-org-machine.sql

For local verification without Tailscale, point the proxy at a local websockify or fake TCP server instead of the registry target:

REMOTE_DESKTOP_TARGET_HOST=127.0.0.1
REMOTE_DESKTOP_TARGET_PORT=8900

2. Railway — Deploy Services​

# Deploy API
cd api && railway up --service humanwork-api

# Deploy Agent
cd ../agent && railway up --service humanwork-agent

Logs:

railway logs --service humanwork-api
railway logs --service humanwork-agent

3. First-time Setup (Migrations + Seed)​

After the API service is running for the first time, initialize the database:

# 1. Run migrations
cd api && npm run typeorm migration:run -- -d dist/data-source.js

# 2. Seed demo users (dev / staging only — do NOT run on production)
npm run seed

The seed script is idempotent — safe to run multiple times.

Demo credentials (dev/staging only)​

See README.md for the canonical seed-account list. Run bash scripts/seed-dev.sh [dev|staging] to seed. Highlights:

EmailPasswordRole
e@humanity.org2a7e7f02superadmin
sarah.chen@h852.work881b1d6baccount_manager
david.kim@h852.work881b1d6bexpert
amy@acmefinancial.com881b1d6borg_admin (Acme Financial)

3a. Database Extensions​

pgvector and pg_trgm extensions are installed automatically on first startup via scripts/setup-db.sh (mounted as a postgres init script in Docker).

For Railway's managed postgres, run once after first deploy:

railway run --service humanwork-postgres \
  psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector; CREATE EXTENSION IF NOT EXISTS pg_trgm;"

4. Vercel — Deploy Frontend​

cd frontend

# First time: link project
vercel link

# Set env vars
vercel env add NEXT_PUBLIC_API_URL production    # https://api.h.work
vercel env add NEXT_PUBLIC_WS_URL production     # wss://api.h.work
# Deploy
vercel --prod

5. GitHub Actions — CI/CD Secrets​

Add these secrets to the repo (Settings → Secrets → Actions):

SecretValue
RAILWAY_TOKENrailway whoami --token output
VERCEL_TOKENVercel account token
VERCEL_ORG_IDFrom .vercel/project.json
VERCEL_PROJECT_IDFrom .vercel/project.json
API_URLhttps://api.h.work

Once set, every push to main triggers auto-deploy. Branch strategy: dev → staging → main (production). Promotion is CI-gated per PR #1559 / docs/promotion-flow.md. Terraform apply is manual-only (workflow_dispatch) per #1577 — infra changes never auto-apply on push.

6. Health Checks​

API_URL=https://api.h.work    # prod; staging: https://api.h853.work, dev: https://api.h852.work

# Liveness
curl "$API_URL/health"

# Readiness (DB + agent connectivity)
curl "$API_URL/ready"

# Metrics
curl "$API_URL/metrics"

7. Local Development​

cp .env.example .env
# Set OPENAI_API_KEY in .env

docker compose up --build

# Verify
curl http://localhost:3000/health

The docker-compose.override.yml is auto-merged and enables hot reload for all services.

8. Rollback​

Railway keeps the last 5 deploys. To rollback:

# Via dashboard: Railway → Service → Deployments → select previous → Rollback

# Via CLI (get deploy ID from logs)
railway rollback --service humanwork-api --deployment <deploy-id>

Troubleshooting​

SymptomCheck
API 503 on /healthrailway logs --service humanwork-api — likely DB timeout
Migrations failCheck DATABASE_URL and DATABASE_SSL=true on Railway
Agent unreachableVerify AGENT_SERVICE_URL (today: Railway internal URL. Post-ECS cutover: Cloud Map agent.humanwork.internal:8000 — see ecs-prod-deployment-spec.md §15 Q1.)
WhatsApp webhook failingVerify TWILIO_SUBACCOUNT_TOKEN in Railway env vars; check Twilio Console for errors
CORS errorsAdd origin to ALLOWED_ORIGINS env var (comma-separated)

Environment Variables​

APP_ENV — The Canonical Environment Flag​

APP_ENV (backend) / NEXT_PUBLIC_APP_ENV (frontend) is the authoritative variable for distinguishing environments at the feature/logic level.

ValueWhere
developmentLocal dev (default)
stagingRailway staging env / Vercel preview
productionRailway production env / Vercel production

Why not NODE_ENV? Next.js sets NODE_ENV=production in any production build, including staging. APP_ENV is explicitly set per Railway/Vercel environment so you can tell staging from production.

Rule: Never set NODE_ENV manually. It is controlled by Node.js and Next.js automatically.

Setting it on Railway​

# Staging environment
railway variables set APP_ENV=staging --environment staging
railway variables set HWORK_DOMAIN=h853.work --environment staging

# Production environment
railway variables set APP_ENV=production --environment production
railway variables set HWORK_DOMAIN=h.work --environment production

Setting it on Vercel​

In Vercel project settings → Environment Variables:

  • NEXT_PUBLIC_APP_ENV = staging → Preview environment
  • NEXT_PUBLIC_APP_ENV = production → Production environment
  • NEXT_PUBLIC_HWORK_DOMAIN = h852.work → Staging/Preview
  • NEXT_PUBLIC_HWORK_DOMAIN = h.work → Production

Note: HWORK_DOMAIN (backend) and NEXT_PUBLIC_HWORK_DOMAIN (frontend) replace the old SPECIALIST_EMAIL_DOMAIN and NEXT_PUBLIC_APP_DOMAIN variables. Remove the old variables from any env where they appear.

Usage in code​

// Backend (api/src/common/env.ts)
import { IS_PRODUCTION, IS_STAGING, IS_DEPLOYED } from "./common/env";

// Frontend (frontend/src/lib/env.ts)
import { IS_PRODUCTION, IS_STAGING, IS_DEPLOYED } from "@/lib/env";

NODE_ENV is still appropriate for purely technical Node.js concerns: TypeORM SQL logging, DB SSL, and secure-cookie flags — these are correct as-is.

9. Multi-pod safety (#865, #1007)​

The API runs active-active across multiple pods (compose/Railway). Two contracts must hold:

  1. Idempotent webhook/inbound paths — channel webhooks (Slack/Email/Telegram/WhatsApp) use the dedup keys defined in ADR-013. Replays land on the same conversation.
  2. Leader-gated cron — periodic jobs (digest emails, queue-stale sweeps, reconciliation) are wrapped in a leader-election lock (#1007). Without this, every pod runs the cron and clients receive N duplicate digests. Cron handlers MUST call leaderLock.acquire(jobName) before executing.

See ADR-018 (Multi-Pod Safety Contract) for the full invariant set. Adding a new cron without leader-gating is a code-review block.

10. Sentry (#845)​

Both agent (Python) and frontend (Next.js) emit to Sentry. The API has emitted since earlier rollouts.

Required env vars (all three services):

SENTRY_DSN=<service-specific DSN>
SENTRY_ENVIRONMENT=<dev|staging|production>   # mirrors APP_ENV
SENTRY_TRACES_SAMPLE_RATE=0.1                 # 10% in prod; raise to 1.0 in staging

Source maps for the frontend are uploaded as part of the vercel --prod step (sentry-cli releases files <ver> upload-sourcemaps). If a release shows minified stack traces only, the upload step failed — check the Vercel build log for the Sentry CLI block.

11. Throttler — shadow mode (#812, #839)​

Per ADR-021 (renumbered 2026-06-01 via #1229), the global ThrottlerModule runs in shadow mode for the first soak window after deploy:

  • #812 — global API throttler (Redis-backed storage, cross-pod shared counters).
  • #839 — per-org channel-webhook throttle (Slack/Email/WhatsApp inbound).

Shadow-mode flag: THROTTLER_MODE=shadow → guards count + log would-have-blocked but do NOT 429. Promote to enforce by setting THROTTLER_MODE=enforce once the dashboard shows steady-state below limits for ≥1 week.

Required env (both modes):

REDIS_URL=<...>                  # MUST be reachable from every API pod
THROTTLER_MODE=shadow|enforce
THROTTLER_WEBHOOK_RPS=<per-org limit>

12. Lago billing (#1069, #1105, #1155)​

The platform bills via Lago — self-hosted; dev on Railway, staging + prod on AWS (consolidated single instance serving both via separate Lago orgs). AWS Lago bring-up tracked in #1287.

Workspace setup steps:

  1. Create the Lago org / workspace (one per HP environment — dev, staging, prod — under the Lago instance the env points at: dev → Railway Lago; staging + prod → AWS Lago).
  2. Issue an API key with scope events:write, customers:read, subscriptions:read.
  3. Wire env on the API service (canonical framing — see docs/ops/railway-env-vars.md §Lago for the per-env URL):
LAGO_API_KEY=<scoped key>
LAGO_API_URL=<Lago instance URL — Railway for dev; AWS for staging + prod (see #1287)>
LAGO_WEBHOOK_SECRET=<hmac shared secret>
  1. Run the customer-backfill script (scripts/backfill-lago-customers.ts) once per environment to sync existing orgs.

Cross-ref: docs/decisions/2026-05-29-lago-pricing-strategy.md defines the pricing model; #1069 wires the event emitters, #1105 the customer sync, #1155 the subscription/invoice surface in /ops/billing. The previously-evaluated Orb vendor is not in use — any Orb references in older docs are historical.

13. Security: Production startup invariants (P0-5)​

The API validates critical environment variables at startup via validateProductionInvariants() in api/src/common/startup-invariants.ts. The process exits with code 1 if any invariant fails, preventing unsafe deployments.

Enforced in production (APP_ENV=production):

InvariantWhat it checksWhy
DEMO_MODE ≠ trueprocess.env.DEMO_MODE !== 'true'/auth/demo-login hands out superadmin JWTs without authentication when DEMO_MODE=true. Single highest-blast-radius misconfiguration. See CLAUDE.md "Security Rules" #5.
JWT_SECRET lengthJWT_SECRET.length >= 64Weak secrets enable JWT forgery
SENTRY_DSN present!!SENTRY_DSNObservability compliance — prod must report errors
MASTER_ENCRYPTION_KEY present!!MASTER_ENCRYPTION_KEYData encryption compliance — encrypts sensitive DB fields
DATABASE_URL uses poolerURL hostname matches *.proxy.rlwy.net, *pgbouncer*, etc.Multi-pod safety — direct connections exhaust PG max_connections
REDIS_URL present!!REDIS_URLCross-pod message passing — BullMQ, Socket.io, throttler, circuit breaker all degrade silently to in-memory mode without it (#1354)

Defense in depth for DEMO_MODE:

  1. Startup guard (runtime): validateProductionInvariants() exits before HTTP listener binds
  2. CI grep guard (pre-deploy): .github/workflows/ci.yml fails if .env.production* contains DEMO_MODE=true
  3. Request-time guard (code regression protection): /auth/demo-login endpoint throws ForbiddenException when APP_ENV === 'production'

All three layers are spec'd with Jest tests (see api/src/common/startup-invariants.spec.ts and api/src/auth/demo-login.spec.ts).

Scanned files (CI + startup):

  • .env.production* — any file matching this glob is checked by CI
  • Runtime environment variables read from Railway/ECS at process start

Behavior on violation:

  • CI: Job check-demo-mode fails with exit code 1, blocking merge/deploy
  • Startup: Process logs error summary and exits with code 1 before binding port
  • Request-time: Endpoint returns HTTP 403 Forbidden

To add a new invariant:

  1. Add check to validateProductionInvariants() and checkProductionInvariants() arrays
  2. Add test cases to startup-invariants.spec.ts
  3. Update this runbook table

14. Migration runner — transaction-each mode (#953)​

The TypeORM migration runner is configured transaction: 'each' (per-migration transactions) rather than 'all' (one big transaction). This is what enables CREATE INDEX CONCURRENTLY migrations — Postgres rejects concurrent index builds inside an outer transaction.

Practical rule:

  • OK in same migration: schema-only DDL, transactional DDL, CREATE INDEX (non-concurrent).
  • Must be its own migration: CREATE INDEX CONCURRENTLY, REINDEX CONCURRENTLY, VACUUM, anything documented as "cannot run inside a transaction block".

A migration that mixes concurrent and non-concurrent statements will fail at runtime even though TypeScript compiles. See #953 for the migration linter that catches this in CI.

Last updated on by franky