h.work — Deployment Guide

Today (2026-06-04): Dev runs on Railway; staging + prod run on AWS ECS Fargate. Terraform + GitHub Actions landed in #1101; staging boot validated #1303; prod bootstrap #1529; CI promotion enforcement #1559; manual terraform-apply #1577; composite deploy actions #1314/#1327. The Railway-hosted setup documented below remains relevant for dev only. For the prod ECS target architecture see docs/ops/ecs-prod-deployment-spec.md. For runbook material when running on ECS see docs/runbooks/incident-response.md.

Domains by environment

The webapp is served via app.{HWORK_DOMAIN} in every env (so production is app.h.work, staging is app.h853.work, dev is app.h852.work). Dev historically ran at the bare dev.h852.work subdomain — no app. prefix — and may still need to be in the CORS allowlist for the dev API explicitly until that legacy host is fully retired.

Environment	Frontend	API	HWORK_DOMAIN env var	Specialist email pattern	CORS allowlist (ALLOWED_ORIGINS)
dev	https://app.h852.work (legacy: https://dev.h852.work)	https://api.h852.work (or Railway URL)	h852.work	{name}@h852.work	https://app.h852.work,https://*.h852.work,https://dev.h852.work
staging	https://app.h853.work	https://api.h853.work	h853.work	{name}@h853.work	https://app.h853.work,https://*.h853.work
production	https://app.h.work + per-org {slug}.h.work	https://api.h.work	h.work	{name}@h.work	https://app.h.work,https://*.h.work

⚠️ The domain humanwork.ai is NOT used. Older docs may reference it from a pre-pivot snapshot — replace with h.work (prod) when found.

Source of truth: api/src/common/env.ts + .env.example — HWORK_DOMAIN and NEXT_PUBLIC_HWORK_DOMAIN control all domain-derived behavior. SPECIALIST_EMAIL_DOMAIN and NEXT_PUBLIC_APP_DOMAIN are deprecated.

Canonical zone-to-env mapping: see hp-knowledge-base — Domains & Environments.

---

Hosting + database pooler per environment

Environment	Runtime	Database	Connection pooler
dev (api.h852.work)	Railway	Railway Postgres (postgres.railway.internal:5432)	None — direct connection. Pool size defaults to DATABASE_POOL_MAX=20 (raised from 5 in #2540: the Ops setup wizard step-3 fans out ~8 parallel API calls and a pool of 5 exhausted under burst).
staging (api.h853.work)	AWS ECS Fargate	RDS PostgreSQL	AWS RDS Proxy (Terraform: infra/terraform/modules/data/main.tf, resource aws_db_proxy.this)
production (api.h.work)	AWS ECS Fargate	RDS PostgreSQL	AWS RDS Proxy (Terraform: infra/terraform/modules/data/main.tf, resource aws_db_proxy.this)

The dev environment is intentionally left without a pooler (decision 2026-06-05, #1233). RDS Proxy handles all production pooling responsibilities on AWS; bringing up a separate PgBouncer on Railway dev added operational surface area for very little benefit at dev's load profile. If multi-pod connection saturation ever becomes a real issue on Railway dev, the path is either (a) put PgBouncer back in — Railway plugin still exists, just unwired — or (b) accelerate the dev-to-AWS migration. Don't reintroduce PgBouncer reflexively; check whether you're actually saturating first.

The startup invariant in api/src/common/startup-invariants.ts enforces in production builds (APP_ENV=production) that DATABASE_URL resolves to a known pooler hostname pattern (*.proxy-*.<region>.rds.amazonaws.com for RDS Proxy, pgbouncer substring for the Railway path). The invariant is dormant on dev because dev runs APP_ENV=development (note: Railway sets NODE_ENV=production on all Railway services — including dev — so NODE_ENV is not the distinguishing variable; APP_ENV is).

---

Architecture

Scope below: dev / Railway only. Staging + prod run on AWS ECS Fargate per ADR-028; for those envs use docs/ops/ecs-prod-deployment-spec.md and the Terraform/GitHub Actions wiring in #1101 / #1303 / #1559. The Railway diagram and steps in this section apply to the dev environment only.

Internet
   │
   ├─→ Vercel (CDN)          frontend (Next.js) — staging + prod redirected to AWS ALB after #1720
   │
   ├─→ AWS ALB (staging, prod)
   │      └─ ECS Fargate: humanwork-api, humanwork-agent, humanwork-rag
   │             ↘ RDS PostgreSQL via RDS Proxy
   │             ↘ ElastiCache Redis
   │
   └─→ Railway (dev only)
          ├─ humanwork-api          NestJS  :3000  /health
          ├─ humanwork-agent        FastAPI :8000  /health
          ├─ Postgres               PostgreSQL 16 + pgvector (direct, no pooler)
          └─ redis-dev              Redis 7

---

Prerequisites

• Railway CLI: npm install -g @railway/cli
• Vercel CLI: npm install -g vercel
• Railway account with a project created
• Vercel account with a project created

---

1. Railway — First-time Setup

# Authenticate
railway login

# Link to existing project (or create new)
railway link

Create services in Railway dashboard

Create 4 services manually in the Railway UI:

• humanwork-postgres — add PostgreSQL plugin
• humanwork-redis — add Redis plugin
• humanwork-api — point to api/ directory
• humanwork-agent — point to agent/ directory

Set environment variables

For each service, configure via Railway dashboard → Variables, or use the CLI:

# API service
railway variables set \
  DATABASE_URL="$DATABASE_URL" \
  REDIS_URL="$REDIS_URL" \
  NODE_ENV=production \
  JWT_SECRET="$(openssl rand -hex 32)" \
  OPENAI_API_KEY="$OPENAI_API_KEY" \
  AGENT_SERVICE_URL="https://humanwork-agent.up.railway.app" \
  ALLOWED_ORIGINS="https://app.h.work,https://*.h.work" \
  HWORK_DOMAIN=h.work \
  PLATFORM_API_TOKEN="$(openssl rand -hex 32)" \
  DATABASE_SSL=true \
  --service humanwork-api

# Agent service
railway variables set \
  OPENAI_API_KEY="$OPENAI_API_KEY" \
  PLATFORM_API_URL="https://humanwork-api.up.railway.app" \
  PLATFORM_API_TOKEN="$PLATFORM_API_TOKEN" \
  --service humanwork-agent

Expert Computer / remote desktop

Expert Computer is a multi-transport production feature, not the retired VNC-only PoC. Machine provisioning, WebRTC/TURN, connector enrollment, TCC, Tailscale relay egress, direct WSS and recovery are documented in the canonical deployment and operations guide.

The API relay dials a raw VNC port; do not point org_machines.vnc_port at websockify. Tailscale remains an operator-managed network dependency and must not be started from NestJS. Local-only relay tests may use REMOTE_DESKTOP_TARGET_HOST and REMOTE_DESKTOP_TARGET_PORT.

---

2. Railway — Deploy Services

# Deploy API
cd api && railway up --service humanwork-api

# Deploy Agent
cd ../agent && railway up --service humanwork-agent

Logs:

railway logs --service humanwork-api
railway logs --service humanwork-agent

Dataset/schema changes spanning api + rag — deploy rag FIRST

The rag service validates the dataset field of ingest/retrieval requests against pydantic Literals, so an api that starts sending a new dataset name (e.g. slack-archive) to an older rag gets hard 422s until rag catches up. For any dataset addition, deploy rag BEFORE api.

On Railway dev, rag has no auto-deploy — it ships only via:

cd rag && railway up --service humanwork-rag

This exact gap caused the first live Slack-archive sync tick to fail on 2026-07-28: api was deployed with the new slack-archive dataset while dev rag was still on the previous build.

---

3. First-time Setup (Migrations + Seed)

After the API service is running for the first time, initialize the database:

# 1. Run migrations
cd api && npm run typeorm migration:run -- -d dist/data-source.js

# 2. Seed demo users (dev / staging only — do NOT run on production)
npm run seed

The seed script is idempotent — safe to run multiple times.

Demo credentials (dev/staging only)

See README.md for the canonical seed-account list. Run bash scripts/seed-dev.sh [dev|staging] to seed. Highlights:

Email	Password	Role
e@humanity.org	2a7e7f02	superadmin
sarah.chen@h852.work	881b1d6b	account_manager
david.kim@h852.work	881b1d6b	expert
amy@acmefinancial.com	881b1d6b	org_admin (Acme Financial)

---

3a. Database Extensions

pgvector and pg_trgm extensions are installed automatically on first startup via scripts/setup-db.sh (mounted as a postgres init script in Docker).

For Railway's managed postgres, run once after first deploy:

railway run --service humanwork-postgres \
  psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector; CREATE EXTENSION IF NOT EXISTS pg_trgm;"

---

4. Vercel — Deploy Frontend

cd frontend

# First time: link project
vercel link

# Set env vars
vercel env add NEXT_PUBLIC_API_URL production    # https://api.h.work
vercel env add NEXT_PUBLIC_WS_URL production     # wss://api.h.work
# Deploy
vercel --prod

---

5. GitHub Actions — CI/CD Secrets

Add these secrets to the repo (Settings → Secrets → Actions):

Secret	Value
RAILWAY_TOKEN	railway whoami --token output
VERCEL_TOKEN	Vercel account token
VERCEL_ORG_ID	From .vercel/project.json
VERCEL_PROJECT_ID	From .vercel/project.json
API_URL	https://api.h.work

Once set, every push to main triggers auto-deploy. Branch strategy: dev → staging → main (production). Promotion is CI-gated per PR #1559 / docs/promotion-flow.md. Terraform apply is manual-only (workflow_dispatch) per #1577 — infra changes never auto-apply on push.

---

6. Health Checks

API_URL=https://api.h.work    # prod; staging: https://api.h853.work, dev: https://api.h852.work

# Liveness
curl "$API_URL/health"

# Readiness (DB + agent connectivity)
curl "$API_URL/ready"

# Metrics
curl "$API_URL/metrics"

---

7. Local Development

cp .env.example .env
# Add the local model credential consumed by dev-up.
./dev-up.sh

# Verify
curl http://localhost:3000/health

The root docker-compose.yml is the consolidated local browser-development stack. dev-up.sh also builds and exports the immutable managed-Hermes runner image before starting it.

---

8. Rollback

Railway keeps the last 5 deploys. To rollback:

# Via dashboard: Railway → Service → Deployments → select previous → Rollback

# Via CLI (get deploy ID from logs)
railway rollback --service humanwork-api --deployment <deploy-id>

---

Troubleshooting

Symptom	Check
API 503 on /health	railway logs --service humanwork-api — likely DB timeout
Migrations fail	Check DATABASE_URL and DATABASE_SSL=true on Railway
Agent unreachable	Verify AGENT_SERVICE_URL (today: Railway internal URL. Post-ECS cutover: Cloud Map agent.humanwork.internal:8000 — see ecs-prod-deployment-spec.md §15 Q1.)
WhatsApp webhook failing	Verify TWILIO_SUBACCOUNT_TOKEN in Railway env vars; check Twilio Console for errors
CORS errors	Add origin to ALLOWED_ORIGINS env var (comma-separated)

---

Environment Variables

APP_ENV — The Canonical Environment Flag

APP_ENV (backend) / NEXT_PUBLIC_APP_ENV (frontend) is the authoritative variable for distinguishing environments at the feature/logic level.

Value	Where
development	Local dev (default)
staging	Railway staging env / Vercel preview
production	Railway production env / Vercel production

Why not NODE_ENV? Next.js sets NODE_ENV=production in any production build, including staging. APP_ENV is explicitly set per Railway/Vercel environment so you can tell staging from production.

Rule: Never set NODE_ENV manually. It is controlled by Node.js and Next.js automatically.

Setting it on Railway

# Staging environment
railway variables set APP_ENV=staging --environment staging
railway variables set HWORK_DOMAIN=h853.work --environment staging

# Production environment
railway variables set APP_ENV=production --environment production
railway variables set HWORK_DOMAIN=h.work --environment production

Setting it on Vercel

In Vercel project settings → Environment Variables:

• NEXT_PUBLIC_APP_ENV = staging → Preview environment
• NEXT_PUBLIC_APP_ENV = production → Production environment
• NEXT_PUBLIC_HWORK_DOMAIN = h852.work → Staging/Preview
• NEXT_PUBLIC_HWORK_DOMAIN = h.work → Production

Note: HWORK_DOMAIN (backend) and NEXT_PUBLIC_HWORK_DOMAIN (frontend) replace the old SPECIALIST_EMAIL_DOMAIN and NEXT_PUBLIC_APP_DOMAIN variables. Remove the old variables from any env where they appear.

Usage in code

// Backend (api/src/common/env.ts)
import { IS_PRODUCTION, IS_STAGING, IS_DEPLOYED } from "./common/env";

// Frontend (frontend/src/lib/env.ts)
import { IS_PRODUCTION, IS_STAGING, IS_DEPLOYED } from "@/lib/env";

NODE_ENV is still appropriate for purely technical Node.js concerns: TypeORM SQL logging, DB SSL, and secure-cookie flags — these are correct as-is.

---

9. Multi-pod safety (#865, #1007)

The API runs active-active across multiple pods (compose/Railway). Two contracts must hold:

1. Idempotent webhook/inbound paths — channel webhooks (Slack/Email/Telegram/WhatsApp) use the dedup keys defined in ADR-013. Replays land on the same conversation.
2. Leader-gated cron — periodic jobs (digest emails, queue-stale sweeps, reconciliation) are wrapped in a leader-election lock (#1007). Without this, every pod runs the cron and clients receive N duplicate digests. Cron handlers MUST call leaderLock.acquire(jobName) before executing.

See ADR-018 (Multi-Pod Safety Contract) for the full invariant set. Adding a new cron without leader-gating is a code-review block.

---

10. Sentry (#845)

Both agent (Python) and frontend (Next.js) emit to Sentry. The API has emitted since earlier rollouts.

Required env vars (all three services):

SENTRY_DSN=<service-specific DSN>
SENTRY_ENVIRONMENT=<dev|staging|production>   # mirrors APP_ENV
SENTRY_TRACES_SAMPLE_RATE=0.1                 # 10% in prod; raise to 1.0 in staging

Source maps for the frontend are uploaded as part of the vercel --prod step (sentry-cli releases files <ver> upload-sourcemaps). If a release shows minified stack traces only, the upload step failed — check the Vercel build log for the Sentry CLI block.

---

11. Throttler — shadow mode (#812, #839)

Per ADR-021 (renumbered 2026-06-01 via #1229), the global ThrottlerModule runs in shadow mode for the first soak window after deploy:

• #812 — global API throttler (Redis-backed storage, cross-pod shared counters).
• #839 — per-org channel-webhook throttle (Slack/Email/WhatsApp inbound).

Shadow-mode flag: THROTTLER_MODE=shadow → guards count + log would-have-blocked but do NOT 429. Promote to enforce by setting THROTTLER_MODE=enforce once the dashboard shows steady-state below limits for ≥1 week.

Required env (both modes):

REDIS_URL=<...>                  # MUST be reachable from every API pod
THROTTLER_MODE=shadow|enforce
THROTTLER_WEBHOOK_RPS=<per-org limit>

---

12. Lago billing (#1069, #1105, #1155)

The platform bills via Lago — self-hosted; dev on Railway, staging + prod on AWS (consolidated single instance serving both via separate Lago orgs). AWS Lago bring-up tracked in #1287.

Workspace setup steps:

1. Create the Lago org / workspace (one per HP environment — dev, staging, prod — under the Lago instance the env points at: dev → Railway Lago; staging + prod → AWS Lago).
2. Issue an API key with scope events:write, customers:read, subscriptions:read.
3. Wire env on the API service (canonical framing — see docs/ops/railway-env-vars.md §Lago for the per-env URL):

LAGO_API_KEY=<scoped key>
LAGO_API_URL=<Lago instance URL — Railway for dev; AWS for staging + prod (see #1287)>
LAGO_WEBHOOK_SECRET=<hmac shared secret>

4. Run the customer-backfill script (scripts/backfill-lago-customers.ts) once per environment to sync existing orgs.

Cross-ref: docs/decisions/2026-05-29-lago-pricing-strategy.md defines the pricing model; #1069 wires the event emitters, #1105 the customer sync, #1155 the subscription/invoice surface in /ops/billing. The previously-evaluated Orb vendor is not in use — any Orb references in older docs are historical.

---

13. Security: Production startup invariants (P0-5)

The API validates critical environment variables at startup via validateProductionInvariants() in api/src/common/startup-invariants.ts. The process exits with code 1 if any invariant fails, preventing unsafe deployments.

Enforced in production (APP_ENV=production):

Invariant	What it checks	Why
DEMO_MODE ≠ true	process.env.DEMO_MODE !== 'true'	/auth/demo-login hands out debug-role JWTs without authentication when DEMO_MODE=true; api/src/auth/auth.controller.ts and deployment env guards are authoritative.
JWT_SECRET length	JWT_SECRET.length >= 64	Weak secrets enable JWT forgery
SENTRY_DSN present	!!SENTRY_DSN	Observability compliance — prod must report errors
MASTER_ENCRYPTION_KEY present	!!MASTER_ENCRYPTION_KEY	Data encryption compliance — encrypts sensitive DB fields
DATABASE_URL uses pooler	URL hostname matches *.proxy.rlwy.net, *pgbouncer*, etc.	Multi-pod safety — direct connections exhaust PG max_connections
REDIS_URL present	!!REDIS_URL	Cross-pod message passing — BullMQ, Socket.io, throttler, circuit breaker all degrade silently to in-memory mode without it (#1354)

Defense in depth for DEMO_MODE:

1. Startup guard (runtime): validateProductionInvariants() exits before HTTP listener binds
2. CI grep guard (pre-deploy): .github/workflows/ci.yml fails if .env.production* contains DEMO_MODE=true
3. Request-time guard (code regression protection): /auth/demo-login endpoint throws ForbiddenException when APP_ENV === 'production'

All three layers are spec'd with Jest tests (see api/src/common/startup-invariants.spec.ts and api/src/auth/demo-login.spec.ts).

Scanned files (CI + startup):

• .env.production* — any file matching this glob is checked by CI
• Runtime environment variables read from Railway/ECS at process start

Behavior on violation:

• CI: Job check-demo-mode fails with exit code 1, blocking merge/deploy
• Startup: Process logs error summary and exits with code 1 before binding port
• Request-time: Endpoint returns HTTP 403 Forbidden

To add a new invariant:

1. Add check to validateProductionInvariants() and checkProductionInvariants() arrays
2. Add test cases to startup-invariants.spec.ts
3. Update this runbook table

---

14. Migration runner — transaction-each mode (#953)

The TypeORM migration runner is configured transaction: 'each' (per-migration transactions) rather than 'all' (one big transaction). This is what enables CREATE INDEX CONCURRENTLY migrations — Postgres rejects concurrent index builds inside an outer transaction.

Practical rule:

• OK in same migration: schema-only DDL, transactional DDL, CREATE INDEX (non-concurrent).
• Must be its own migration: CREATE INDEX CONCURRENTLY, REINDEX CONCURRENTLY, VACUUM, anything documented as "cannot run inside a transaction block".

A migration that mixes concurrent and non-concurrent statements will fail at runtime even though TypeScript compiles. See #953 for the migration linter that catches this in CI.