Skip to main content

Haystack Deploy Runbook

Hosting (2026-06-04): Dev: Railway today (managed via rag/railway.toml). Staging/Prod: AWS ECS Fargate via the humanwork-rag service. Cutover via #1455 (Railway-only) was reverted; #1527 (humanwork-rag ECS service + RAGFlow cleanup in infra/terraform/) + #1541 (rag-deploy.yml workflow) establishes ECS as canonical for staging/prod. Cloud Map service URL: humanwork-rag.humanwork.internal:1416. See ADR-024, ADR-025, ADR-026.

Railway (dev)​

Railway autodeploys from the dev branch on push, per rag/railway.toml. Private networking only — humanwork-rag.railway.internal:1416. No public ingress.

ECS Fargate (staging + prod)​

Staging and prod humanwork-rag run on ECS Fargate in the shared humanwork cluster, registered in Cloud Map at humanwork-rag.humanwork.internal:1416. Terraform source of truth: infra/terraform/modules/stack/main.tf (sets rag_internal_url = "http://humanwork-rag.${local.cloud_map_namespace}:1416").

Deploys are automatic on merge — .github/workflows/rag-deploy.yml triggers on pushes to staging and main that touch rag/** (mirroring the API and Agent pipelines):

  • Push to staging (touching rag/**) → auto-deploy to staging.
  • Push to main (touching rag/**) → auto-deploy to production.
  • dev targets Railway (see below) and is ignored by the ECS pipeline.

Manual override is still available via workflow_dispatch:

  1. GitHub → Actions → Deploy Rag → Run workflow.
  2. Select environment: staging or environment: prod.
  3. Optionally pin an image tag (defaults to sha-<short> from the chosen branch).
  4. Workflow renders the task-def, aws ecs update-service, waits for stability, runs the smoke check below.

Container health check (per #1586): curl -fsS http://localhost:1416/status from the task definition.

Downstream callers (notably humanwork-api) must send X-API-Key: $HAYSTACK_API_KEY (#1588) — not Authorization: Bearer.

Schema bootstrap​

Run:

cd api && npm run migrations:run

This applies the CreateHaystackSchema migration (api/migrations/1780500000001-CreateHaystackSchema.ts).

Per ADR-027, humanwork-api ECS task definitions opt into RUN_MIGRATIONS_ON_BOOT=true on first prod boot. Manual migrations:run is only needed for out-of-band application.

OpenRouter smoke test​

Run this against a known org with seed data:

curl -X POST $HAYHOOKS_URL/retrieval/run \
  -H "X-API-Key: $HAYSTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "...", "org_id": "<seed-org>"}'

Expect HTTP 200 and a non-empty results array.

For staging/prod, HAYHOOKS_URL is http://humanwork-rag.humanwork.internal:1416 (reachable only from inside the VPC — exec into an ECS task or use the bastion).

Rollback procedure​

ECS (staging + prod)​

Roll back to the previous task-def revision:

# Find the previous revision
aws ecs describe-task-definition --task-definition humanwork-rag-<env> --query 'taskDefinition.revision'

# Update the service to the previous revision
aws ecs update-service \
  --cluster humanwork-<env> \
  --service humanwork-rag \
  --task-definition humanwork-rag-<env>:<prev-rev>

Wait for aws ecs wait services-stable. Mirrors the rollback section in incident-response.md.

Railway (dev only)​

Railway keeps prior deployments. Roll back with the Railway UI by selecting the previous deployment and choosing rollback.

To roll back from the CLI, first find the deployment ID to restore:

railway deployment list

Then redeploy that specific deployment:

railway deployment redeploy <previous-deployment-id>
Last updated on by Degen