Skip to main content

Lago AWS Migration Runbook

Issue: #1287
Owner: DevOps / Infra Lead (Emanuele Graziano — @grazianognoll)
Workflow: .github/workflows/lago-aws-migrate.yml
Status: Active — migrate each env (dev → staging → prod)

Pre-conditions​

All of the following must be satisfied before starting:

#CheckNotes
1Blocker #702 closedKaito integration prerequisite — confirmed closed 2026-06-19
2AWS Lago instance provisioned per-environmentSee Step 1
3LAGO_API_KEY, LAGO_WEBHOOK_SECRET, LAGO_SPECIALIST_PLAN_CODE, LAGO_STRIPE_PAYMENT_PROVIDER_CODE, STRIPE_SECRET_KEY populated in GitHub environment secretsRequired for bootstrap
4RAILWAY_TOKEN with project write access and RAILWAY_PROJECT_ID stored in GitHub environment secretsRequired only for teardown (dry_run=false)
5Current Railway Lago URL documented for rollback referenceSee each env's LAGO_API_URL in Railway dashboard

Step-by-Step Migration​

1. Provision AWS Lago Instance​

Deploy self-hosted Lago on AWS (ECS/EKS or EC2 + Docker Compose) per environment:

# Each environment gets its own Lago instance:
#   dev     → e.g. https://lago-dev.internal.humanity.org
#   staging → e.g. https://lago-staging.internal.humanity.org
#   prod    → e.g. https://lago.internal.humanity.org
  • Follow the Lago self-hosting guide.
  • Ensure the Lago instance is reachable from the API service (VPC peering / security groups as needed).
  • Note the base URL — you'll need it as the lago_api_url workflow input.

2. Create Lago Organisations and Run Bootstrap​

Each environment's Lago instance needs its HP organisation created during AWS Lago provisioning.

The bootstrap script (api/scripts/lago/bootstrap.ts) ensures the Specialist Monthly plan exists and, when STRIPE_SECRET_KEY is set, registers or updates the Stripe payment provider. It does not create Lago organisations or add-ons. It is idempotent — safe to re-run.

Run manually to verify connectivity before triggering the workflow:

LAGO_API_URL=https://lago-dev.internal.humanity.org \
LAGO_API_KEY=<key> \
STRIPE_SECRET_KEY=<stripe_secret_key> \
LAGO_STRIPE_PAYMENT_PROVIDER_CODE=<stripe_provider_code> \
BILLING_PROVIDER=lago \
npx ts-node api/scripts/lago/bootstrap.ts

Repeat for staging and prod with their respective URLs/keys.

3. Run Workflow — dry_run=true (validate)​

Trigger .github/workflows/lago-aws-migrate.yml via GitHub Actions UI or CLI for each environment:

gh workflow run lago-aws-migrate.yml \
  --ref dev \
  -f environment=dev \
  -f lago_api_url=https://lago-dev.internal.humanity.org \
  -f dry_run=true

What dry_run=true does:

  • Checks out the repo and installs dependencies.
  • Runs bootstrap.ts against the AWS Lago URL.
  • Validates API connectivity (HTTP 200 from /api/v1/organizations).
  • Does NOT decommission Railway Lago — the teardown step is skipped.

Wait for the workflow to pass before proceeding to Step 4.

4. Validate​

Confirm after dry_run=true succeeds:

  • Workflow run shows green in GitHub Actions.
  • Lago API health check returned HTTP 200.
  • HP organisation exists in the AWS Lago admin UI.
  • Specialist Monthly plan and Stripe payment provider configuration match what was on Railway Lago.
  • Spot-check billing events in the Lago event stream (send a test subscription via API).

5. Cut Traffic — Update LAGO_API_URL​

Update the LAGO_API_URL environment variable in each runtime environment (Railway / ECS / k8s secret) to point to the AWS Lago URL:

# Before (Railway):
LAGO_API_URL=https://lago-dev.up.railway.app

# After (AWS):
LAGO_API_URL=https://lago-dev.internal.humanity.org

Redeploy the API service after updating the env var. Monitor error rates and billing event ingestion for 15–30 minutes.

6. Run Workflow — dry_run=false (decommission Railway Lago)​

Once traffic is confirmed healthy on AWS Lago, tear down Railway Lago:

gh workflow run lago-aws-migrate.yml \
  --ref dev \
  -f environment=dev \
  -f lago_api_url=https://lago-dev.internal.humanity.org \
  -f dry_run=false

Repeat for staging and prod.

What dry_run=false adds:

  • The Decommission Railway Lago deployments step runs.
  • Calls railway service delete --project <RAILWAY_PROJECT_ID> --environment <env> --service lago-<env> to remove the Railway Lago service.

7. Repeat for staging and prod​

Follow steps 3–6 for staging, then prod. Recommended cadence:

DayAction
D+0dev dry_run=true → validate → cut → decommission
D+1staging dry_run=true → validate → cut → decommission
D+3prod dry_run=true → validate → cut → decommission

Rollback​

Rollback is a single env-var change — no DB migration required.

If issues are detected after cutting traffic to AWS Lago:

  1. Revert LAGO_API_URL in the environment back to the Railway Lago URL.
  2. Redeploy the API service.
  3. Traffic immediately routes to Railway Lago again.
  4. Investigate the AWS Lago issue before retrying.
# Example: revert dev
LAGO_API_URL=https://lago-dev.up.railway.app   # previous Railway URL
# → redeploy API

The workflow also documents this: see the # Rollback comment in lago-aws-migrate.yml and the job summary step.

Acceptance Checklist (from Issue #1287)​

  • AWS Lago instance provisioned for dev, staging, and prod
  • api/scripts/lago/bootstrap.ts runs successfully against all three AWS Lago instances
  • GHA workflow .github/workflows/lago-aws-migrate.yml exists with workflow_dispatch + dry_run boolean input
  • dry_run=true workflow passes for dev, staging, prod
  • LAGO_API_URL env var updated in all three runtime envs to AWS URLs
  • API service redeployed and billing events verified on AWS Lago
  • dry_run=false workflow passes — Railway Lago services decommissioned
  • api/src/common/env.ts comment updated (Railway annotation removed)
  • This runbook reviewed and signed off by Emanuele Graziano
Last updated on by Degen