Skip to main content

ADR-028: Retire agent-runtime per-org Railway provisioner — ECS-only

Status: Accepted (date 2026-06-03; impl PR #1584 merged) Authors: paulhumanity

Implementation shipped via PR #1584 (chore(agent-runtime): decommission RailwayPerOrgProvisioner — ECS-only), which closed #1450. railway_per_org has been removed from the AgentRuntimeKind union; supported kinds are now shared_railway + ecs_per_org.

Context​

The agent runtime is the service boundary that lets the platform route an Org's agent traffic to a concrete running agent service. The current API surface has three provisioner kinds:

  • shared_railway — the default shared agent service path.
  • ecs_per_org — the intended per-Org isolation path on ECS Fargate.
  • railway_per_org — the legacy per-Org Railway GraphQL provisioner.

Per-Org Railway was originally built for #362 Phase B and shipped in PR #589. It gave the API a default-off adapter that could create one Railway service per Org, inject runtime config, deploy an agent image, persist the resulting agent_runtimes row, and tear the service down later.

That was the right bridge while Railway was the backend substrate. It is no longer the target architecture. PR #1101 introduced ECS Fargate Terraform and GitHub Actions for staging and production. The shared agent still exists as a shared runtime path, but per-Org isolation should now land on ECS instead of creating more Railway services.

The prior cleanup framing in #1408 tried to bundle deletion of this provisioner with the RAGflow-to-Haystack script cleanup and cited ADR-009. That was the wrong scope. ADR-009 is about RAGflow per-tenant Railway services, not the agent runtime. The agent-runtime provisioner has live imports, database state implications, environment implications, and rollback implications, so it needs its own ADR before #1450 removes code.

This ADR uses slot 028 because ADR-021 (throttler) and ADR-027 (TypeORM migrations only) are already taken. Channel credential isolation (#1471) will land in a future slot, TBD.

Decision​

Drop railway_per_org from the AgentRuntimeKind union and from AGENT_RUNTIME_PROVISIONER_KINDS.

After decommission, supported agent-runtime provisioners are:

  • shared_railway — default for shared infrastructure and immediate fallback.
  • ecs_per_org — per-Org isolation when a dedicated runtime is required.

railway_per_org must not be accepted by config validation, created by the factory, written by migration scripts, or kept as a TypeScript union member after the implementation PR.

Staleness check​

PR #1260 added hasRailwayPerOrgTargetMismatch to reject stale Railway runtime bindings whose stored project/environment no longer matches the configured target. Removing railway_per_org deletes that exact safety logic, so the replacement decision is explicit:

Option A: port the check to ECS. The ECS analogue would compare persisted runtime identity against the configured ECS cluster, service/task family, subnet/security-group set, or task definition family before reusing a row.

Option B: drop the cross-cutting check and let the ECS provisioner own reconcile. ECS per-Org runtime identity is not a project/environment pair; it is a set of AWS resources whose desired state should be reconciled by the provisioner and health service. A stale ECS task or service should be detected by DescribeServices/DescribeTasks, task-definition comparison, and normal health transitions inside EcsPerOrgProvisioner, not by Railway-specific lifecycle branching in AgentLifecycleService.

Decision: choose Option B. Do not add an ECS analogue of hasRailwayPerOrgTargetMismatch during the Railway deletion. The implementation PR should remove the Railway helper and its lifecycle call sites. When ecs_per_org becomes fully implemented, its reconcile path should reject or replace stale AWS resources inside the ECS provisioner itself.

Migration plan​

The implementation PR for #1450 MUST include, in its description:

  • SELECT COUNT(*) FROM agent_runtimes WHERE runtime_kind = 'railway_per_org' against dev and prod, with results pasted verbatim.
  • Env audit verifying no deployment sets AGENT_RUNTIME_PROVISIONER=railway_per_org (Railway services, ECS task definitions, GitHub Actions secrets).
  • Migration script applied, or rationale for "no rows to migrate".

The code PR will not merge until the audit data is present in its description. This ADR captures the decision; the audit captures the evidence the decision is safe to apply.

Database​

The implementation PR for #1450 must run this query against dev and prod:

SELECT COUNT(*) FROM agent_runtimes WHERE runtime_kind = 'railway_per_org';

If rows exist, the implementation PR for #1450 must add a one-shot migration script at api/scripts/migrate-agent-runtimes-off-railway-per-org.ts. The script should support a dry run and two explicit actions:

  • Convert rows to ecs_per_org only for Orgs that require dedicated isolation and have an ECS target ready.
  • Delete or tear down stale railway_per_org rows for Orgs that should fall back to shared_railway.

The code PR cannot narrow AgentRuntimeKind until the audit data is present in its description and no live row still carries runtime_kind = 'railway_per_org'.

Environment​

Any deployment with AGENT_RUNTIME_PROVISIONER=railway_per_org must flip before the code deletion merges. Valid target values are shared_railway for the shared runtime path or ecs_per_org where per-Org isolation is deliberately enabled.

The implementation PR for #1450 must include an env audit verifying no deployment sets AGENT_RUNTIME_PROVISIONER=railway_per_org. The env audit must cover Railway service variables, ECS task definitions, and GitHub Actions secrets or variables that can set AGENT_RUNTIME_PROVISIONER.

Code​

Implementation is tracked by #1450. The code PR must wait for this ADR because it deletes a live provisioner, removes database enum-like values from TypeScript, and changes the supported runtime provisioning contract.

Consequences​

Positive consequences:

  • Approximately -1,500 LoC from deleting the provisioner, its tests, and the Railway runtime safety helper.
  • Simpler NestJS dependency-injection graph and provisioner factory.
  • One less provisioning path to maintain, observe, and test.
  • Lower Railway spend by preventing future per-Org Railway service creation.

Negative consequences:

  • ECS-only per-Org isolation means there is no per-Org Railway escape hatch if ECS provisioning regresses.
  • Mitigation: shared_railway remains available as the immediate shared-runtime fallback while ECS per-Org provisioning is fixed.

Rollback​

If ECS provisioning breaks in production, set AGENT_RUNTIME_PROVISIONER=shared_railway on the affected deployment as the immediate mitigation.

If a per-Org isolation guarantee is required during the outage, restore api/src/agent-runtime/provisioners/railway-per-org.provisioner.ts, api/src/agent-runtime/railway-runtime-safety.ts, the factory/module wiring, and the matching tests from git history. The implementation PR for #1450 must cite the deletion commit in its rollback notes; this ADR cannot cite that future commit before the code deletion exists.

Open questions​

  • Exact DB action for any audited railway_per_org rows: convert to ecs_per_org or delete and rely on shared_railway.
  • Exact env flips required by the audit across Railway services, ECS task definitions, and GitHub Actions.
  • The deletion commit hash for rollback documentation in the #1450 implementation PR.
  • The detailed ECS per-Org reconcile contract when EcsPerOrgProvisioner is promoted from placeholder to full implementation.

References​

Last updated on by Degen