ADR-010: AWS Secrets Manager for staging + production secrets

Date: 2026-06-19 Status: Accepted — implemented. Staging + prod ECS tasks read all sensitive config from AWS Secrets Manager via the task-definition secrets block; no secrets remain in the environment block. Issue: #830 (P0-6 — AWS Secrets Manager migration). Related: #701 (ECS migration), #2330 (no-overwrite safety). Authors: documented as-built (the migration shipped incrementally with the Terraform modules/secrets + modules/iam work).

Context

Originally all deployed secrets (DATABASE_URL, JWT_SECRET, MASTER_ENCRYPTION_KEY, OPENAI_API_KEY, Twilio, Resend, Stripe, …) lived in plain env vars on the ECS task definition environment block. That fails a security audit and leaks every secret to anyone who can read the task definition (aws ecs describe-task-definition). MVP P0 §6 mandated migration to AWS Secrets Manager before the first paying client.

This ADR records the as-built design (which differs from the original issue proposal in namespace and rotation posture) so it isn't re-litigated.

Decision

1. Secrets live in AWS Secrets Manager, referenced by ARN

Each deployed env (staging, prod) has its secrets under a per-env, per-app namespace:

humanwork-<env>/app/<NAME>      e.g. humanwork-prod/app/JWT_SECRET
humanwork-<env>/rds/credentials (the RDS-Proxy master credential JSON)

Namespace note: the original #830 proposal used humanwork/prod/*. The as-built scheme is humanwork-<env>/app/* (env in the prefix segment, /app/ sub-namespace), matching var.name_prefix (humanwork-<env>) used everywhere else in the stack. Functionally equivalent; this is the canonical form.

The ECS task definition references them in the secrets block (never environment):

"secrets": [
  { "name": "JWT_SECRET",            "valueFrom": "arn:aws:secretsmanager:eu-central-1:<acct>:secret:humanwork-prod/app/JWT_SECRET-XXXXXX" },
  { "name": "MASTER_ENCRYPTION_KEY", "valueFrom": "arn:aws:secretsmanager:eu-central-1:<acct>:secret:humanwork-prod/app/MASTER_ENCRYPTION_KEY-XXXXXX" }
  // …52 secrets total per service
],
"environment": [
  { "name": "APP_ENV",   "value": "production" },   // non-sensitive config only
  { "name": "DEMO_MODE", "value": "false" },
  { "name": "HWORK_DOMAIN", "value": "h.work" }
  // URLs, bucket names, pool/timeout knobs — nothing sensitive
]

ECS injects the resolved values into the container at start; they never appear in the task definition or describe-task-definition output.

2. Two Terraform-managed secret classes (`modules/secrets`)

The module (infra/terraform/modules/secrets/main.tf) splits secrets into two for_each maps with deliberately different lifecycles:

generated_secrets — Terraform generates a strong value once (random_id(...).hex), seeds it at creation, then stops managing the value (lifecycle { ignore_changes = [secret_string] }). Used for values the platform owns and can self-generate (e.g. JWT_SECRET, AGENT_SERVICE_SECRET).
external_secrets — Terraform creates the secret container + a REPLACE_ME placeholder value, then stops managing the value (ignore_changes). An operator sets the real value out-of-band (CLI / console). Used for third-party credentials Terraform must never see (e.g. OPENAI_API_KEY, STRIPE_SECRET_KEY, TWILIO_AUTH_TOKEN).

Consequence — REPLACE_ME is expected on a freshly-applied env until an operator populates each external secret. A secret still reading REPLACE_ME in a deployed env is an unfinished provisioning step, not a Terraform bug. See the rotation/population runbook: docs/runbooks/secret-rotation.md.

No-overwrite guarantee (#2330): because both classes carry ignore_changes = [secret_string], a later terraform apply never reverts a value an operator set in the console/CLI back to the generated seed or REPLACE_ME.

3. Least-privilege IAM

The task execution role is granted secretsmanager:GetSecretValue scoped to this env's secret ARNs only (var.execution_secret_arns, fed from the secrets module outputs) — not * (infra/terraform/modules/iam/main.tf, execution_secrets policy). A task in one env/service cannot read another's secrets. kms:Decrypt is * (the secrets use the AWS-managed Secrets Manager key; scoping further would require a CMK).

The terraform-plan role also needs GetSecretValue on this env's secrets (the AWS provider calls it while refreshing aws_secretsmanager_secret_version during plan); it is scoped the same way. AWS-managed ReadOnlyAccess deliberately excludes GetSecretValue.

Status of the original acceptance criteria

Criterion	Status
staging + prod ECS tasks boot from Secrets Manager	✅ both task defs use the `secrets` block (52 secrets each)
0 secret leakage in `environment` block	✅ `environment` holds only non-sensitive config
IAM per-task least-privilege	✅ `GetSecretValue` scoped to the env's ARNs
ADR committed	✅ this document
`.env.example` documents the pattern	✅ references Secrets-Manager-in-staging/prod
Automatic rotation (JWT 90d, MASTER annual)	❌ deferred — see Rotation below
Rotation runbook tested	⏳ runbook authored (secret-rotation.md); manual procedure

Rotation (deferred to a follow-up — not safe to automate as-is)

Automatic rotation of JWT_SECRET and MASTER_ENCRYPTION_KEY is not enabled because both have application-side prerequisites; naive rotation causes an outage or data loss:

MASTER_ENCRYPTION_KEY (AES-256-GCM for integration_credentials): rotating it makes existing ciphertext undecryptable unless the app supports key-versioning / envelope encryption + a re-encryption pass. Annual, manual, with a re-encryption plan.
JWT_SECRET: rotating invalidates every live session unless the verifier accepts multiple signing keys (a key set / kid). Today it is a single secret, so rotation logs everyone out.

Until those prerequisites exist, rotation is manual (documented in the runbook). Auto-rotation is tracked separately so it isn't enabled prematurely.

Consequences

Adding a new deployed secret = add it to generated_secrets or external_secrets in the env's Terraform, apply, then (for external) set the real value out-of-band; reference it in the task def secrets block. Never add a secret to environment.
Operators must populate every external_secrets value after a fresh apply (watch for REPLACE_ME).
Rotation is manual today; do not enable Secrets Manager auto-rotation on JWT_SECRET/MASTER_ENCRYPTION_KEY without the app-side prerequisites.

Context​

Decision​

1. Secrets live in AWS Secrets Manager, referenced by ARN​

2. Two Terraform-managed secret classes (modules/secrets)​

3. Least-privilege IAM​

Status of the original acceptance criteria​

Rotation (deferred to a follow-up — not safe to automate as-is)​

Consequences​