Runbook: Secret rotation & population (staging / prod)

Secrets live in AWS Secrets Manager and are injected into ECS tasks via the task-definition secrets block. See ADR-010 for the design. This runbook covers manual population and rotation — there is no automatic rotation today (see the rotation caveats below).

• Region: eu-central-1
• Namespace: humanwork-<env>/app/<NAME> (e.g. humanwork-prod/app/JWT_SECRET)
• Envs: staging, prod

Values are seeded by Terraform once and then left alone (ignore_changes = [secret_string]), so the CLI/console is the source of truth for the live value and a later terraform apply will not overwrite it.

A. Populate a REPLACE_ME external secret (initial setup)

External secrets (third-party credentials) are created with a REPLACE_ME placeholder. Find any that still need a real value:

R=eu-central-1
ENV=prod   # or: staging
# No --max-results: the CLI auto-paginates the whole namespace. There are >100
# secrets across api/agent/frontend, so a 100 cap would silently miss the tail
# and report a false "all clear".
aws secretsmanager list-secrets --region $R \
  --query "SecretList[?starts_with(Name,'humanwork-${ENV}/')].Name" --output text \
  | tr '\t' '\n' | sort | while read -r n; do
    v=$(aws secretsmanager get-secret-value --region $R --secret-id "$n" --query SecretString --output text 2>/dev/null)
    [ "$v" = "REPLACE_ME" ] && echo "NEEDS VALUE: $n"
  done

Set the real value:

aws secretsmanager put-secret-value --region eu-central-1 \
  --secret-id humanwork-prod/app/OPENAI_API_KEY \
  --secret-string 'sk-...'

Then roll the service so tasks pick up the new value (secrets are read at container start — a running task keeps its old value):

aws ecs update-service --region eu-central-1 \
  --cluster humanwork-prod-cluster --service humanwork-prod-api \
  --force-new-deployment

B. Rotate an existing secret (safe — most secrets)

For credentials with no app-side coupling (API keys, webhook secrets, channel tokens): rotate at the provider, update the secret, roll the service.

# 1. (provider) issue the new credential / rotate at source
# 2. update the value
aws secretsmanager put-secret-value --region eu-central-1 \
  --secret-id humanwork-prod/app/RESEND_API_KEY --secret-string '<new>'
# 3. roll each service that consumes it (api / agent / frontend)
aws ecs update-service --region eu-central-1 \
  --cluster humanwork-prod-cluster --service humanwork-prod-api --force-new-deployment
# 4. verify the new tasks are healthy, then revoke the old credential at the provider
aws ecs describe-services --region eu-central-1 \
  --cluster humanwork-prod-cluster --services humanwork-prod-api \
  --query 'services[0].deployments[].{status:status,running:runningCount,rollout:rolloutState}'

ECS rolls the service (new tasks healthy before old drain), so this is zero-downtime as long as the new value is valid.

C. Rotate JWT_SECRET — ⚠️ logs everyone out

JWT_SECRET signs/verifies all auth tokens. It is a single key today (no multi-key / kid support), so rotating it invalidates every live session — all users must log in again.

• Only rotate on suspected compromise. Not a routine operation until multi-key verification ships (see follow-up issue referenced in ADR-010).
• Procedure: put-secret-value → roll all services (api especially) → expect a wave of 401s; users re-authenticate. Communicate the forced logout.
• Do not enable Secrets Manager auto-rotation on this secret.

D. Rotate MASTER_ENCRYPTION_KEY — ⚠️ data-loss risk, do NOT rotate naively

MASTER_ENCRYPTION_KEY is the AES-256-GCM key for integration_credentials (and any other at-rest encrypted columns). Rotating the value without re-encrypting existing data makes that data permanently undecryptable.

• Do not put-secret-value a new key on its own. There is no key-versioning / envelope-encryption support yet.
• A real rotation requires: (1) app support for decrypting with the old key while encrypting with the new, (2) a backfill that re-encrypts every row, (3) only then retiring the old key. Until that exists, treat this key as non-rotatable in place.
• If rotation is forced (compromise): coordinate an engineering change first; this is an incident, not a runbook step.

Verify a task is reading the expected secret wiring

TD=$(aws ecs describe-services --region eu-central-1 --cluster humanwork-prod-cluster \
  --services humanwork-prod-api --query 'services[0].taskDefinition' --output text)
# Names + which ARN each resolves from (values are NOT shown by ECS — by design):
aws ecs describe-task-definition --region eu-central-1 --task-definition "$TD" \
  --query 'taskDefinition.containerDefinitions[0].secrets[].{name:name,from:valueFrom}' --output table

Notes

• Never put a secret in the task-def environment block — it leaks via describe-task-definition. New secrets go through modules/secrets + the secrets block (see ADR-010).
• Rolling a service is required for any value change; in-flight tasks keep the value they booted with.
• After populating/rotating, re-run the REPLACE_ME audit (section A) to confirm nothing is left as a placeholder.