Tavus persona provisioning automation + drift audit — design

• Date: 2026-07-27
• Issue: #4575 — no CI/CD automation for provision-tavus-personas.ts; tool/prompt drift on live personas goes undetected
• Status: approved (design review 2026-07-27)

Context — what changed since the issue was filed

Issue #4575 (2026-07-17) predates the A2 refactor b6435e6b4 (2026-07-22), which demoted api/scripts/provision-tavus-personas.ts to first-time PAL create + Face (replica) assignment only. Ongoing prompt/tools sync onto existing personas moved to TavusAdapterService.sync() (runs on every config publish, per-(org, specialist) PAL, content-hash gated, marker-upserts VIDEO_CALL_TOOLS_PROMPT and binds the 5 video-call tools via reconcilePalTools).

Remaining gaps this design closes:

1. First-time provisioning of new catalog rows is still manual — the exact #4574 failure mode ("11 new specialists, someone must remember to shell in").
2. No drift detection for:
   • catalog personas (specialists.tavus_persona_id) — nothing self-heals these; the 372-persona incident in the issue lived here;
   • OSA PALs (org_specialist_assignments.tavus_persona_id) that never receive another config publish.
3. The script's --dry-run today only lists unprovisioned rows — it cannot detect drift on existing personas.

Explicitly out of scope: the one-time backfill of the 372 stale prod personas (the first audit run produces the real number; remediation is decided after this lands), gitflow-deploy.yml integration (decision: manual dispatch only), dev/Railway environments.

Decisions (design review, 2026-07-27)

Question	Decision
Scope	Provision workflow + nightly drift audit
372 backfill	Audit-first; decide remediation from its output
Provision trigger	Manual workflow_dispatch only (external API touching live fleet stays human-gated)
Alerting	Slack via existing secrets.SLACK_WEBHOOK; audit run also fails red
Environments	staging + prod (ECS); dev stays manual via railway run
Audit coverage	Catalog personas (full expected set) + OSA PALs (light check)
Drift definition	Tool-name set + VIDEO_CALL_TOOLS_PROMPT_MARKER presence. NOT deep schema equality — personas legitimately carry manual console edits and the adapter deliberately preserves them
Unprovisioned rows	Audit also reports catalog rows WHERE tavus_persona_id IS NULL
Result plumbing	Audit script prints machine-readable summary; GHA reads it from CloudWatch and posts Slack (approach A)

Approaches rejected: (B) container posts to Slack itself — webhook URL leaks into DescribeTasks output or needs a new secrets-store entry, and Slack formatting gets baked into an API-image script; (C) single shared "tavus-ops" task-def with command overrides — breaks the established one-family-per-script convention (seed-catalog, backfill-thumbnails) and merges log groups.

Components

1. New script — api/scripts/audit-tavus-personas.ts

Read-only everywhere: no DB writes, no Tavus writes (GET-only; reuses the GET-retry etiquette of the existing scripts / TavusClient policy: 3× retry with backoff, writes never retried — n/a here).

Inputs (env): TAVUS_API_KEY, DATABASE_URL. No GITHUB_PAT needed (no manifest access).

Checks, in order:

1. Catalog fleet. For every specialists row WHERE is_catalog = TRUE AND tavus_persona_id IS NOT NULL: GET the persona; drifted when
   • any expected tool name is missing from layers.llm.tools[].function.name, where the expected set = union of active tavus_function_definitions rows (latest active version per name, same query as buildPersonaTools) + getVideoCallToolDefinitions() names; or
   • system_prompt does not contain VIDEO_CALL_TOOLS_PROMPT_MARKER. A 404 persona (catalog↔Tavus divergence) counts as drifted with reason missing_persona.
2. OSA PALs (light). For every org_specialist_assignments row with a non-null tavus_persona_id: same check but the expected tool set is only getVideoCallToolDefinitions() names (that is all reconcilePalTools guarantees; the DB library tools are catalog-persona-only).
3. Unprovisioned. SELECT count(*), array_agg(slug) FROM specialists WHERE is_catalog = TRUE AND tavus_persona_id IS NULL.

Output: per-finding human-readable log lines, then a single final line

AUDIT_SUMMARY {"env":"prod","checked_catalog":N,"checked_osa":N,"catalog_drifted":N,"osa_drifted":N,"unprovisioned":N,"drifted_slugs":[...],"unprovisioned_slugs":[...]}

Slug arrays are truncated to 25 entries each ("...and N more" handled on the Slack side from the counts) to keep the line CloudWatch- and Slack-safe.

Exit codes: 0 clean, 2 findings (drift and/or unprovisioned), 1 operational error (env missing, DB/Tavus unreachable, pre-flight exception). The workflow maps 2 → Slack + red, 1 → red without Slack drift report.

2. Terraform — infra/terraform/modules/stack/

Following the backfill_thumbnails block verbatim (same API image, local.api_environment, local.api_secrets — which carry TAVUS_API_KEY and DATABASE_URL; GITHUB_PAT did NOT pre-exist and is newly registered in modules/secrets external_secrets by this work, so the operator must inject the real value out-of-band before the first non-dry provision run — terraform only creates a REPLACE_ME placeholder):

• aws_ecs_task_definition.provision_tavus — family ${var.name_prefix}-provision-tavus, command ["node", "dist/scripts/provision-tavus-personas.js"], own log group /ecs/${var.name_prefix}-provision-tavus.
• aws_ecs_task_definition.audit_tavus — family ${var.name_prefix}-audit-tavus, command ["node", "dist/scripts/audit-tavus-personas.js"], own log group /ecs/${var.name_prefix}-audit-tavus.
• Outputs: provision_tavus_task_definition_family, audit_tavus_task_definition_family (mirrors seed_catalog_task_definition_family).
• modules/iam app_deploy policy: one added statement — logs:GetLogEvents, logs:FilterLogEvents scoped to the two new log-group ARNs (:* stream suffix). The role today has no logs:* at all; scope stays minimal.

Both scripts must be emitted into dist/scripts/ by the API build (verify provision-tavus-personas.ts compiles into the image the same way seed-specialist-catalog.ts and backfill-specialist-tavus-thumbnails.ts do; the audit script follows the identical pattern).

Note: provision-tavus-personas.ts writes CSVs to api/scripts/output/ — ephemeral in Fargate. Acceptable: the same data is in the task log; the CSVs are a local-operator convenience. No change.

3. Workflow — .github/workflows/provision-tavus-personas.yml

workflow_dispatch only. Inputs:

• environment: choice staging | prod, default staging
• dry_run: boolean, default true
• limit: string, optional (empty = no limit)

Structure mirrors seed-specialists-catalog.yml exactly: concurrency group provision-tavus-${{ inputs.environment }} (no cancel-in-progress), permissions: id-token: write / contents: read, environment: for OIDC trust, resolve vars.AWS_APP_DEPLOY_ROLE_ARN_{PROD,STAGING}, reuse the API service's network configuration, aws ecs run-task, poll describe-tasks until STOPPED (not the capped tasks-stopped waiter — aws-cli#1295), fail on nonzero container exit code, surface failures[] when no task is placed.

dry_run/limit are passed via --overrides containerOverrides.command (append --dry-run / --limit N to the base command) — zero script changes; the flags already exist.

Timeout: 30 min (372-persona pre-flight is ~400 sequential GETs; seed's 15 min is too tight).

4. Workflow — .github/workflows/audit-tavus-personas.yml

Triggers:

• schedule: nightly 0 5 * * * (05:00 UTC — before the 06:00 specialists-catalog-sync, so the audit sees yesterday's steady state) → prod
• workflow_dispatch: environment choice staging | prod for on-demand runs

Same OIDC/run-task/poll skeleton as §3 against the audit-tavus family. Then:

• exit 0 → green, no notification.
• exit 2 → fetch the AUDIT_SUMMARY line via aws logs filter-log-events on the audit log group (filter pattern AUDIT_SUMMARY, newest event), parse with jq, post Slack Block Kit message to secrets.SLACK_WEBHOOK: env, drifted counts per fleet, unprovisioned count, first slugs, link to the run and to the provision workflow ("run this to fix unprovisioned rows"). Then exit 1 so the run shows red in Actions.
• exit 1 (or missing summary line) → red run; Slack message with "audit failed to run — check /ecs/humanwork-{env}-audit-tavus logs" (no drift details).
• SLACK_WEBHOOK unset → ::warning:: and skip the post (pattern from prod-patrol), still red.

Data flow

Error handling

Failure	Behavior
Tavus GET fails after retries (audit)	exit 1 — operational error, not drift; red run, "failed to run" Slack
Persona 404 (audit)	counted as drift, reason missing_persona
DB unreachable	exit 1
run-task places no task	surface failures[], fail (existing pattern)
Provision pre-flight name assertion fails	script throws → container exit 1 → workflow red (existing behavior, unchanged)
logs:FilterLogEvents denied (terraform not yet applied)	audit workflow degrades to "findings detected, could not read summary" Slack + red

Testing / verification

• Audit script: unit tests colocated per repo convention — fingerprint comparison (missing tool name, missing marker, 404 persona, clean persona), expected-set assembly per fleet (catalog = DB∪code, OSA = code only), summary-line shape, exit-code mapping. Tavus/pg mocked.
• Workflows: actionlint (runs in PR checks); dry-run dispatch of the provision workflow against staging with dry_run=true as the smoke test after terraform apply.
• Terraform: terraform-plan.yml on the PR shows exactly two task-defs, two log groups, one IAM statement, two outputs.
• End-to-end: first manual dispatch of the audit against prod produces the real drift/unprovisioned numbers for the #4575 remediation decision.

Rollout

0. Inject real GITHUB_PAT into humanwork-{staging,prod}/app/GITHUB_PAT in Secrets Manager (terraform seeds REPLACE_ME; provision fails loudly with a GitHub 401 until injected).
1. Merge (dev) → terraform plan/apply for staging + prod stacks (task-defs are inert until a run-task references them).
2. Dispatch audit on staging → verify Slack message shape.
3. Dispatch provision on staging with dry_run=true → verify [DRY] output in /ecs/humanwork-staging-provision-tavus (the workflow smoke test from Testing above).
4. Dispatch audit on prod → real numbers for the 372-persona remediation call (tracked in #4575 / #4574 thread, separate from this design).
5. Nightly schedule takes over.