Runbook — Auto-sent audit loop: demo preconditions & pre-flight

The Loop A audit answers Terence's "how do you know the agent isn't quietly getting things wrong?": every delivered auto-sent reply is sampled, an independent LLM judge grades it, an Expert can confirm, and we report the rates.

Code path (all merged): D1 samples every delivered agent reply into the reservoir keyed by the Hermes {sessionId, messageId} (conversations.service.ts, after dispatchExpertReply) → AUTO_SENT_AUDIT job reads the prior closed day's samples, reads each turn from SessionDB by locator, LLM-judges it, writes an audit_verdict → silent_error_rate reports judgeFlagged (LLM suspect+wrong) and confirmedError (Expert-confirmed wrong) separately.

Preconditions for verdicts to appear (dev)

#	Precondition	Blocking?	Who
1	D1 deployed + REDIS_URL set (reservoir is Redis-backed)	yes	ops (done)
2	OPENROUTER_API_KEY set (LlmService key)	yes — without it the judge's generateObject returns null and every draft soft-fails to skip → zero verdicts	ops
3	A closed UTC day that already had D1 running + real agent traffic	yes	time/data — D1 samples from deploy-time forward, so prior days are empty; the earliest cohort is the first full day after deploy
4	Real auto-sent (agent) replies in that day	yes	data — Expert-authored replies are not sampled by design
5	LLM_PURPOSE_DAILY_CAP_USD_AUDIT_LOOP	NO — optional	ops (recommended)

Important correction: the per-purpose cost cap (#5058) does not gate the judge. checkPurposeBudget returns null when the cap is unset, so assertBudgetAvailable skips the purpose branch and the judge runs (bounded only by the org's daily budget). Set the cap only to limit spend, not to enable the loop. The real LLM prerequisite is OPENROUTER_API_KEY (#2).

Pre-flight (run this before locking the demo)

1. Confirm OPENROUTER_API_KEY is set on the target env (else no verdicts).
2. Ensure the target env saw some real agent auto-replies today (drive a few if needed).
3. After the day closes (UTC), run the judge on that cohort — as SuperAdmin:

   POST /analytics/audit/auto-sent/run        { "day": "YYYY-MM-DD" }   # specific closed day
   # or the generic scheduler trigger (judges "yesterday"):
   POST /scheduler/jobs/auto-sent-audit/trigger

   The response is { cohorts, judged, skipped }. judged > 0 means it worked; judged = 0, skipped > 0 usually means the LLM key is missing (soft-fail).
4. Read the results and confirm the demo actually lands:
   • GET /analytics/silent-error-rate → non-null judgeFlagged.rate (and confirmedError once an Expert reviews).
   • GET /analytics/audit/spot-check → at least one genuinely suspect/wrong reply with a defensible rationale.
   • Have an Expert submit a verdict on it (POST /analytics/audit/spot-check/verdict) to exercise the judge-flag → human-confirm loop — the credible half of the story.
5. If no compelling flagged example exists in real data, pick another cohort or use a clearly-disclosed seeded case. This is the #1 demo risk and it is data, not code.

Notes

• The on-demand run is idempotent (audit_verdict ON CONFLICT DO NOTHING) — safe to re-run while rehearsing.
• At pilot volume, judging the whole cohort is fine; reservoir sampling (Algorithm-R + daily K) remains the production scaling mechanism.