Runbook — Legacy confidence analytics
Owner: confidence-recalibration · Alert severity: informational
ConfidenceRecalibrationService still computes historical quality statistics for
existing (client × Specialist) assignment data. Its thresholds are compatibility
and analytics metadata only. They do not hold, auto-send, release, escalate, or
otherwise change a managed Hermes reply.
What an alert means
An alert means the historical confidence band has a high measured wrong-rate. It does not mean client delivery was stopped, and changing the stored threshold must not change delivery behavior.
The alert carries the client, Specialist, current compatibility threshold, sampled wrong-rate, and evidence path. Use it to inspect model quality or decide whether the legacy analytics job should be retired. Do not treat it as an Expert queue or reply policy control.
What to do
- Inspect
/ops/clients/{orgId}and confirm the sample is large enough. - Use the evidence to improve the Specialist configuration, model, or evaluation data when warranted.
- Do not withhold, release, or reroute a successful reply because of this signal.
- If the alert is noisy, disable or remove the legacy analytics job rather than adding a delivery gate.
Runtime invariant
Managed delivery follows ADR-046: Humanwork sends only the real message and real attachments through ACP; Hermes owns the session and returns one canonical reply. Only an explicit human request or a genuine managed-turn failure creates Expert work. Confidence values are never an authority boundary.