Feature: DLQ Admin Surface

Reference: the SuperAdmin Dead Letter Queue surface for outbound channel dispatch failures — what lands there, what the UI shows, and the retry / drain / inspect operations. Last updated: 2026-06-29 Audience: SuperAdmin, Ops, Engineering, QA Surface: /ops/dlq (Ops — SuperAdmin only) Related: DLQ Handling runbook, BullMQ Queue Backlog runbook, Notifications, API reference — Admin · DLQ

---

TL;DR

When a canonical Hermes reply or genuine Expert-authored follow-up can't be delivered to a client's channel (Slack / WhatsApp / Telegram / Email), the dispatch isn't silently lost — it is persisted to the channel_failures Dead Letter Queue (DLQ). A background worker retries it on a schedule; anything it can't recover surfaces at /ops/dlq for a human to inspect, replay, or drop.

The DLQ is the safety net behind the ChannelDispatcherService three-layer outbound path: withCircuit → withRetry → channelFailures.push. The first two layers handle transient blips; the DLQ catches what survives them so a message never just disappears.

---

When items hit the DLQ

A row is written to channel_failures (status queued) whenever ChannelDispatcherService exhausts its in-process protection and the .catch(channelFailures.push) fires. Typical causes:

Cause	Example error	Notes
Vendor outage / 5xx	circuit breaker open, ETIMEDOUT	The circuit breaker trips after 5 consecutive failures; subsequent sends fast-fail straight to the DLQ until it resets.
Auth / credential failure	Slack invalid_auth, token_revoked	Slack permanent errors (channel_not_found, not_in_channel, token_revoked, …) are promoted straight to dead — they can never be redelivered (slack-errors.util.ts, #3059).
Rate limiting	429, too many requests	Retry after the provider cooldown clears.
Email delivery failure	Resend / SMTP / delivery errors	Check Resend status + the org's sending domain.
Timeout	timed out reaching the channel provider	Often transient; the retry worker usually clears these.

Not every failed send reaches the DLQ. A reply to a disconnected client Slack workspace is marked failed but deliberately not enqueued — re-driving it would loop forever (channel-dispatcher.service.ts, #2403).

Scope / privacy. DLQ payloads contain raw outbound content — customer phone numbers, email addresses, message bodies. The surface is SuperAdmin only (JwtAuthGuard + PlatformRolesGuard('superadmin')). Experts and AMs must never see other clients' raw outbound payloads (dlq-admin.controller.ts). Any new DLQ list endpoint must keep this scope.

---

Status lifecycle

channel_failures.status is one of four values:

DB value	UI label	Meaning	Acted on by
queued	Queued	Newly captured; eligible for the retry worker.	Retry worker, manual replay
retrying	Retrying	Picked up / in a retry cycle.	Retry worker
dead	Failed	Exhausted retries, or a permanent error. Needs a human.	Manual replay / drop
drained	Drained	Resolved — either retried successfully or manually cleared.	Terminal (replayable)

   push()                 retry worker / manual replay
     │                              │
     ▼                              ▼
 ┌────────┐   worker picks up   ┌──────────┐  success   ┌─────────┐
 │ queued │ ──────────────────▶ │ retrying │ ─────────▶ │ drained │
 └────────┘                     └────┬─────┘            └─────────┘
     ▲                               │ retry_count ≥ MAX      ▲
     │ replay (reset to queued)      ▼                        │ drain
     │                          ┌─────────┐                   │
     └───────────────────────── │  dead   │ ──────────────────┘
                                └─────────┘

• queued / retrying together are the pending count — what the nav badge, the alert banner, and /health care about.
• dead is shown as "Failed" in the UI (statusLabel() maps it).
• Replay is only valid from dead or drained (REQUEUEABLE_STATUSES) — it resets the row to queued with retry_count = 0 so the worker re-attempts it.

---

The automatic retry worker

Before a human ever looks at the DLQ, ChannelFailuresRetryService runs every 5 minutes, driven by a BullMQ repeatable job registered by SchedulerService (job channel-failures-retry, cron */5 * * * *). The scheduler upserts the repeat with a stable jobId (scheduler-channel-failures-retry); BullMQ's jobId global uniqueness is what guarantees exactly one pod runs each tick under active-active (not leader election). When REDIS_URL is unset (CI / single-instance dev) the scheduler is a no-op and the worker never fires. Each tick:

1. Selects queued rows where retry_count < DLQ_MAX_RETRY and created_at is older than DLQ_MIN_AGE_MS (a cooldown so the worker stays out of the way of the in-process withRetry + circuit breaker), oldest first, up to DLQ_BATCH_SIZE per tick.
2. Rebuilds the original dispatch from the conversation's latest Expert + inbound messages and re-invokes ChannelDispatcherService.dispatchExpertReply.
3. Updates status by outcome:
   • success → drained
   • throw → retry_count++; once it reaches DLQ_MAX_RETRY → dead (and a Sentry exception is captured on the final attempt only).
   • circuit open → counts as an attempt but doesn't call the vendor SDK; the breaker controls the real call rate independently of the cron cadence.
   • permanent Slack error → promoted straight to dead (no Sentry, no further retries).
   • conversation deleted / no Expert message → treated as drained (nothing left to redeliver).

So most transient failures clear themselves within a few ticks. /ops/dlq exists for what doesn't.

Tuning knobs

Env var	Default	Effect
DLQ_MAX_RETRY	5	Attempts before a row goes dead.
DLQ_MIN_AGE_MS	60000	Cooldown before the worker first picks up a queued row.
DLQ_BATCH_SIZE	50	Rows examined per 5-minute tick.
DLQ_ALERT_THRESHOLD	100	Pending count that fires a Slack ops alert (30-min cooldown).
CHANNEL_FAILURES_RETRY_CRON	*/5 * * * *	Override the worker cadence.

---

What the UI shows

The page (/ops/dlq) has three parts.

1. Overview cards

Computed from the newest 500 pending rows:

• Total pending — queued + retrying (the same number the nav badge and alert banner show).
• By channel — pending breakdown across Slack / Telegram / WhatsApp / Email.
• By error — top error labels among pending rows.
• Oldest age — age of the oldest pending row (a staleness signal).

When pending exceeds 500 the cards note "Showing newest 500" — the breakdowns are a sample, not an exact total. Trust /admin/dlq/stats for the authoritative pending count.

2. Filters

• Status — All / Queued / Retrying / Failed / Drained (defaults to Queued).
• Channel — All, or any channel present in the data.
• Org ID — free-text, debounced; scopes to one client.
• Rows — 50 / 100 / 250 / 500 page size.
• Refresh — re-pulls stats + entries. The view also re-ticks relative timestamps every 15s.

3. Entries table

Columns: select checkbox, Created (relative, hover for absolute; shows status inline when the status filter is "All"), Channel badge, Org (monospace, truncated, hover for full ID), Error (truncated label + a How to fix hint icon — see below), Retries, Payload preview (truncated JSON), and per-row Actions.

Clicking a row opens the Payload dialog: id, created, channel, status, the contextual "How to fix" guidance (with an optional reference link, e.g. the Resend status page), and the full pretty-printed JSON payload.

---

Operations

All three operations are available per-row (table actions) and in bulk (toolbar, driven by the row selection — shift-click selects a range). Mutations are optimistic with rollback on error and a success/error toast.

Operation	Endpoint	What it does	Eligibility
Inspect	GET /admin/dlq (row click)	Opens the full payload + resolution guidance. Read-only.	Any row
Replay	POST /admin/dlq/replay { ids }	Resets rows to queued, retry_count = 0; the retry worker picks them up on its next tick (≤ 5 min).	Only dead / drained rows. The bulk button shows an (eligible/selected) count and only replays the eligible subset.
Drop	POST /admin/dlq/drain { ids }	Marks rows drained without redelivery. Use when the message is no longer relevant or is a poison payload. Confirmation required; not reversible (though a drained row can later be replayed).	Any row

Replay vs Drop. Replay only makes sense once the underlying cause is fixed — otherwise the row just fails again and walks back to dead. The per-row How to fix hint tells you what to check first (vendor status, Slack token, sending domain, rate-limit cooldown). Drop when you've decided the message should not be re-sent.

Resolution guidance ("How to fix")

Every row carries a contextual hint derived from its error + channel (#2726), so the surface tells admins how to act, not just that something failed. Matches include circuit-breaker trips, Slack auth, email/Resend delivery (links to resend.com/status), rate limits, and timeouts, with a generic "inspect payload + channel logs, then Replay" fallback.

---

Alerting & observability

• Nav badge + alert banner. DLQ mutations and retry-worker outcomes emit a scoped status event that updates the SuperAdmin badge and banner. Opening or explicitly refreshing /ops/dlq performs one bounded authoritative stats read; idle time performs no interval read.
• Slack ops alert. When pending crosses DLQ_ALERT_THRESHOLD (default 100) the retry worker fires a Slack alert, gated by a 30-minute Redis cooldown so it doesn't spam every 5-minute tick (#2726).
• Prometheus metrics. channel_failures_pending{status} (current gauge by status) and channel_failures_total{channel,outcome} (terminal outcomes). Use these for dashboards/alerts rather than scraping the table.

When the DLQ has rows and you need to triage them, follow the DLQ Handling runbook.

---

Key source files

Concern	File
Admin endpoints	api/src/channels/dlq-admin.controller.ts
DLQ persistence + queries	api/src/channels/channel-failures.service.ts
Retry worker	api/src/channels/channel-failures-retry.service.ts
Entity / status enum	api/src/common/entities.ts (ChannelFailure)
Metrics	api/src/channels/channels.metrics.ts
Ops UI	frontend/src/app/ops/dlq/page.tsx
Alert banner	frontend/src/components/ops/DlqAlertBanner.tsx