Skip to main content

Feature: DLQ Admin Surface

Reference: the SuperAdmin Dead Letter Queue surface for outbound channel dispatch failures โ€” what lands there, what the UI shows, and the retry / drain / inspect operations. Last updated: 2026-06-29 Audience: SuperAdmin, Ops, Engineering, QA Surface: /ops/dlq (Ops โ€” SuperAdmin only) Related: DLQ Handling runbook, BullMQ Queue Backlog runbook, Notifications, API reference โ€” Admin ยท DLQ

TL;DRโ€‹

When an Expert-approved reply can't be delivered to a client's channel (Slack / WhatsApp / Telegram / Email), the dispatch isn't silently lost โ€” it is persisted to the channel_failures Dead Letter Queue (DLQ). A background worker retries it on a schedule; anything it can't recover surfaces at /ops/dlq for a human to inspect, replay, or drop.

The DLQ is the safety net behind the mandatory 3-layer outbound pattern: withCircuit โ†’ withRetry โ†’ channelFailures.push. The first two layers handle transient blips in-process; the DLQ catches what survives them so a message never just disappears (Slack/WhatsApp pre-#550 had only a DLQ stub and black-holed replies).

When items hit the DLQโ€‹

A row is written to channel_failures (status queued) whenever ChannelDispatcherService exhausts its in-process protection and the .catch(channelFailures.push) fires. Typical causes:

CauseExample errorNotes
Vendor outage / 5xxcircuit breaker open, ETIMEDOUTThe circuit breaker trips after 5 consecutive failures; subsequent sends fast-fail straight to the DLQ until it resets.
Auth / credential failureSlack invalid_auth, token_revokedSlack permanent errors (channel_not_found, not_in_channel, token_revoked, โ€ฆ) are promoted straight to dead โ€” they can never be redelivered (slack-errors.util.ts, #3059).
Rate limiting429, too many requestsRetry after the provider cooldown clears.
Email delivery failureResend / SMTP / delivery errorsCheck Resend status + the org's sending domain.
Timeouttimed out reaching the channel providerOften transient; the retry worker usually clears these.

Not every failed send reaches the DLQ. A reply to a disconnected client Slack workspace is marked failed but deliberately not enqueued โ€” re-driving it would loop forever (channel-dispatcher.service.ts, #2403).

Scope / privacy. DLQ payloads contain raw outbound content โ€” customer phone numbers, email addresses, message bodies. The surface is SuperAdmin only (JwtAuthGuard + PlatformRolesGuard('superadmin')). Experts and AMs must never see other clients' raw outbound payloads (dlq-admin.controller.ts). Any new DLQ list endpoint must keep this scope.

Status lifecycleโ€‹

channel_failures.status is one of four values:

DB valueUI labelMeaningActed on by
queuedQueuedNewly captured; eligible for the retry worker.Retry worker, manual replay
retryingRetryingPicked up / in a retry cycle.Retry worker
deadFailedExhausted retries, or a permanent error. Needs a human.Manual replay / drop
drainedDrainedResolved โ€” either retried successfully or manually cleared.Terminal (replayable)
   push()                 retry worker / manual replay
     โ”‚                              โ”‚
     โ–ผ                              โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”   worker picks up   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”  success   โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚ queued โ”‚ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ถ โ”‚ retrying โ”‚ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ถ โ”‚ drained โ”‚
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜                     โ””โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜            โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
     โ–ฒ                               โ”‚ retry_count โ‰ฅ MAX      โ–ฒ
     โ”‚ replay (reset to queued)      โ–ผ                        โ”‚ drain
     โ”‚                          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”                   โ”‚
     โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ โ”‚  dead   โ”‚ โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                                โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
  • queued / retrying together are the pending count โ€” what the nav badge, the alert banner, and /health care about.
  • dead is shown as "Failed" in the UI (statusLabel() maps it).
  • Replay is only valid from dead or drained (REQUEUEABLE_STATUSES) โ€” it resets the row to queued with retry_count = 0 so the worker re-attempts it.

The automatic retry workerโ€‹

Before a human ever looks at the DLQ, ChannelFailuresRetryService runs every 5 minutes, driven by a BullMQ repeatable job registered by SchedulerService (job channel-failures-retry, cron */5 * * * *). The scheduler upserts the repeat with a stable jobId (scheduler-channel-failures-retry); BullMQ's jobId global uniqueness is what guarantees exactly one pod runs each tick under active-active (not leader election). When REDIS_URL is unset (CI / single-instance dev) the scheduler is a no-op and the worker never fires. Each tick:

  1. Selects queued rows where retry_count < DLQ_MAX_RETRY and created_at is older than DLQ_MIN_AGE_MS (a cooldown so the worker stays out of the way of the in-process withRetry + circuit breaker), oldest first, up to DLQ_BATCH_SIZE per tick.
  2. Rebuilds the original dispatch from the conversation's latest Expert + inbound messages and re-invokes ChannelDispatcherService.dispatchExpertReply.
  3. Updates status by outcome:
    • success โ†’ drained
    • throw โ†’ retry_count++; once it reaches DLQ_MAX_RETRY โ†’ dead (and a Sentry exception is captured on the final attempt only).
    • circuit open โ†’ counts as an attempt but doesn't call the vendor SDK; the breaker controls the real call rate independently of the cron cadence.
    • permanent Slack error โ†’ promoted straight to dead (no Sentry, no further retries).
    • conversation deleted / no Expert message โ†’ treated as drained (nothing left to redeliver).

So most transient failures clear themselves within a few ticks. /ops/dlq exists for what doesn't.

Tuning knobsโ€‹

Env varDefaultEffect
DLQ_MAX_RETRY5Attempts before a row goes dead.
DLQ_MIN_AGE_MS60000Cooldown before the worker first picks up a queued row.
DLQ_BATCH_SIZE50Rows examined per 5-minute tick.
DLQ_ALERT_THRESHOLD100Pending count that fires a Slack ops alert (30-min cooldown).
CHANNEL_FAILURES_RETRY_CRON*/5 * * * *Override the worker cadence.

What the UI showsโ€‹

The page (/ops/dlq) has three parts.

1. Overview cardsโ€‹

Computed from the newest 500 pending rows:

  • Total pending โ€” queued + retrying (the same number the nav badge and alert banner show).
  • By channel โ€” pending breakdown across Slack / Telegram / WhatsApp / Email.
  • By error โ€” top error labels among pending rows.
  • Oldest age โ€” age of the oldest pending row (a staleness signal).

When pending exceeds 500 the cards note "Showing newest 500" โ€” the breakdowns are a sample, not an exact total. Trust /admin/dlq/stats for the authoritative pending count.

2. Filtersโ€‹

  • Status โ€” All / Queued / Retrying / Failed / Drained (defaults to Queued).
  • Channel โ€” All, or any channel present in the data.
  • Org ID โ€” free-text, debounced; scopes to one client.
  • Rows โ€” 50 / 100 / 250 / 500 page size.
  • Refresh โ€” re-pulls stats + entries. The view also re-ticks relative timestamps every 15s.

3. Entries tableโ€‹

Columns: select checkbox, Created (relative, hover for absolute; shows status inline when the status filter is "All"), Channel badge, Org (monospace, truncated, hover for full ID), Error (truncated label + a How to fix hint icon โ€” see below), Retries, Payload preview (truncated JSON), and per-row Actions.

Clicking a row opens the Payload dialog: id, created, channel, status, the contextual "How to fix" guidance (with an optional reference link, e.g. the Resend status page), and the full pretty-printed JSON payload.

Operationsโ€‹

All three operations are available per-row (table actions) and in bulk (toolbar, driven by the row selection โ€” shift-click selects a range). Mutations are optimistic with rollback on error and a success/error toast.

OperationEndpointWhat it doesEligibility
InspectGET /admin/dlq (row click)Opens the full payload + resolution guidance. Read-only.Any row
ReplayPOST /admin/dlq/replay { ids }Resets rows to queued, retry_count = 0; the retry worker picks them up on its next tick (โ‰ค 5 min).Only dead / drained rows. The bulk button shows an (eligible/selected) count and only replays the eligible subset.
DropPOST /admin/dlq/drain { ids }Marks rows drained without redelivery. Use when the message is no longer relevant or is a poison payload. Confirmation required; not reversible (though a drained row can later be replayed).Any row

Replay vs Drop. Replay only makes sense once the underlying cause is fixed โ€” otherwise the row just fails again and walks back to dead. The per-row How to fix hint tells you what to check first (vendor status, Slack token, sending domain, rate-limit cooldown). Drop when you've decided the message should not be re-sent.

Resolution guidance ("How to fix")โ€‹

Every row carries a contextual hint derived from its error + channel (#2726), so the surface tells admins how to act, not just that something failed. Matches include circuit-breaker trips, Slack auth, email/Resend delivery (links to resend.com/status), rate limits, and timeouts, with a generic "inspect payload + channel logs, then Replay" fallback.

Alerting & observabilityโ€‹

  • Nav badge + alert banner. DlqAlertBanner polls GET /admin/dlq/stats every 30s (paused on hidden tabs) and shows a dismissible SuperAdmin banner linking to /ops/dlq whenever pending > 0.
  • Slack ops alert. When pending crosses DLQ_ALERT_THRESHOLD (default 100) the retry worker fires a Slack alert, gated by a 30-minute Redis cooldown so it doesn't spam every 5-minute tick (#2726).
  • Prometheus metrics. channel_failures_pending{status} (current gauge by status) and channel_failures_total{channel,outcome} (terminal outcomes). Use these for dashboards/alerts rather than scraping the table.

When the DLQ has rows and you need to triage them, follow the DLQ Handling runbook.

Key source filesโ€‹

ConcernFile
Admin endpointsapi/src/channels/dlq-admin.controller.ts
DLQ persistence + queriesapi/src/channels/channel-failures.service.ts
Retry workerapi/src/channels/channel-failures-retry.service.ts
Entity / status enumapi/src/common/entities.ts (ChannelFailure)
Metricsapi/src/channels/channels.metrics.ts
Ops UIfrontend/src/app/ops/dlq/page.tsx
Alert bannerfrontend/src/components/ops/DlqAlertBanner.tsx
Last updated on by Degen