DLQ Handling Runbook

Operator triage for the channel-dispatch Dead Letter Queue (channel_failures): "the DLQ has rows / the pending banner is up / an alert fired — what now?" Feature reference: features/dlq-admin.md. For queue depth problems on the BullMQ channels-outbound queue (as opposed to dead-lettered sends), use bullmq-queue-backlog.md instead.

Mental model. The DLQ is the last line of defense for outbound replies, behind withCircuit → withRetry. A row here means an Expert-approved message did not reach the client after the in-process layers gave up. A background worker auto-retries queued rows every 5 minutes; this runbook is for the rows it can't clear (dead) or a pending pile-up that's growing faster than it drains. Triage surface is /ops/dlq (SuperAdmin only).

1. Signals

Signal	Where	Meaning
DLQ pending banner	/ops/* (SuperAdmin)	queued + retrying > 0. Dismissible per session.
:rotating_light: DLQ has N pending… Slack alert	Ops Slack channel	Pending crossed DLQ_ALERT_THRESHOLD (default 100). 30-min cooldown.
Nav badge count	Ops nav	Same pending number from GET /admin/dlq/stats.

Metrics (Grafana / Prometheus):

# current pending by status
channel_failures_pending{status="queued"}
channel_failures_pending{status="retrying"}
# terminal outcomes over time — is the worker draining or are rows going dead?
rate(channel_failures_total{outcome="drained"}[15m])
rate(channel_failures_total{outcome="dead"}[15m])
sum by (channel, outcome) (channel_failures_total)

A healthy DLQ trends toward zero pending: rows arrive queued, the worker drains them within a few ticks. Rising dead rate or flat-high pending is the actionable state.

2. First 5 minutes — read, don't act

Open /ops/dlq. Do not start replaying yet — replaying before the root cause is fixed just walks rows back to dead.

1. Scope it. Read the overview cards: total pending, by channel, by error, oldest age. One channel? One error class? One org (filter by Org ID)? A single-channel/single-error spike is a vendor or credential problem; spread-across-everything is more likely Redis/worker/infra.
2. Read the "How to fix" hint on the dominant error (hover the ? in the Error column, or open a row). It tells you what to check first.
3. Confirm the worker is running. If drained rate is zero while queued is high, the retry worker (or its scheduler) may be down — see §4 "worker not draining".

Stats endpoint for an authoritative pending count (the cards sample the newest 500):

curl -s -H "Authorization: Bearer $SA_TOKEN" "$API_BASE/admin/dlq/stats" | jq
# { "stats": { "queued": .., "retrying": .., "dead": .., "drained": .. }, "pending": .. }

3. Common root causes

Root cause	How to confirm	What to do
Vendor outage (Slack/Twilio/Resend down)	Errors cluster on one channel with circuit/timeout/5xx; vendor status page is red.	Wait. The breaker + worker recover automatically once the vendor heals. Replay any dead rows after recovery. Don't drop — these are deliverable.
Auth / credentials	Slack invalid_auth / token_revoked; email domain errors. Usually one org.	Fix the credential (reinstall Slack app for the org / re-verify sending domain), then Replay the affected rows. Note: permanent Slack errors are already dead and will never auto-retry.
Rate limiting	429 / too many requests on one channel.	Let the cooldown clear, then Replay. If chronic, reduce send pressure for that org/channel.
Poison payload	One org/conversation fails repeatedly with the same non-transient error; retry_count climbing to MAX.	Inspect the payload (row click). If it can't be delivered, Drop it. Capture the id + payload sample for follow-up first.
Worker not draining	queued high, drained rate ~0, oldest age climbing.	The retry worker isn't ticking — see §4.
Conversation deleted	Worker logs conv … gone; marking drained.	None — the worker self-heals these to drained.

4. Worker not draining

The retry worker is ChannelFailuresRetryService, driven by a BullMQ repeatable job that SchedulerService registers (job channel-failures-retry, cron */5 * * * *). A stable jobId dedups the schedule across pods, so the tick runs on exactly one pod per cycle — there is no separate leader lock for it. If queued rows aren't moving to drained/dead:

1. Check the scheduler is ticking. In API logs look for DLQ retry tick: / DLQ retry tick done: lines roughly every 5 minutes.

   aws logs tail /ecs/humanwork-api --since 15m | grep -i "DLQ retry tick"
   No tick lines → the BullMQ scheduler queue isn't firing repeatable jobs (treat like bullmq-queue-backlog.md §3 worker triage: check Redis health, API task health, recent deploys). The scheduler queue owns the repeatable job; confirm it has workers and isn't backed up.
2. Check Redis. The BullMQ scheduler queue (including its jobId dedup), the circuit breaker, and the alert cooldown all live in Redis. redis-cli -u "$REDIS_URL" ping must return PONG. With REDIS_URL unset or Redis down, the scheduler is a no-op and the worker never fires.
3. Check the cooldown window. Rows younger than DLQ_MIN_AGE_MS (default 60s) are intentionally skipped — brand-new queued rows won't move on the very first tick. This is expected, not a fault.
4. Stuck behind a still-open breaker. If the vendor is down, retries fast-fail as CircuitOpenError (counted, but no SDK call). Rows stay queued/retrying until the breaker resets. This is correct back-pressure — fix the vendor, don't force it.

5. Manual remediation

All actions are on /ops/dlq (per-row buttons or bulk via selection — shift-click for ranges). Prefer the UI; it scopes and confirms for you.

• Replay (dead / drained only) → resets rows to queued, retry_count = 0; the worker re-attempts within ≤ 5 min. Only after the root cause is fixed. Bulk replay shows an (eligible/selected) count and replays only the eligible subset.
• Drop → marks rows drained without redelivery. For poison payloads or messages no longer worth sending. Confirm dialog required. Preserve the id + payload sample first if it may be needed for RCA.
• Inspect → row click opens the full payload + "How to fix" guidance.

Direct API (scripted bulk, SuperAdmin token):

# replay a set of ids
curl -s -X POST -H "Authorization: Bearer $SA_TOKEN" -H "Content-Type: application/json" \
  -d '{"ids":["<id1>","<id2>"]}' "$API_BASE/admin/dlq/replay" | jq
# drop (drain) a set of ids
curl -s -X POST -H "Authorization: Bearer $SA_TOKEN" -H "Content-Type: application/json" \
  -d '{"ids":["<id1>","<id2>"]}' "$API_BASE/admin/dlq/drain" | jq

Order of operations for a vendor/credential incident: fix the cause → confirm vendor/credential healthy → Replay dead rows → watch pending drain to zero. Replaying first just re-deads everything.

6. Knobs (incident-time tuning)

Change via env / deploy, not at runtime. See features/dlq-admin.md.

Env var	Default	When to touch
DLQ_MAX_RETRY	5	Raise temporarily if a recovering vendor needs more attempts before rows go dead.
DLQ_MIN_AGE_MS	60000	Rarely. Lowering makes the worker pick up fresh rows sooner.
DLQ_BATCH_SIZE	50	Raise to drain a large backlog faster (watch vendor rate limits).
DLQ_ALERT_THRESHOLD	100	Tune alert noise.
CHANNEL_FAILURES_RETRY_CRON	*/5 * * * *	Speed up the drain cadence during an incident.

7. Postmortem follow-ups

Use the incident template in incident-response.md. Capture:

• Peak pending, peak dead rate, channel(s) affected, time to drain to zero.
• Root cause (vendor incident, credential, deploy, poison payload) with a link.
• How many rows were auto-drained vs manually replayed vs dropped.
• Whether the in-process layers (retry/circuit breaker) should have caught it — if so, file a tuning/test follow-up so the row never reaches the DLQ next time.
• If a poison payload caused it, add a regression test on the dispatch path.