Skip to main content

Notification Failures Runbook

Triage for the in-app AppNotification pipeline (#915): users not receiving notifications, the bell/inbox not updating in real time, badge counts not clearing, or notifications missing entirely. Pipeline reference: features/notifications.md.

Mental model. Two surfaces, one DTO. The REST store (app_notifications table + /notifications) is the source of truth and survives reconnect. The Socket.io push (/notifications namespace, notification:new event) is a best-effort latency optimization with no replay. If the REST store has the row but the push didn't arrive, the bug is in the real-time leg; if the store has no row, the bug is upstream in the producer.

1. Signals​

There are no dedicated notification metrics — the observability surface is structured logs from NotificationsGateway and RedisIoAdapter. Watch for:

Log line (source)Meaning
REDIS_URL not set (RedisIoAdapter)Gateway fell back to the in-memory adapter — cross-pod push is broken under active-active
Failed to initialize Redis adapter / pub-sub error (RedisIoAdapter)Redis reachable issue → cross-pod delivery degraded
Auth failed for client … / connected without token (NotificationsGateway)Client JWT rejected at the WS handshake → never joins user:<id> room
token missing user id (NotificationsGateway)Token verified but has no userId/sub → disconnected

2. Decide which leg is broken​

Ask the affected user (or check the DB) whether the notification exists in history:

-- Does the row exist at all? (source of truth)
SELECT id, event_type, read_at, created_at
FROM app_notifications
WHERE recipient_user_id = '<userId>'
ORDER BY created_at DESC
LIMIT 20;
  • Row is present, but it didn't pop in real time → real-time leg. Go to §3.
  • Row is absent → the producer never called NotificationsService.create(). Go to §4.
  • Row is present and read_at is set but the badge still shows it → frontend state / dedup. Go to §5.

3. Real-time push not arriving​

The push only reaches a socket that (a) authenticated and (b) joined user:<recipientUserId>, and the emit only crosses pods through Redis.

  1. Redis adapter. Confirm REDIS_URL is set on every API pod. A REDIS_URL not set warning means a pod is on the in-memory adapter: an emit on pod A never reaches the user's socket on pod B. This is a deploy/config bug — restore REDIS_URL (root CLAUDE.md mandates it in prod, #1354). Verify Redis pub/sub connectivity from a pod.
  2. Handshake / room join. Look for Auth failed for client … or token missing user id around the user's connect attempts. A rejected handshake means the socket never joined user:<id> and will receive nothing. Cause is usually an expired/short JWT or a clock-skewed JWT_SECRET mismatch — have the user re-auth; confirm JWT_SECRET parity across pods.
  3. Expected, not a bug: no replay. If the user was disconnected when the notification fired, the event is gone — there is no backfill on reconnect. The frontend reconciles by calling GET /notifications on mount/reconnect; confirm that fetch is happening (Network tab). If the row is in GET /notifications but the bell didn't update on reconnect, it's a frontend reconcile bug (§5), not a delivery loss.

4. Notification never created (producer side)​

The row is absent, so the producer didn't persist it. Producers create best-effort and swallow per-recipient errors so one failure doesn't block the batch — a silent miss is possible.

Known producers (extend this table as new ones land):

event_typeProducerSource
billing.payment_failedBillingPaymentFailedNotifierapi/src/billing/infrastructure/billing-payment-failed.notifier.ts
workspace_provisioning_failedWorkspaceProvisioningProcessorapi/src/workspace-lifecycle/workspace-provisioning.processor.ts
  1. Confirm the upstream event actually fired (e.g. the Lago payment_failed webhook was received; the BullMQ provisioning job actually exhausted retries).
  2. Confirm the producer resolved a recipient userId — if the org has no assigned AM / no billing-responsible member, there is no one to notify and create() is never called. That's a data gap (assign an AM), not a pipeline failure.
  3. Check producer logs for caught exceptions around the notifications.create() call.

5. Badge / inbox state wrong​

  • Badge won't clear after read. PATCH /notifications/:id/read is idempotent and only flips read_at once. Confirm the PATCH returns 200 (not 403 — that means the caller isn't the owner; not 404 — wrong id). The unread count is derived client-side from readAt === null, so a stale list in NotificationContext is the usual cause — a refetch (GET /notifications) should reconcile.
  • Duplicate rows in the bell. The frontend dedupes incoming notification:new against the REST list by id. True DB duplicates (two rows, two ids) mean a producer fired twice — the module does not dedup by (recipient, type); idempotency is the producer's responsibility (e.g. a re-run BullMQ job). Decide at the producer whether the duplicate is acceptable (often it is — it's just a heads-up).

6. Escalation​

Last updated on by Degen