ADR-013: Channel Inbound Dedup Keys

2026-06-05 update: Email inbound is now Gmail API polling (PR #1722, commit d8333558). CF Worker fallback during 2-week soak (#1441). Dedup keys for email remain Message-ID either way.

Date: 2026-05-28 Status: Accepted Issue: #937 (this ADR); implementation #928 / #929 (DB layer); pre-existing InboundDedupService (P0-3) Authors: Tom Tang, claude Related: ADR-018 (multi-pod context), ADR-019 (idempotency tier)

Context

Channel webhooks (Slack / WhatsApp / Telegram / Email) deliver the same logical event multiple times under normal operation:

• LB retries on transient 5xx
• Vendor-side retry policies (Twilio retries WhatsApp delivery up to 8 times over 24h on non-200; Slack retries event_callback on timeout)
• Client-side double-clicks (the user clicks "Send" twice in the email client / Slack thread)
• DLQ replay (per P0-8, channel_failures rows are re-processed by the retry worker)

Without idempotency, each retry could produce a duplicate conversation row, a duplicate message row, and a duplicate downstream queue item — visible to the customer as the same message appearing 2-8 times in their thread.

Under active-active multi-pod deployment (ADR-018), the same retry can land on a different api pod on each attempt. Per-pod in-process dedup state (a Map) is incoherent because pod A's Map and pod B's Map are independent — for this reason all dedup state must live in shared infrastructure (Redis + Postgres).

Decision

Two-layer idempotency, ordered by speed:

1. Layer 1 — Redis fast-path: InboundDedupService.checkDuplicate(channel, key) issues SET NX EX <ttl> against dedup:<channel>:<key>. First webhook wins (returns "OK", caller proceeds); replays see existing key, return false, caller short-circuits with 200.
2. Layer 2 — Postgres durable backstop: partial unique index on messages(channel_message_id) WHERE IS NOT NULL + conversations(customer_id, channel) WHERE status='pending' AND channel<>'email'. If a webhook bypasses Redis (TTL expired, Redis briefly unavailable, vendor sent a fresh event_id for the same logical event), the INSERT fails with Postgres 23505. Caller catches + refetches the winning row.

Channel-specific dedup key choice (vendor uniqueness guarantee):

Channel	Key field	Why	Where in vendor payload
Slack	event_id (fallback event_ts)	Per-workspace unique by Slack design. event_ts is a microsecond timestamp + channel ID composite, also workspace-unique.	event.event_id / event.ts
WhatsApp (Twilio)	MessageSid	Account-unique by Twilio design (SM... or MM... prefix).	body.MessageSid
Telegram	update_id	Bot-unique sequence number. Per-bot uniqueness is guaranteed by Telegram.	body.update_id
Email (Cloudflare relay)	Message-ID header	RFC 5322 globally-unique. Cloudflare relay preserves the upstream header.	parsedMail.messageId

Why not a generic timestamp + sender hash:

• Vendor-provided IDs are guaranteed unique by the vendor's own infrastructure (no collision risk we control)
• Timestamps collide under burst (10 emails from the same sender in the same second → same hash → false positive dedup → real second message dropped)
• The vendor ID is in the payload anyway — using it costs nothing and avoids cryptographic hashing

Options considered

1. Single Redis layer only (no DB constraints) — Rejected: Redis outage or TTL expiry admits duplicates. Cross-pod consistency depends on Redis being up; if Redis is briefly unavailable for 5s, every retry in that window leaks through.

2. Single DB constraint only (no Redis) — Rejected: every webhook is processed through to the SQL INSERT before dedup decision, paying DB round-trip cost on every retry storm. Slack alone can fire 1000 events/s during a backfill.

3. Generic hash key (e.g. sha256(senderId || timestamp || content)) — Rejected per "Why not a generic timestamp" above.

4. Per-channel idempotency-key header (Idempotency-Key: ...) — Considered for future client-initiated POSTs (not webhooks). Vendor webhooks don't accept custom headers, so this is orthogonal — it's a planned addition for the platform's own POST endpoints, not a replacement for vendor-key dedup.

5. Slack event_id only (no event_ts fallback) — Rejected: a small fraction of Slack event_callback payloads omit event_id. The fallback to event_ts (which Slack always sends) covers the gap; both fields are workspace-unique, so the fallback is correct, just less efficient (one extra Redis key for the rare omission).

Consequences

Positive:

• Active-active correct: no in-process Map; both layers are pod-independent
• Two-layer defense: Redis catches 99% in ~1ms; DB catches the rest in ~10ms
• Vendor-aligned: we use the same key the vendor uses to track their own message identity
• Cheap to debug: the dedup key is in the original payload, visible in audit logs

Negative:

• 24h Redis TTL costs ~1KB per webhook × peak QPS — small but non-zero
• DB partial unique indices add write cost (~5μs per INSERT)
• The two layers can disagree if Redis goes down briefly: Redis says "first time" (because it just rebooted), DB says "duplicate" (because the previous attempt landed). Caller code handles this — DB winner is treated as authoritative.

Operational:

• Dedup TTL of 24h matches the longest vendor retry window (Twilio WhatsApp = 24h). Replays after 24h are admitted as "new messages" — this is intentional; a 24h+ replay is operationally indistinguishable from a fresh send.

Validation

• InboundDedupService.spec.ts — Redis layer unit tests
• conversations-dedup-constraint.spec.ts (#929) — DB layer behavior + catch-and-refetch
• Manual: Slack Events API "Resend" button on a known event → second delivery returns 200 quickly, no duplicate row created.

Reference implementation files

• api/src/channels/inbound-dedup.service.ts — Redis layer
• api/migrations/1751300000001-AddDedupUniqueConstraints.ts — DB layer
• api/src/conversations/conversations.service.ts:386+ — findOrCreateByChannel catch + refetch

Decision authority

Decision recorded under the issue-first workflow rule established 2026-05-27. Future channel additions (e.g. WeChat, Teams) MUST extend the dedup-key table above and update InboundDedupService before shipping the inbound handler.