Skip to main content

ADR-021: Throttler Rate-Limit Strategy

Status as of 2026-06-06: shadow-mode observation window from 2026-05-26 has elapsed; verify enforce-flip in current config.

Date: 2026-05-26 Status: Accepted (Decision 6 from MVP P0 NFR plan) Issue: #803 (shadow mode); #805 (webhook decorators); #806 (Sentry alert); #807 (429/503 differentiation) Authors: tylerhumanity (Tom Tang)

Context​

#531 removed ThrottlerModule from app.module.ts (citing per-Specialist billing replacement). However, the change left five @Throttle decorators in the codebase (auth.controller.ts:49/374/385 + kb.controller.ts:81 + conversations.controller.ts:33). These decorators are metadata-only — without a registered ThrottlerModule, they have no effect. The codebase silently dropped its only application-layer rate limit.

Under active-active multi-pod deployment (ADR-018), the absence of rate limiting creates several risk surfaces:

  • DDoS on channel webhooks: Slack retry storms (#507 incident) hammer the API; no rate limit means runaway cost + queue depth.
  • Brute-force on auth endpoints: /auth/login, password reset — uncapped.
  • Excessive client API usage: Single buggy client floods /conversations write endpoints; impacts all tenants.

P0-9k (#743) re-registered ThrottlerModule with Redis storage for cross-pod consistency. This ADR documents the six locked sub-decisions that flesh out the strategy.

Decision​

Re-enable global throttling with cross-pod Redis storage, but apply it cautiously via six sub-decisions:

6.0 — Storage library: @nest-lab/throttler-storage-redis​

  • Community-maintained, actively updated, compatible with @nestjs/throttler v6.
  • Alternatives rejected: nestjs-throttler-storage-redis (older, less maintained); custom ioredis wrapper (NIH; reinventing rate-limit storage).

6.1 — Strategy: per-org default with per-IP fallback for unauthenticated routes​

  • OrgThrottlerGuard extracts orgId from JWT (user.orgMemberships[0].orgId || user.orgId).
  • Falls back to req.ip for unauthenticated routes (channel webhooks, /auth/login).
  • Per-org tracking is strictly better than per-IP for our workload — a single org's bursty traffic should not affect other orgs even if they share an IP (corporate NAT).
  • Locked plan said "per-IP default" but the implementation went directly to per-org (validated as an improvement).

6.2 — Existing five @Throttle decorators: kept at historical numeric limits​

  • auth.controller.ts:49 → 10 req/min
  • auth.controller.ts:374 → 5 req/min
  • auth.controller.ts:385 → 10 req/min
  • kb.controller.ts:81 → 30 req/min
  • conversations.controller.ts:33 → 100 req/min (per PRD §14)
  • Limits set pre-#531 by the original authors. No evidence the numbers are wrong. Adjust only after we observe Sentry shadow-mode "would-have-blocked" data.

6.3 — Missing decorators: add to 4 channel webhooks​

Per-channel limits sized to expected vendor retry behavior:

  • @Post("slack/events") → 1000 req/min/IP
  • @Post("email/inbound") → 100 req/min/IP (Cloudflare batches)
  • @Post("whatsapp/inbound") → 1000 req/min/IP
  • @Post("telegram/...") → 500 req/min/IP

Out of scope for MVP: admin endpoints, file uploads, billing webhook. Adding throttle to these without empirical data risks misconfigured limits. Revisit in P1.

6.4 — Fail mode: Fail-soft + Sentry alert​

  • When Redis storage is unavailable at module init: ThrottlerStorageRedisService constructor returns; module loads with in-memory fallback storage (per-pod). System continues to function — rate limiting is per-pod instead of cross-pod, which is degraded but not broken.
  • When Redis disconnects at runtime: Sentry custom alert (throttler.storage.disconnected) fires within 30s. Ops investigates.
  • Alternative rejected: hard fail-closed (return 503 for all requests during Redis outage). Service availability outweighs the brief reduction to per-pod throttling.

6.5 — Rollout: Shadow mode 1 week → enforce​

  • Initial deploy: THROTTLER_SHADOW_MODE=true in dev / staging / prod.
  • OrgThrottlerGuard.throwThrottlingException() logs [SHADOW] would-have-blocked instead of throwing.
  • Operator observes Sentry logs for one week, validates no false-positive blocks on legitimate traffic patterns (Indian UAT scripted runs, integration tests, customer bursts).
  • Prod flips THROTTLER_SHADOW_MODE=false (kill switch via env var, no restart needed).

This is the safest deploy strategy for a behavior change that affects every API endpoint.

6.6 — Boundary with backpressure (P0-9l): distinct HTTP signaling​

Limit typeSemanticHTTP statusHeadersBody
Rate limit (9k)"Client too fast"429 Too Many RequestsX-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset{ "code": "rate_limit", ... }
Backpressure (9l)"Server overloaded"503 Service UnavailableRetry-After{ "code": "backpressure", ... }

Client retry logic must distinguish: 429 → exponential backoff per X-RateLimit-Reset; 503 → fixed retry per Retry-After. Documented in docs/api/CLIENT_RETRY_STRATEGY.md (planned, P1-7 follow-up).

Rejected alternatives​

Roll our own throttler storage in ioredis​

Rejected. ~150 LoC reimplementation of a well-tested community library. Maintenance burden outweighs the benefit of one fewer dep.

Per-IP only (no per-org)​

Rejected for production. Multiple orgs sharing a NAT (corporate / VPN exit IP) would compete for one bucket. Per-org isolation is fundamental to multi-tenant fairness.

Hard fail-closed when Redis storage unavailable​

Rejected. Outage in throttler storage should not become an outage in the API. Per-pod fallback storage is degraded but functional. ADR-019's "best-effort" tier explicitly covers this.

Immediate enforce (no shadow mode)​

Rejected. Five @Throttle decorators were noop for an unknown period (between #531 and #743). Deploy day flipping them live without observation = SEV-2 risk. The shadow mode adds one week of observability for zero ongoing cost.

One throttle for all endpoints​

Rejected. Different endpoints have different traffic profiles. /auth/login should be ~10/min (brute-force prevention). /conversations should be ~100/min (normal user activity). Channel webhooks should be ~1000/min (vendor retry storms). A global limit cannot serve all three.

Consequences​

Positive​

  • Cross-pod consistency. Same client hitting different pods sees one global limit, not 2× when LB balances.
  • DDoS protection on webhooks. Slack/Twilio retry storms (real incident #507) capped at sustainable rate.
  • Per-org fairness. Buggy tenant cannot exhaust shared resources.
  • Observability. Sentry alerts on Redis storage outages + shadow-mode "would-have-blocked" data inform tuning.

Negative​

  • Operational complexity. Two new env vars (THROTTLER_SHADOW_MODE, implicit REDIS_URL). Operator must remember to flip shadow off after observation period.
  • Latency. Each request adds ~1ms Redis round-trip. Acceptable.
  • One-time risk window. When prod enforces (flip SHADOW_MODE=false), legitimate bursts that exceed limits will get 429. Mitigation: observe shadow-mode logs first.

Operational checklist (post-merge)​

  • Verify THROTTLER_SHADOW_MODE=true set in prod ECS task definition.
  • Sentry filter set up for [SHADOW] would-have-blocked log breadcrumbs.
  • One-week observation period scheduled.
  • Flip cron / ticket created to switch SHADOW_MODE=false after observation.

Review​

Revisit this ADR when:

  • Real prod traffic produces unexpected 429s (need to retune limits).
  • We add a new endpoint that should be throttled.
  • A new throttling library / storage adapter becomes available with significant advantages.

Cross-references:

Last updated on by Degen