MVP P0 — Non-Functional Requirements (NFR) Plan (Final)

Author: tylerhumanity (Tom Tang) Scope: MVP-phase NFR — direct dependencies of Paul's 5/26 "production-ready hosting" goal Out of scope: Functional items (4-channel outbound media / HITL UI / Email threading / WhatsApp number pool) / Pre-Sale P0 / P1 / P2 Deployment model: multi-instance active-active Effort scale: XS = < 3 days / S = 1 week / M = 2-3 weeks / L = > 3 weeks Status: Sprint-ready (grounded in code facts; all file:line references verified)

Context

Deployment Model: Multi-Instance Active-Active

What active-active means:

N pods serve traffic simultaneously, LB does round-robin / least-conn
No leader node / no sticky sessions / no single-point state
All shared state lives in PG / Redis / external infrastructure

Why active-active:

Option	Choice	Reason
Active-Active	✅	Best ROI / linear scaling / no leader-switch latency
Active-Passive	❌	Half the resources wasted / in-flight work lost on switchover
Active-Active + Sticky	❌	Partially mitigates in-memory state but blocks scale-out
Leader-Only-Write	❌	Write throughput bottleneck / complex switchover

Active-Active Hard Constraints (10 rules)

#	Constraint	Implemented in
AA1	All in-memory state externalized (Redis / PG)	P0-9 9a/9d
AA2	All cron / scheduled tasks run exactly once globally	P0-9 9b/9c
AA3	All write operations idempotent	P0-9 9d/9e/9f/9g + P0-3
AA4	All async work persisted	P0-9 9e/9f + P0-8
AA5	All writes have race protection (PG unique + atomic `UPDATE...WHERE`)	P0-9 9g/9j
AA6	LB has no sticky session	MVP-P0-2
AA7	Health checks fast + independent of in-flight state	MVP-P0-2 + P0-7
AA8	Graceful shutdown ≥ 2× max in-flight LLM timeout	MVP-P0-9 SIGTERM handler
AA9	DB connection pool across pods does not exceed PG `max_connections`	MVP-P0-1 RDS Proxy
AA10	Secrets / config from a single source of truth	MVP-P0-6 Secrets Manager

Prerequisites

Dependency	Source	Blocks
Schema migration expand-contract pattern	Pre-Sale-P0-5	P0-9 "rolling update zero data loss" acceptance
PG `max_connections` upper bound pinned	MVP-P0-1 RDS Proxy	AA9
BullMQ Redis connection	Existing `persona-cleanup.processor.ts` pattern	All of 9b/9c/9d/9e/9f
Sentry DSN	MVP-P0-4	P0-9 silent-degradation alerts

Bucket Notation

Bucket A = dev only (Railway / docker-compose)
Bucket B = application-layer change (dev + staging + prod synchronized)
Bucket C = staging + prod only (AWS)

Data Consistency Contract

Code fact: The entire codebase has 13 explicit transactions across 8 files; core F1/F2/F3 flows contain only 5 of them; F1/F2 sendMessage performs 17 write steps but only 2 use an inner transaction; zero usages of optimistic locking. MVP does not pursue strict ACID consistency — it uses layered acceptance instead.

Complete location of all 13 transactions

Core flow (5): conversations.service.ts:1146/1202 + expert-queue.service.ts:297/676/758
Peripheral (8): agent-lifecycle.service.ts:83 + organizations.service.ts:189/1291/1733 + onboarding.service.ts:237/454 + slug-reservation.service.ts:68 + whatsapp-pairing.service.ts:138

6-tier consistency classification

Tier	Example	MVP changes?
🟢 Strong consistency	F3 queue+conv inner tx / Snooze resurface inner tx	P0-9 9g/9h/9i/9j expand coverage
🟢 Strong idempotency	DB unique + `INSERT ON CONFLICT`	P0-9 9g
🟡 Eventual (within ~30s)	Socket.io emit / audit log	No change — best-effort by design
🟡 At-least-once	BullMQ retry + dedup	Within P0 scope
🔴 Best-effort	KB delete Haystack→DB / outbound "delivered" ground truth	KB delete unchanged; other in P0-8 / Pre-Sale-P0-4
🔴 Known Lost Update	history_summary / messageId / quota	P0-9 9i fixes history_summary; quota deferred to P1

Four "intentionally accepted" inconsistencies (codified in ADR-019)

Audit log is not in the same transaction as the business write
Socket.io emit does not guarantee exactly-once
KB delete Haystack→DB has no saga
Cross-service writes (Agent ↔ NestJS) have no distributed transaction

30-Dimension Production Readiness Coverage Matrix

#	Dimension	Current	MVP P0 coverage
1	Idempotency	🔴	P0-3 + P0-9 9d/9e/9f/9g
2	Data consistency	🔴	P0-9 9g/9h/9i/9j + ADR-019
3	Concurrency control	🔴	P0-9 9g/9j
4	Transaction boundaries	🟡	P0-9 9h
5	Unique constraints	🟡	P0-9 9g
6	State machine design	🟡	P0-9 9j
7	Message reliability	🔴	P0-9 9e/9f
8	Retry mechanism	🟡	P0-8
9	Timeout mechanism	🟢	N/A
10	Circuit breaker	🔴	P0-9 9a + P0-8
11	Rate limiting	🔴	P0-9 9k
12	Graceful degradation	🟢	P0-2 + P0-4 alert linkage
13	Backpressure	🔴	P0-9 9l
14	High availability	🔴	P0-1
15	Disaster recovery	🔴	Pre-Sale-P0-2
16	Multi-instance statelessness	🔴	P0-9 9a/9b/9c/9d
17	Load balancing	🟡	P0-1
18	Service discovery	🟢	N/A
19	Database HA	🔴	P0-1
20	Cache consistency	🟡	P1-4 (must use in-process LRU + Redis pub/sub invalidation; per-pod stale cache is worse than no cache under active-active)
21	Queue backlog handling	🔴	P0-8 + P0-9 9l
22	Observability	🔴	P0-4 + P0-7
23	Alerting	🔴	P0-4 + P0-7
24	Release rollback	🟡	P0-1 (CodeDeploy circuit-breaker auto-rollback)
25	Configuration management	🔴	P0-6
26	Authz + security	🟡	P0-5 + Pre-Sale-P0-3
27	Multi-tenant isolation	🟢	Baseline OK + P2-1 depth
28	External-dependency resilience	🔴	P0-9 9m
29	Cost + capacity planning	🔴	Pre-Sale-P0-4
30	Operations runbooks	🔴	P0-7 + Pre-Sale-P0-1/2/5

Coverage: 30/30

Deployment Topology

dev branch         →   Railway dev
                       ├─ humanwork-{api,agent,frontend}
                       ├─ Railway PG / Redis
                       └─ docker-compose locally

staging branch     →   AWS staging (mirror prod)
                       ├─ ECS Fargate (multi-task)
                       ├─ RDS Multi-AZ + Proxy
                       ├─ ElastiCache cluster mode
                       ├─ ALB + WAF + CloudFront/Vercel
                       └─ Secrets Manager

main branch        →   AWS production
                       ├─ Identical shape to staging
                       ├─ PagerDuty alerting
                       └─ Reserved Instance

"Staging mirrors prod" hard constraint: Terraform IaC guarantees staging/prod are structurally identical.

Core Flow Sequence Diagrams (NFR Slice)

F1 Inbound Message Flow

 Slack/Email/WhatsApp/Telegram
        │
        ▼ POST /channels/<ch>/<event>
 channels.controller.ts
   1. signature verify (HMAC)              ← 🟢 4 channels covered
   2. audit log write
   3. setImmediate(routeXxx)               ← 🔴 in-process × 8 sites (MVP-P0-3 + 9e)
   4. return 200
          │ (async fire-and-forget)
          ▼
 ConversationsService
   findOrCreateByChannel                   ← 🟡 retries cause duplicates (MVP-P0-3 + 9g dedup)
   sendMessage → AgentClient.chat
          │
          ▼
 AgentClient.chat
   fetch + AbortSignal.timeout(60s)        ← 🟢 timeout configured
   catch → fallback message + flagForReview ← 🟢 5-layer fail-safe
          │
          ▼
 ExpertQueueItem.save + Socket.io emit     ← 🟢 Redis adapter wired

F2 Agent Draft Flow

 ConversationsService.sendMessage
   │
   ▼
 Phase A: AgentClient.forOrg(orgId)        ← 🟡 DB lookup per request, no cache (P1-4 will fix)
   │
   ▼
 KB retrieval (best-effort, try/catch)     ← 🟢 fail-soft
   │
   ▼
 fetch agent /v1/chat (60s timeout)
   ├─ 200 → reply + confidence + cost
   ├─ timeout/5xx → fallback message       ← 🟢
   └─ 429 → fallback message               ← 🟡 no cross-vendor fallback (P2-2 will fix)
   │
   ▼
 DB write: agent_message + queue_item      ← 🟡 multi-statement non-atomic (P0-9 9h fixes expert msg orphan)

F3 Expert Release Flow

 Expert UI → POST /expert-queue/:id/release
   │
   ▼
 ExpertQueueService.releaseAgentMessage
   1. validate + msgRepo.save               ← P0-9 9h: move into inner tx
   2. queueRepo.update(awaiting_client)     ← P0-9 9j: switch to UPDATE...RETURNING
   3. emitAgentMessageSent (Socket.io)      ← 🟢 client UI shows "sent" immediately
   4. setImmediate(dispatchPromise.catch)   ← 🔴 fire-and-forget (MVP-P0-8 + 9f)
   5. return 200
          │
          ▼
 ChannelDispatcherService.dispatchExpertReply
   ├─ email    → Resend  + withRetry(3) ✅  + ❌ CB / DLQ ✅
   ├─ whatsapp → Twilio  ❌ retry / ❌ CB   + DLQ ✅ (P0-8 + 9a)
   ├─ slack    → Slack   ❌ retry / ❌ CB   + DLQ ✅ (P0-8 + 9a)
   └─ telegram → TG bot  ❌ retry / ❌ CB   + DLQ ✅ (P0-8 + 9a)
   
   Note: dispatchTelegram (channel-dispatcher.service.ts:381-442) directly
   does new TelegramBot + bot.sendMessage() with no CB wrapping. All 4
   outbound channels lack CB.

 channel_failures table (#557 persistent DLQ) ✅
        │
        ▼ 🔴 0 retry worker (MVP-P0-8 + 9f)

F4 Realtime Notification Flow

 NestJS Service → NotificationsGateway.emitXxx
   │
   ▼
 Socket.io server
   ├─ RedisIoAdapter ✅ wired (main.ts:30-32)
   │   ├─ REDIS_URL set → Redis pub/sub
   │   └─ REDIS_URL unset → in-memory fallback ⚠️ silent
   ├─ pub/sub.on('error') → logger.error only ← 🟡 no runtime alert (MVP-P0-2 risk)
   └─ emit to(`org:${orgId}` / `experts`)
   │
   ▼
 Client (PortalChat / WorkspaceQueue)
   ├─ Socket.io client (auto reconnect ✅)
   └─ 🔴 8s polling fallback still present (P1-7 will remove)

MVP P0 — Nine Items

Priority order (based on code audit)

#	Item	Rationale
1	MVP-P0-9 Multi-Instance Active-Active Safety Bundle	Unified carrier for 3 hard requirements (active-active / no data loss / idempotency); 13 sub-tasks
2	MVP-P0-3 Channel Inbound Dedup	Largest current reliability gap (3/4 channels bare); a P0-9 sub-task
3	MVP-P0-8 Outbound consistency + retry/CB unification	Demo-killer risk ("UI shows sent but peer never received"); a P0-9 sub-task
4	MVP-P0-4 Sentry + alerting channels	Without Sentry the alerts from the 3 items above are invisible
5	MVP-P0-1/6 ECS + Secrets Manager	Emanuele in-flight
6	MVP-P0-2 Socket.io Redis deployment validation	Adapter already wired; remaining work is deploy + remove polling
7	MVP-P0-7 Incident Readiness	Status page / OnCall / Incident SOP
8	MVP-P0-5 Startup invariants	Multi-invariant extension, XS effort

Effort overview: XS(1) + S(3) + M(4) + L(1) = 8-10 weeks (Bucket B/C items parallelizable)

MVP-P0-1. AWS ECS Active-Active Infrastructure `[Bucket C]` `[L]`

NFR category: Availability + Scalability

Plan (scope of Emanuele's ECS PR):

staging + prod ECS Fargate (Multi-AZ + desired_count >= 2 from day 1)
RDS Multi-AZ + RDS Proxy
- Pool sizing hard rule: pool_size_per_pod × pod_count ≤ Proxy_max × 0.8
- Initial: pool=10 / pod=2 → 20 conn ≤ 80
- TypeORM extra: { max: <pool_size_per_pod> } set explicitly
ElastiCache Redis cluster mode + Multi-AZ
ALB target group: no sticky / round-robin / drain ≥ 120s / deregistration delay ≥ 120s
Health checks: liveness /health shallow / readiness /ready deep (PG+Redis)
ECR + CodeDeploy rolling: minimumHealthyPercent=100 / maximumPercent=200 / deploymentCircuitBreaker enabled
Terraform IaC (staging.tf vs prod.tf diff < environment parameters)
CloudWatch logs + Container Insights

Acceptance: staging+prod identical shape / Multi-AZ failover drill / rolling deploy circuit-breaker test / connection draining verified / Terraform diff = only env params

MVP-P0-2. Socket.io Redis Adapter Multi-Pod Validation `[Bucket B+C]` `[S]`

NFR category: Availability (multi-pod deployment enabler)

Code state: Redis adapter already wired at api/src/main.ts:30-32, with automatic fallback when REDIS_URL is unset.

Remaining work:

PR-A: ~~Install adapter~~ ✅ done
PR-B: staging runs 2 pods × 100 sockets, soak 1 week + monitor pub/sub error rate
PR-C: Frontend PortalChat removes polling fallback (P1-7)
PR-D: prod runs 2 pods (Emanuele ECS PR scope)

Risk: pub/sub.on('error') only logs; needs runtime alert hooked into P0-4 Sentry

Acceptance: staging 2 pod × 100 socket × 30 min, zero message loss / 8s polling fallback removed / Redis reconnect failure alert fires

MVP-P0-3. Channel Inbound Dedup Across 4 Channels `[Bucket B]` `[S]` 【#1 priority】

NFR category: Reliability (input idempotency)

Code state: All 4 channel webhooks are bare. Only Email has internal Message-ID dedup inside routeEmail (channels.controller.ts:1053); Slack/WhatsApp/Telegram have zero dedup. setImmediate × 8 sites appear at channels.controller.ts:229/589/880/914/949/1241/1650/1943.

Plan: lightweight dedup (independent of full BullMQ adoption):

// api/src/channels/inbound-dedup.service.ts
async checkDuplicate(channel, key): Promise<boolean> {
  const result = await redis.set(`dedup:${channel}:${key}`, '1', 'EX', 86400, 'NX');
  return result !== 'OK';
}

Channel-aware dedup key (ADR-013):

Slack = event_id (fallback event_ts)
WhatsApp = MessageSid
Telegram = update_id
Email = Message-ID header

Acceptance: 4-channel dedup unit tests pass / retry storm produces zero duplicates / Slack #507 lands using the same pattern

MVP-P0-4. Application-Layer Sentry + Alert Channels `[Bucket B+C]` `[S]`

NFR category: Observability

Plan:

Sentry across 4 services: API NestJS / Agent Python / Frontend Next.js
Alert channels: dev — log only / staging — Slack #engineering / prod — PagerDuty + Slack #alerts
9 Sentry projects (3 services × 3 environments) with naming SOP
beforeSend hook scrubs PII (no raw body)

Acceptance: 4 services integrated / 3 environments have separate projects / staging error → Slack / prod PagerDuty path verified

MVP-P0-5. Production Startup Invariants Guard `[Bucket B]` `[XS]`

NFR category: Security (production safety)

Code state: The /auth/demo-login endpoint is already IS_PRODUCTION-gated (auth.controller.ts:88-97). This task extends it to a multi-invariant startup guard.

Plan:

if (process.env.APP_ENV === 'production') {
  const invariants = [
    [process.env.DEMO_MODE !== 'true', 'DEMO_MODE=true forbidden'],
    [(process.env.JWT_SECRET?.length ?? 0) >= 64, 'JWT_SECRET >= 64 chars'],
    [!!process.env.SENTRY_DSN, 'SENTRY_DSN required'],
    [!!process.env.MASTER_ENCRYPTION_KEY, 'MASTER_ENCRYPTION_KEY required'],
    [process.env.DATABASE_URL?.includes('proxy'), 'DATABASE_URL must use RDS Proxy'],
  ];
  for (const [ok, msg] of invariants) {
    if (!ok) { console.error(`FATAL: ${msg}`); process.exit(1); }
  }
}

+ CI lint: .env.production* must not contain DEMO_MODE=true

Acceptance: With APP_ENV=production and any invariant violated → API refuses to start / CI fails

MVP-P0-6. AWS Secrets Manager Migration `[Bucket C]` `[M]`

NFR category: Security (credential management)

Plan: Container runtime config uses secrets references (DATABASE_URL / JWT_SECRET / MASTER_ENCRYPTION_KEY / OPENAI_API_KEY / Twilio / Resend) instead of env vars. IAM role per task; automatic rotation enabled. For dev, pick one of 1Password Connect / Doppler / .env.local and standardize.

Acceptance: staging+prod runtime loads from Secrets Manager / zero secrets in plain env vars / IAM least-privilege per task

MVP-P0-7. Incident Readiness Bundle `[Bucket B+C]` `[M]`

NFR category: Operability (incident response)

Plan:

Status page: BetterStack or statuspage.io; clients can subscribe
External uptime probe: monitor /health across 4 services × 3 endpoints
OnCall rotation: PagerDuty primary + secondary, weekly rotation + escalation chain
Incident Response SOP: SEV-1/2/3 grading + communication templates + post-mortem template

Acceptance: Status page public URL live / OnCall scheduled ≥ 2 weeks ahead / SEV-2 drill completed

MVP-P0-8. Outbound Consistency + 4-Channel Retry/CB Unification `[Bucket B]` `[M]`

NFR category: Reliability (delivery guarantee)

Code state: F3 release path is fire-and-forget (expert-queue.service.ts:367); client UI already shows "sent" but the peer may not have received. The four channels have uneven protection depth:

Channel	retry	circuit breaker	DLQ persist
Email (Resend)	✅ withRetry(3, exp 200ms)	❌	✅ #557
Telegram	❌	❌ (CB exists only on inbound path)	✅ #557
Slack	❌	❌	✅ #557
WhatsApp	❌	❌	✅ #557

The DLQ table channel_failures persists failures ✅ but has zero retry worker.

Plan:

Three-layer protection across 4 channels: withRetry(3, exp) + new CircuitBreaker(channel)
DLQ retry worker (BullMQ scheduled): every 30s scan rows where status='queued' AND retryCount < 5, exponential backoff resend; over threshold → status='dead' → AM notify
Client-side consistency (optional, defer to P1): only emit agent_message_sent after channel ACK

Acceptance: All 4 channels have retry + CB + DLQ persist / DLQ retry worker proven / threshold breach triggers alert

MVP-P0-9. Multi-Instance Active-Active Safety Bundle `[Bucket B]` `[M]`

NFR category: Availability + Reliability + Idempotency (cross-cutting)

Three hard requirements:

R1 Multi-instance active-active deployment does not break business correctness
R2 Zero data loss
R3 Idempotency

Active-active-specific challenges:

Consecutive webhooks from the same client may land on different pods
The same Expert's double-click may cross pods
During rolling deploy, N-1 pods running + 1 draining + 1 starting = mixed-state coexistence

13 Sub-tasks

Category	Count	Sub-tasks
Multi-pod safety	7	9a / 9b / 9c / 9d / 9e / 9f / 9g
Data consistency	3	9h / 9i / 9j
30-dimension patches	3	9k / 9l / 9m

Effort: XS×8 + S×5 ≈ 3-4 weeks

9a. CircuitBreaker → Redis-backed + add CB to 4 outbound channels `[S]`

Use Redis INCR + EXPIRE for cross-pod shared failure counter
Replace in-process state in circuit-breaker.util.ts
4 existing instantiations: channels.controller.ts:125-127 slackCb/emailCb/teamsCb (teams, not WhatsApp) + telegram.service.ts:31 (inbound)
Add 4 outbound CB: in channel-dispatcher.service.ts, wrap dispatchEmail/Slack/WhatsApp/Telegram with withCircuit("<channel>:outbound")
Teams scope decision: confirm whether Teams ships with MVP; if not, drop teamsCb

9b. SchedulerService → BullMQ scheduled jobs `[S]`

5 cron tasks (scheduler.service.ts:38-44) migrate to Queue.add(repeat: cron) + stable jobId:
- daily-summary (0 8 * * *)
- review-summary (0 9 * * *)
- inventory-alert (0 */4 * * *)
- weekly-report (0 9 * * 1)
- snooze-resurface (*/1 * * * *)
Reuse persona-cleanup.processor.ts pattern
BullMQ guarantees each cron runs exactly once globally across pods

9c. `@Cron` health check → leader election `[XS]`

Convert agent-runtime-health.service.ts:20 @Cron("*/1 * * * *") to a BullMQ job or Redis SET NX leader-election pattern

9d. processedEvents Map → Redis NX `[XS]`

Replace in-process Map at agent-api.service.ts:35 with redis.set(key, '1', 'EX', ttl, 'NX')
Apply the same pattern to Tavus dedup (webhook-router.service.ts:53 memDedupe)

9e. setImmediate × 10 inbound → BullMQ `channel-inbound` queue `[S]`

Replace 10 setImmediate sites with channelInboundQueue.add(...):
- channels.controller.ts:229/589/880/914/949/1241/1650/1943 (8 sites)
- conversations.service.ts:1043 (auto-respond fire-and-forget)
- telegram.service.ts:55 (inbound auto-reply)
jobId reuses 9d channel dedup key
BullMQ worker resumes unfinished jobs after restart

9f. setImmediate outbound → BullMQ `channel-outbound` queue `[S]`

3 fire-and-forget sites migrate to BullMQ (bundled with P0-8 retry worker):
- expert-queue.service.ts:367 (Expert release dispatch)
- conversations.service.ts:1043 (auto-respond dispatch — overlaps with 9e; routed through this queue from the outbound perspective)
- telegram.service.ts:55 (same)

9g. DB unique constraints + TOCTOU → INSERT ON CONFLICT `[S]`

Add @Unique to messages.channel_message_id (migration)
Add partial unique on conversations(customerId, channel) WHERE status='pending' (migration)
Rewrite findOrCreateByChannel as INSERT ... ON CONFLICT (customerId, channel) WHERE status='pending' DO NOTHING RETURNING *

9h. Move F3 expert message write into inner transaction `[XS]`

expert-queue.service.ts:274 msgRepo.save(expertMessage) currently lives outside the inner tx at :297
Move msgRepo.save inside manager.transaction so it commits atomically with the queue + conv state flip
Prevents the orphan state "expert message written, but state not flipped"

9i. history_summary → atomic JSONB SQL `[XS]`

conversations.service.ts:978-987 is a read-modify-write; switch to PG jsonb_set:

UPDATE conversations
SET agent_context = jsonb_set(
  coalesce(agent_context, '{}'::jsonb),
  '{history_summary}',
  (... trim to last 10 ...)
)
WHERE id = $1

TypeORM gotcha: raw jsonb_set bypasses entity hooks. Before shipping, grep @AfterUpdate.*Conversation to confirm no listener depends on it. If any listener exists, use SELECT FOR UPDATE instead.

9j. State machine checks → atomic `UPDATE ... WHERE ... RETURNING` `[XS]`

Rewrite 5 TOCTOU patterns to:

UPDATE expert_queue
SET status = 'awaiting_client', last_activity_at = NOW()
WHERE id = $1 AND status != 'resolved'
RETURNING *

Confirmed 5 in-MVP sites:
- expert-queue.service.ts:270 (respond status check)
- expert-queue.service.ts:553 (resolved status check)
- conversations.service.ts:781-783 (awaiting_client → pending)
- conversations.service.ts:1141 (existingQueueItem awaiting_client → pending)
- conversations.service.ts:1186 (existingQueueItem pending messageId advance)
Review by explicit file:line list per PR to avoid touching UI-only status checks

9k. Re-register global ThrottlerModule + Redis storage `[XS]`

Code state: app.module.ts:112 comment says "ThrottlerModule was removed as part of #531", but 5 @Throttle decorators still exist: auth.controller.ts:49/374/385 + kb.controller.ts:81 + conversations.controller.ts:33
Hidden dependency: npm i @nest-lab/throttler-storage-redis

New config:

ThrottlerModule.forRootAsync({
  useFactory: () => ({
    throttlers: [{ name: 'default', limit: 100, ttl: 60000 }],
    storage: new ThrottlerStorageRedisService(redisUrl),  // cross-pod shared
  }),
})

Active-active mandates Redis storage (in-memory default would multiply limits by N pods)

Rate-Limit Policy (Decision 6 — 6 locked sub-decisions):

#	Sub-decision	Locked choice
6.0	Library	`@nest-lab/throttler-storage-redis`
6.1	Throttling strategy	per-IP default; login endpoints + Idempotency-Key endpoints switch to per-user (defer to P1)
6.2	Existing 5 limits	Keep historical numbers unchanged; channel webhooks get separate limit (see 6.3)
6.3	Missing endpoints	MVP scope adds 4 channel webhooks (DDoS defense): Slack `event_id` endpoint 1000/min/IP; Email/WhatsApp/Telegram similarly; admin / upload / billing webhook deferred to P1
6.4	Fail mode	Fail-soft + Sentry alert: Redis storage unavailable → fall back to per-pod in-memory counter (accept temporary active-active degradation) + `redis.storage.disconnected` Sentry alert MUST fire
6.5	Rollout strategy	Shadow mode 1 week → ThrottlerGuard counts but does not reject, logs `would-have-blocked`; after 1-week observation, switch to enforce
6.6	Boundary with 9l	9k uses HTTP 429 + `X-RateLimit-Remaining` header ("client too fast"); 9l uses HTTP 503 + `Retry-After` header ("server overloaded"); response body includes `code: 'rate_limit'` / `code: 'backpressure'` for differentiation

See ADR-021 Throttler Rate-Limit Strategy for full rationale.

9l. BullMQ queue depth metric + Sentry alert `[XS]`

Emit metrics: bullmq.queue.<name>.waiting/active/delayed/failed
Threshold alert: waiting > 1000 || failed > 100 / 5min → Slack alert
HTTP inbound backpressure: webhook controllers return 429 when in-flight count exceeds N

9m. OpenRouter retry 3× + rate-limit alert `[XS]`

Code state: openrouter.client.ts:8/30/113 currently retries once on 429/503 with 2s backoff
Change: 3 retries with exponential backoff (1s/2s/4s) on [429, 502, 503, 504]
Sentry alert: openrouter.consecutive_failures > 5 triggers
Extend the same pattern to Twilio / Resend

Graceful Shutdown Contract (companion to R2)

// api/src/main.ts
process.on('SIGTERM', async () => {
  logger.log('SIGTERM received, draining...');
  await app.close();                           // wait for HTTP handlers to finish
  await Promise.all([
    channelInboundWorker.close(),
    channelOutboundWorker.close(),
    personaCleanupWorker.close(),
  ]);
  await dataSource.destroy();
  process.exit(0);
});

Deployment side: ECS stopTimeout ≥ 120s / BullMQ worker close() waits for current job / 60s LLM timeout fits within the stopTimeout budget

Key Files Inventory

New files:
  api/src/common/redis-circuit-breaker.util.ts       (9a)
  api/src/common/redis-idempotency.util.ts           (9d)
  api/src/channels/inbound-dedup.service.ts          (P0-3 + 9d)
  api/src/channels/channel-inbound.processor.ts      (9e)
  api/src/channels/channel-outbound.processor.ts     (9f, bundled with P0-8)
  api/src/channels/channel-failures.processor.ts     (P0-8 retry worker)
  api/migrations/*-add-message-dedup-unique.ts        (9g)
  api/migrations/*-add-conversation-pending-unique.ts (9g)
  docs/decisions/013-channel-inbound-dedup.md        (new ADR)
  docs/decisions/018-multi-pod-safety-contract.md    (new ADR)
  docs/decisions/019-data-consistency-contract.md    (new ADR)
  docs/decisions/020-throttler-rate-limit-strategy.md (new ADR)

Modified:
  api/src/main.ts                                     (SIGTERM handler)
  api/src/app.module.ts                               (9k ThrottlerModule + Redis storage)
  api/src/channels/circuit-breaker.util.ts            (9a Redis-backed)
  api/src/channels/channels.controller.ts             (9a + 9e + P0-3)
  api/src/channels/channel-dispatcher.service.ts      (9a outbound CB + P0-8)
  api/src/channels/telegram/telegram.service.ts       (9a inbound + 9f)
  api/src/channels/channel-failures.service.ts        (P0-8 markDead / markRetried)
  api/src/scheduler/scheduler.service.ts              (9b BullMQ)
  api/src/agent-runtime/agent-runtime-health.service.ts (9c leader election)
  api/src/agent-api/agent-api.service.ts              (9d)
  api/src/tavus/webhook-router.service.ts             (9d)
  api/src/expert-queue/expert-queue.service.ts        (9f + 9g + 9h + 9j)
  api/src/conversations/conversations.service.ts      (9f + 9g + 9i + 9j)
  api/src/shared/llm/openrouter.client.ts             (9m retry 3 + alert)
  api/src/notifications/redis-io.adapter.ts           (runtime fallback alert; 9c risk)

MVP P0 Code Reuse Index

Item	Reuse
MVP-P0-1 ECS	Emanuele's ECS PR (in-flight)
MVP-P0-2 Socket.io Redis	✅ `api/src/notifications/redis-io.adapter.ts` (already wired)
MVP-P0-3 dedup	`api/src/common/redis.util.ts`
MVP-P0-4 Sentry	`api/src/main.ts` bootstrap pattern
MVP-P0-5 Startup guard	`api/src/common/env.ts` `IS_PRODUCTION`
MVP-P0-6 Secrets	Emanuele's ECS PR
MVP-P0-7 Incident	PagerDuty + Slack (bundled with MVP-P0-4)
MVP-P0-8 Outbound	Email withRetry / Telegram CB / `channel_failures` table (#557) / `persona-cleanup.processor.ts` BullMQ pattern
MVP-P0-9 Multi-Pod	`persona-cleanup.processor.ts` (9b/9e/9f) + `agent-lifecycle.service.ts:85` PG advisory lock (9c reference) + `common/redis.util.ts` (9a/9d)

Decisions Awaiting Paul / E

#	Decision	Options	Recommendation
1	RDS Proxy vs PgBouncer on ECS	A. RDS Proxy / B. PgBouncer	A — managed + auto-failover
2	IaC tool	A. Terraform / B. CDK / C. Pulumi	A — industry standard
3	dev Secrets management	A. 1Password Connect / B. Doppler / C. .env.local	Pick one; standardize
4	Status page tool	A. statuspage.io / B. cachet / C. BetterStack	A or C — avoid self-hosting
5	Teams channel scope	In/out of MVP	Affects 9a `teamsCb` handling
6	Throttler Redis storage lib	A. `@nest-lab/throttler-storage-redis` / B. roll your own	A — locked
6.1	Throttling strategy	per-IP / per-user / per-org / composite	per-IP default (per-user deferred to P1)
6.2	Existing 5 limits	keep / adjust	Keep historical numbers
6.3	Missing endpoints	only restore 5 / +channel webhooks / full coverage	+4 channel webhooks (others deferred to P1)
6.4	Redis unavailable fail mode	fail-open / fail-closed / fail-soft	Fail-soft + Sentry alert
6.5	Rollout strategy	big-bang / shadow / ramp	Shadow mode 1 week → enforce
6.6	Boundary with 9l backpressure	header / status / body	429 + `X-RateLimit-Remaining` (9k) vs 503 + `Retry-After` (9l)

Verification Plan

Standard sprint cadence

git checkout -b feat/architecture-XXX
cd api && npm test
docker compose down -v && docker compose up -d
curl localhost:3000/health
gh pr create --title "..."
# merge → staging auto-deploy → staging smoke → prod cutover

End-to-end staging NFR smoke test (mandatory per deploy)

/health returns ok on all services
Trigger a Sentry test event → staging Slack receives alert (P0-4)
Cross-org IDOR check (org A token hits org B endpoint → 403)
Channel inbound webhook retry → dedup rejects duplicates (P0-3)
Channel outbound failure injection → DLQ retry worker resends successfully (P0-8)
Socket.io 2 pods × 50 sockets × 5 min, zero message loss (P0-2)
RDS Multi-AZ failover drill (kill one AZ) (P0-1)
Production startup invariants triggered (DEMO_MODE=true → refuse to start) (P0-5)
Status page + uptime probe path verified (P0-7)
staging 2 pods × 24h soak; each of the 5 crons fires exactly once (not N times)
Kill pod mid-deploy → BullMQ resumes → 100 inbound webhooks zero loss
Expert UI double-click 10× → only 1 expert message + 1 outbound

Active-Active Acceptance (NFR-flavored metrics)

R1 business correctness:

Business error rate < 0.1% (5xx + business-level errors / total requests)
5 cron tasks each fire exactly once per cycle (daily-summary / review-summary / inventory-alert / weekly-report / snooze-resurface)
Sentry exception count cross-pod variance < 20%
Cross-pod CircuitBreaker state sharing verified
Cross-pod conversation continuation / same webhook cross-pod resend verified

R2 zero data loss:

100 webhooks during deploy + restart → BullMQ resumes → all persisted
Kill pod after Expert release → outbound queue resumes → channel delivers
Rolling update at 100 RPS → zero loss + zero 5xx
Blue-Green drain 120s + zero data loss

R3 idempotency + data consistency:

Each channel resends 100× (spread across pods) → no duplicates
Expert UI double-click 10× (LB intentionally routes across pods) → only 1 outbound
Inject failure in F3 inner tx → no orphan message
5 concurrent messages → history_summary loses none
Client retries POST /messages with same Idempotency-Key → only 1 record persisted

Capacity baseline test

Run in parallel with MVP P0:

k6 100 concurrency / 5 min / bidirectional traffic
Capture: P95 latency / 5xx / PG connection / Redis QPS / LLM tokens
Store in docs/PERFORMANCE_BASELINE.md
Use real numbers to decide P1 trigger order

Risk Register

#	Risk	Likelihood	Impact	Mitigation
1	Emanuele's ECS PR covers prod but not staging	🟡	🔴	PR review checks staging
2	RDS Proxy session pinning conflicts with TypeORM	🟡	🟡	ADR-014 benchmark (planned, not yet authored)
3	PgBouncer transaction mode incompatibility	🟡	🟡	Disable `prepareStatements`
4	Socket.io Redis adapter silently falls back to in-memory	🟡	🔴	pub/sub error → Sentry alert
5	Sentry leaks PII	🟡	🔴	`beforeSend` hook scrubs
6	Webhook dedup key channel mismatch	🟡	🟡	ADR-013 lists all 4 channels
7	dev/staging/prod config drift	🔴	🟡	Terraform + CI check
8	staging ≠ prod causes UAT false positives	🟡	🔴	Mandatory mirror checklist
9	dev secret tools heterogeneous → drift	🟡	🟡	Pick one tool, standardize
10	OnCall not scheduled before first drill	🟡	🟡	Schedule = hard P0-7 acceptance
11	P0-9 BullMQ rollout introduces new failure modes	🟡	🔴	Retry 3 + setImmediate fallback + alert
12	P0-9 DB unique constraints cause INSERT failures on existing data	🟡	🟡	INSERT ON CONFLICT + pre-deploy cleanup
13	P0-9 cron-to-BullMQ leaves stale legacy tasks	🟡	🟡	Clear schedule pre-deploy; stable jobId
14	P0-9 graceful shutdown 120s still has LLM timeout	🟢	🟡	LLM 60s timeout + 2× buffer
15	P0-9 three-requirement implementation across many PRs	🔴	🔴	13 sub-tasks share an epic; joint acceptance
16	ALB sticky misconfigured	🟡	🔴	`stickiness.enabled=false` explicit + IaC enforced
17	Rolling update mixed N + N+M pod versions	🟡	🟡	`minimumHealthyPercent=100` + circuit-breaker auto-rollback; schema migrations expand-contract
18	BullMQ workers contend across pods	🟢	🟡	Built-in distributed lock; tune concurrency
19	Health-check misjudgment causes pod flapping	🟡	🔴	liveness shallow / readiness deep; remove from LB, don't restart
20	9h tx rollback rate increases	🟢	🟡	Monitor + inner-failure retry once
21	9i `jsonb_set` collides with TypeORM cache	🟡	🟡	Raw query bypasses entity cache; ADR-019 codifies
22	9j `UPDATE...RETURNING` not directly supported by TypeORM	🟡	🟡	`createQueryBuilder().returning()` or raw query
23	"Intentionally accepted" consistency mistaken for violation by audit	🟡	🟡	ADR-019 explicitly lists; cite in review
24	9k Throttler Redis storage misconfigured	🟡	🔴	Health check verifies Redis storage writable
25	9l BullMQ metric exhausts Sentry quota	🟡	🟡	Sample + aggregated metric
26	9m OpenRouter retry 3× cumulative latency too long	🟡	🟡	1s+2s+4s=7s still inside 60s; monitor P95

ADR Inventory (status updated 2026-05-28 per #937)

Number reservations have been reconciled with the existing docs/decisions/ files. Existing ADRs in the 009/010 slots were renumbered to free those numbers (workspace-lifecycle → 011, client-portal-ssl-strategy → 015). The 3 ADRs marked T1 below were written in #937; the rest deferred (T2 = pair with infra PRs; T3 = write on demand).

ADR	Topic	Linked to	Status
~~ADR-009~~	~~Haystack shared-instance vs per-tenant~~ — archived 2026-06-02, superseded by ADR-024 (Haystack/Hayhooks migration)	Haystack ops	🗄️ archived
ADR-011	Google Workspace lifecycle (renamed from 009 in #937)	GSuite migration	✅ existing
ADR-013	Channel Inbound Dedup Keys	P0-3	✅ T1 — shipped in #937
ADR-015	Client portal subdomain SSL (renamed from 010 in #937)	infra	✅ existing
ADR-018	Multi-Pod Safety Contract (AA1-AA10 pinned)	P0-9	✅ T1 — shipped in #937
ADR-019	Data Consistency Contract (4 "intentionally accepted" paths)	P0-9	✅ T1 — shipped in #937
ADR-021 (planned)	AWS Deployment Topology	P0-1	T2 — pair with Emanuele ECS PR
ADR-022 (planned)	AWS Secrets Management	P0-6	T2 — pair with Emanuele ECS PR
ADR-023 (planned)	Observability Baseline (Sentry projects naming)	P0-4	T3 — on Sentry rollout
ADR-024 (planned)	RDS Proxy + TypeORM Prepared Statements	P0-1	T2 — pair with ECS
ADR-025 (planned)	Incident Readiness	P0-7	T3 — on OnCall rotation start
ADR-026 (planned)	Outbound Delivery Consistency	P0-8	T3 — in-code comments sufficient for now
ADR-027 (planned)	Throttler Rate-Limit Strategy	P0-9 9k	T3 — Decision 6.1-6.6 in this doc covers the design

Cross-reference: GSuite Migration (#587/#678)

Architectural coupling: GSuite users scale with (Org × Specialist) → IntegrationCredential row count / OAuth refresh traffic / Secrets Manager entries all grow.

NFR impact:

P0-6 Secrets Manager: GSuite OAuth refresh tokens move into Secrets Manager with auto-rotation
P1-4 Redis cache: GSuite tokens are a hot-read object
P1-5 Read replica: GSuite sync jobs use the replica, not the primary

One-line Summary

MVP P0 = NFR directly required by Paul's 5/26 production-ready hosting + 3 hard requirements (active-active / zero data loss / idempotency) + 30-dimension production readiness + data consistency contract.

9 items / 8-10 weeks, with the core being P0-9 Multi-Instance Active-Active Safety Bundle (13 sub-tasks ≈ 3-4 weeks):

Multi-pod safety (7): CB→Redis (9a) / Scheduler→BullMQ (9b) / @Cron leader (9c) / processedEvents→Redis NX (9d) / setImmediate inbound→BullMQ (9e) / setImmediate outbound→BullMQ (9f) / DB unique + ON CONFLICT (9g)

Data consistency (3): F3 msg into tx (9h) / history_summary atomic SQL (9i) / state-machine atomic UPDATE (9j)

30-dimension patches (3): Throttler Redis (9k) / BullMQ queue depth alert (9l) / OpenRouter retry 3× (9m)

Key facts (verified by code line:number): 13 explicit transactions in the entire codebase / only 5 in core flows / zero optimistic locking / 10 setImmediate sites / 5 state-machine TOCTOU sites / 5 leftover @Throttle decorators / 5 cron tasks / all 4 outbound channels lack CB.

30-dimension coverage: 30/30.

Out of scope: Pre-Sale P0 / P1 scaling / P2 mature-product evolution / functional items — tracked via GitHub issues and docs/architecture/README.md. ARCHITECTURE_OPTIMIZATION_2026-05-26.md was referenced but never committed to the repo.

Context​

Deployment Model: Multi-Instance Active-Active​

Active-Active Hard Constraints (10 rules)​

Prerequisites​

Bucket Notation​

Data Consistency Contract​

Complete location of all 13 transactions​

6-tier consistency classification​

Four "intentionally accepted" inconsistencies (codified in ADR-019)​

30-Dimension Production Readiness Coverage Matrix​

Deployment Topology​

Core Flow Sequence Diagrams (NFR Slice)​

F1 Inbound Message Flow​

F2 Agent Draft Flow​

F3 Expert Release Flow​

F4 Realtime Notification Flow​

MVP P0 — Nine Items​

Priority order (based on code audit)​

MVP-P0-1. AWS ECS Active-Active Infrastructure [Bucket C] [L]​

MVP-P0-2. Socket.io Redis Adapter Multi-Pod Validation [Bucket B+C] [S]​

MVP-P0-3. Channel Inbound Dedup Across 4 Channels [Bucket B] [S] 【#1 priority】​

MVP-P0-4. Application-Layer Sentry + Alert Channels [Bucket B+C] [S]​

MVP-P0-5. Production Startup Invariants Guard [Bucket B] [XS]​

MVP-P0-6. AWS Secrets Manager Migration [Bucket C] [M]​

MVP-P0-7. Incident Readiness Bundle [Bucket B+C] [M]​

MVP-P0-8. Outbound Consistency + 4-Channel Retry/CB Unification [Bucket B] [M]​

MVP-P0-9. Multi-Instance Active-Active Safety Bundle [Bucket B] [M]​

13 Sub-tasks​

9a. CircuitBreaker → Redis-backed + add CB to 4 outbound channels [S]​

9b. SchedulerService → BullMQ scheduled jobs [S]​

9c. @Cron health check → leader election [XS]​

9d. processedEvents Map → Redis NX [XS]​

9e. setImmediate × 10 inbound → BullMQ channel-inbound queue [S]​

9f. setImmediate outbound → BullMQ channel-outbound queue [S]​

9g. DB unique constraints + TOCTOU → INSERT ON CONFLICT [S]​

9h. Move F3 expert message write into inner transaction [XS]​

9i. history_summary → atomic JSONB SQL [XS]​

9j. State machine checks → atomic UPDATE ... WHERE ... RETURNING [XS]​

9k. Re-register global ThrottlerModule + Redis storage [XS]​

9l. BullMQ queue depth metric + Sentry alert [XS]​

9m. OpenRouter retry 3× + rate-limit alert [XS]​

Graceful Shutdown Contract (companion to R2)​

Key Files Inventory​

MVP P0 Code Reuse Index​

Decisions Awaiting Paul / E​

Verification Plan​

Standard sprint cadence​

End-to-end staging NFR smoke test (mandatory per deploy)​

Active-Active Acceptance (NFR-flavored metrics)​

Capacity baseline test​

Risk Register​

ADR Inventory (status updated 2026-05-28 per #937)​

Cross-reference: GSuite Migration (#587/#678)​

One-line Summary​

Context

Deployment Model: Multi-Instance Active-Active

Active-Active Hard Constraints (10 rules)

Prerequisites

Bucket Notation

Data Consistency Contract

Complete location of all 13 transactions

6-tier consistency classification

Four "intentionally accepted" inconsistencies (codified in ADR-019)

30-Dimension Production Readiness Coverage Matrix

Deployment Topology

Core Flow Sequence Diagrams (NFR Slice)

F1 Inbound Message Flow

F2 Agent Draft Flow

F3 Expert Release Flow

F4 Realtime Notification Flow

MVP P0 — Nine Items

Priority order (based on code audit)

MVP-P0-1. AWS ECS Active-Active Infrastructure `[Bucket C]` `[L]`

MVP-P0-2. Socket.io Redis Adapter Multi-Pod Validation `[Bucket B+C]` `[S]`

MVP-P0-3. Channel Inbound Dedup Across 4 Channels `[Bucket B]` `[S]` 【#1 priority】

MVP-P0-4. Application-Layer Sentry + Alert Channels `[Bucket B+C]` `[S]`

MVP-P0-5. Production Startup Invariants Guard `[Bucket B]` `[XS]`

MVP-P0-6. AWS Secrets Manager Migration `[Bucket C]` `[M]`

MVP-P0-7. Incident Readiness Bundle `[Bucket B+C]` `[M]`

MVP-P0-8. Outbound Consistency + 4-Channel Retry/CB Unification `[Bucket B]` `[M]`

MVP-P0-9. Multi-Instance Active-Active Safety Bundle `[Bucket B]` `[M]`

13 Sub-tasks

9a. CircuitBreaker → Redis-backed + add CB to 4 outbound channels `[S]`

9b. SchedulerService → BullMQ scheduled jobs `[S]`

9c. `@Cron` health check → leader election `[XS]`

9d. processedEvents Map → Redis NX `[XS]`

9e. setImmediate × 10 inbound → BullMQ `channel-inbound` queue `[S]`

9f. setImmediate outbound → BullMQ `channel-outbound` queue `[S]`

9g. DB unique constraints + TOCTOU → INSERT ON CONFLICT `[S]`

9h. Move F3 expert message write into inner transaction `[XS]`

9i. history_summary → atomic JSONB SQL `[XS]`

9j. State machine checks → atomic `UPDATE ... WHERE ... RETURNING` `[XS]`

9k. Re-register global ThrottlerModule + Redis storage `[XS]`

9l. BullMQ queue depth metric + Sentry alert `[XS]`

9m. OpenRouter retry 3× + rate-limit alert `[XS]`

Graceful Shutdown Contract (companion to R2)

Key Files Inventory

MVP P0 Code Reuse Index

Decisions Awaiting Paul / E

Verification Plan

Standard sprint cadence

End-to-end staging NFR smoke test (mandatory per deploy)

Active-Active Acceptance (NFR-flavored metrics)

Capacity baseline test

Risk Register

ADR Inventory (status updated 2026-05-28 per #937)

Cross-reference: GSuite Migration (#587/#678)

One-line Summary