Skip to main content

ADR-019: Data Consistency Contract — Layered Acceptance, 4 Intentionally-Accepted Inconsistencies

Date: 2026-05-28 Status: Accepted Issue: #937 (this ADR) Authors: Tom Tang, claude Related: ADR-018 (the multi-pod context in which these guarantees hold), ADR-021 (throttler rate-limit — renumbered from ADR-020 in #1229)

Context​

The platform makes 14 explicit DB transactions across 9 files, with only 5 in the core F1/F2/F3 customer flows. F1/F2 sendMessage performs 17 write steps but only 2 inside an inner transaction. Zero usages of optimistic locking. Cross-service writes (NestJS API ↔ Python agent service) have no distributed transaction.

This is deliberate. Strict ACID across every operation would:

  • Bloat the transaction graph (every inbound webhook becomes a multi-row tx → contention)
  • Couple the lifecycle of dissimilar concerns (audit log failures must NOT roll back the business write that triggered them)
  • Tie the platform's availability to its weakest external dependency (Socket.io → Redis → web client; any link down = whole request fails)

But "we accept some inconsistency" is dangerous unless codified. Without this ADR, a future engineer (or auditor) will see "audit log written outside the tx" and "fix" it — undoing the design and introducing the failure modes the original choice prevented.

This ADR enumerates the 6 consistency tiers we use, the 4 inconsistencies we intentionally accept (and why), and the detection method for each.

Decision: 6-tier consistency classification​

TierSymbolExample operationsHow enforced
Strong consistency🟢F3 Expert release: msg save + queue flip + conv flip in single inner tx (expert-queue.service.ts:288) — all 3 commit together or all roll back. Snooze resurface inner tx.TypeORM manager.transaction(...)
Strong idempotency🟢DB unique + INSERT ON CONFLICT DO NOTHING RETURNING * (P0-9g, #929). Same webhook retry → same row, no duplicate.Partial unique indices + catch 23505 + refetch
Eventual (~30s)🟡Audit log writes outside business tx. Socket.io emit. history_summary JSONB advance.Async fire-after-commit; bounded delay
At-least-once🟡BullMQ work (channels-inbound, channels-outbound, scheduler) + InboundDedupService. Handler must be idempotent.BullMQ retry + dedup key on jobId
Best-effort🔴KB delete Haystack→DB (no saga; orphan possible on partial fail). Outbound "delivered" ground truth (Resend says ok != inbox received).Compensating ops via runbook
Known lost-update🔴history_summary JSONB advance (atomic SQL since P0-9i but legacy reads-stale-then-merges still possible). Per-org message quota counter (race window admits 1-2 extras under burst).Documented + tested as "acceptable drift"

The 4 intentionally-accepted inconsistencies​

Each below is a design choice, not a bug. Reviewers / auditors finding these in production MUST NOT "fix" them without explicit revisit of this ADR.

Accepted #1: Audit log is NOT in the same transaction as the business write​

Location: every AuditService.log(...) call across the codebase. Examples: conversations.service.ts:872, expert-queue.service.ts:327.

The shape:

await this.queueRepo.manager.transaction(async (em) => {
  // ... business writes (Expert message + queue flip + conv flip) ...
});  // tx commits here

// audit log is OUTSIDE the tx
await this.auditService.log({ action: 'queue.respond', ... });

Why intentional:

  • Audit log failure (write to audit_log row blocked by FK / disk full / advisory lock contention) MUST NOT roll back the business write. A customer's expert reply MUST be delivered even if observability is degraded.
  • Audit log latency is dominated by the JSONB serialization + index update; bringing it inside the business tx doubles tx duration → more lock contention → throughput cliff.

Failure mode (acceptable):

  • Business write commits + audit log write fails → 1 unaudited operation in audit_log. Detection: Sentry captures the audit-log exception. Compensation: ops can re-derive the missed audit from messages + expert_queue timestamps (the source-of-truth data is in the business tables).

Detection: grep -rn "@AfterUpdate\|@AfterInsert.*audit" api/src/ returns nothing — there are no entity hooks coupling audit to business writes. If a PR adds one, this ADR is being violated.

Accepted #2: Socket.io emit does NOT guarantee exactly-once​

Location: notifications.gateway.ts emitAgentMessageSent, emitQueueItemAdded, etc. Called by services after commit.

The shape:

await this.convRepo.save(...);
// commit succeeded
this.notificationsGateway?.emitAgentMessageSent({...});  // fire-after-commit, best-effort

Why intentional:

  • Socket.io delivery semantics are at-most-once at the protocol level. Redis adapter (PR #449) adds cross-pod fan-out but does NOT add delivery confirmation.
  • Socket.io with Redis adapter cross-pod fan-out (verified via scripts/load/socketio-fanout-test.mjs); reconnect path on sustained drops; no polling fallback (removed in #843). The reconnect banner is the user-facing recovery signal.
  • Adding an ACK / persistent outbox would double per-event cost and add a new failure path (outbox row but no Socket.io fire).

Failure mode (acceptable):

  • WS connection blip → client misses the live event. Socket.io's built-in reconnect catches up on transient drops; on sustained drops the reconnect banner surfaces in the UI. Customer sees the message with no data loss.

Detection: any code attempting to record "Socket.io delivered" as a persisted flag is violating this ADR. There is no socket_io_delivery_log table on purpose.

Accepted #3: KB delete Haystack→DB has no saga​

Location: kb.controller.ts DELETE handler. Cascades to Haystack pgvector storage + local kb_documents row.

The shape:

  1. Mark kb_documents.deleted_at in PG
  2. Call Haystack to drop the embedding
  3. If step 2 fails, the row stays soft-deleted in PG but the embedding remains in Haystack

Why intentional:

  • A full saga (compensating PG re-undelete on Haystack failure) requires durable state for the orchestrator + retry logic + observability for "stuck" sagas. Complexity outweighs the actual failure rate.
  • Orphan embeddings in Haystack are harmless to query correctness (the retrieval layer joins to PG and filters out deleted_at IS NOT NULL).

Failure mode (acceptable):

  • Haystack holds orphan vectors. Storage cost only. No correctness impact on user-facing search.

Detection / compensation: a quarterly ops job sweeps kb_documents with deleted_at IS NOT NULL and re-issues the Haystack delete for any orphans found. Owner: KB pause/reactivate runbook (archived during the P3-6 ADR sweep; see the new docs/runbooks/kb-pause-reactivate.md).

Accepted #4: Cross-service writes (NestJS ↔ Python agent) have no distributed transaction​

Location: AgentClient.chat() in agent.client.ts and the receiving side in agent/main.py.

The shape:

  • NestJS sends POST /chat to agent service
  • Agent processes (writes own per-org workspace files, calls LLM, returns reply)
  • NestJS saves the returned agent_message row in PG

If the agent successfully processed + wrote workspace files but the response was lost (network blip between agent and NestJS), the agent has done work but NestJS has no record.

Why intentional:

  • A distributed tx (2PC across HTTP) requires both endpoints to participate in a coordinator protocol. Implementation cost is high; the failure rate it would protect against is low.
  • Agent-side workspace writes are themselves idempotent (same request payload → same file outputs). NestJS retry produces the same workspace state.
  • The reply that NestJS would have saved is regenerable: same user message + same agent prompt → similar (not byte-identical, LLM is non-deterministic) reply.

Failure mode (acceptable):

  • Workspace has files from an attempt that NestJS doesn't know about. A subsequent user message regenerates the conversation. Storage cost = a few KB of orphan workspace files; cleanup via the per-org agent's WorkspaceManager housekeeping.

Detection: there is no two-phase commit code anywhere; if one appears, this ADR is being violated.

Options considered​

  1. Strict ACID everywhere — Rejected: every audit write would have to commit with its business write, doubling lock duration and adding new failure modes (a sudden audit_log lock contention storm would cascade into business-write failures). Not worth the contention cost.

  2. Distributed transactions across services — Rejected: 2PC's failure modes (orphan locks on coordinator failure) and operational complexity exceed the value for our actual failure rate (~10⁻⁴ per cross-service request).

  3. Eventual consistency everywhere — Rejected: F3 Expert release MUST be all-or-nothing within the api boundary, or the Expert workspace and client portal can disagree about whether the message was sent. The inner-tx is non-negotiable for the core flow.

Consequences​

Positive:

  • Each layer's failure does not cascade to others (audit fail ≠ business fail; Socket.io fail ≠ DB fail; agent fail ≠ DB fail)
  • Operations team understands what failure modes are "expected" vs "alarming"
  • New engineers reading the code know the consistency tier of each operation (this doc + comments)

Negative:

  • Tier 🟡 / 🔴 paths need explicit documentation per call site (every audit call could be miscategorized as a bug)
  • Audit / compliance review needs this ADR in hand to read the code without flagging the 4 accepted inconsistencies

Validation​

  • grep -rn "auditService.log" api/src/ confirms audit calls are outside manager.transaction blocks
  • expert-queue.service.ts:288-320 shows the F3 inner tx (Tier 🟢 strong consistency) with the 3 writes
  • useNotifications.ts confirms Socket.io reconnect path is the Tier 🟡 recovery mechanism — the 8s polling fallback was removed in #843; the reconnect banner is the user-facing fallback when WS drops
  • The KB pause/reactivate runbook documents the orphan cleanup procedure (archived during the P3-6 ADR sweep; current runbook is docs/runbooks/kb-pause-reactivate.md)

Decision authority + enforcement​

This ADR is the source of truth for:

  • Reviewing PRs that move writes into / out of transactions
  • Responding to audit / compliance questions about consistency guarantees
  • Onboarding new engineers ("which tier is this operation?")

PRs that change the consistency tier of an operation (e.g. moving an audit call inside a business tx, adding a saga to KB delete, re-introducing a polling fallback) MUST cite this ADR + the affected tier in the description, AND update this ADR's table.

Reference​

  • MVP P0 NFR FINAL §"Data Consistency Contract" — original 6-tier classification
  • CLAUDE.md §"Data Consistency Contract" — operational summary
  • ADR-018 (multi-pod safety) for the active-active context
  • ADR-013 (dedup keys) for Tier 🟡 at-least-once + idempotency interaction
Last updated on by Degen