Skip to main content

Runbook: Isolation Violations

Always P0. A cross-Specialist or cross-Org data leak is a SLA + GDPR breach. Page Platform Lead + Eusden immediately. Do not wait for business hours.

This runbook covers incidents where the isolation matrix from ADR-020 was violated — most commonly, a Specialist context reading another Specialist's customer PII inside the same Org. See OWNERS.md for on-call contact.

Symptoms — how a violation surfaces​

The leak rarely fires its own alert. It usually shows up as one of:

  1. Customer complaint — "Bob the KYC Specialist replied citing facts from my conversation with Sarah Customer Service." Treat any "the AI knew something it shouldn't have known" report as P0 until ruled out.
  2. Expert workspace review — Expert opens a queue item and sees prior-message context belonging to a different Specialist in the sidebar.
  3. Audit log diff — expert_access grant history shows a user accessed a conversation outside their scope.
  4. Sentry pattern — [OrgRlsInterceptor] warns of missing app.current_specialist_id GUC paired with a Conversation / Message query.
  5. chk_conversations_specialist_id_not_null CHECK constraint violation in PG logs — indicates new write path missed the matrix row 1 enforcement.
  6. Billing anomaly — a customer's monthly bill includes runs from an OSA they never paid for; root cause may be cross-OSA agent_runs query.

If you see any of the above, follow the steps below in order. Do not skip to remediation before scoping the blast radius.

Step 1 — Contain (within 15 min)​

1a. Identify the suspected leak path​

From the customer report or Sentry trace, capture:

  • The affected org_id
  • The Specialist context that wrongly saw the data (call it Specialist-A)
  • The Specialist context that owned the data (call it Specialist-B)
  • The user account that observed the leak (user_id, role)
  • Approximate timestamp window

Document these in the incident channel before doing anything else. Forensics first, fixes second.

1b. Revoke the leaking user's session immediately​

-- Force session re-auth for the user that observed the leak
UPDATE users SET token_version = token_version + 1 WHERE id = '<user_id>';

Then in Slack / email, notify the user their session was reset for security review.

1c. Optional: quarantine the affected OSA(s)​

If Specialist-A's OSA appears to be issuing queries that ignore specialist_id:

-- Soft-suspend the OSA so no new inbound traffic routes to it.
UPDATE org_specialist_assignments
   SET gsuite_suspended_at = now()
 WHERE id = '<osa_id_of_specialist_a>';

The inbound email partial index idx_osa_email_alias_active excludes suspended OSAs; new email to that alias will fall to the legacy fallback path with a Sentry breadcrumb. Active conversations remain readable to the legitimate Expert; new sends pause until the underlying bug is patched.

Step 2 — Scope the blast radius (next 30 min)​

Run these queries with app.current_role_domain='internal' (i.e. an HP staff token, not a tenant token).

2a. How many Conversations are missing specialist_id entirely​

SELECT count(*) AS total_null,
       count(DISTINCT org_id) AS orgs_affected
  FROM conversations
 WHERE specialist_id IS NULL
   AND created_at >= now() - interval '7 days';

Any number > 0 within the last 7 days indicates a code path that bypassed chk_conversations_specialist_id_not_null somehow (the CHECK is NOT VALID so historical rows are legal; new rows should fail INSERT). If new rows are NULL, find the offending write site fast — see Step 3.

2b. Find Conversations where the assigned OSA does not match the conversation's specialist_id​

SELECT c.id, c.org_id, c.specialist_id AS conv_specialist,
       o.id AS osa_id, o.specialist_id AS osa_specialist
  FROM conversations c
  LEFT JOIN org_specialist_assignments o
    ON o.org_id = c.org_id AND o.specialist_id = c.specialist_id
 WHERE c.specialist_id IS NOT NULL
   AND o.id IS NULL  -- no matching OSA for that (org, specialist)
 LIMIT 100;

Any row here is a Conversation pointing at a Specialist who isn't actually assigned to the Org. Treat each row as a potential leak — pull message contents and review with the customer.

2c. Find Expert queue items / messages that crossed Specialist boundaries​

SELECT q.id, q.org_id, q.assigned_expert_id, q.specialist_id,
       c.specialist_id AS conv_specialist
  FROM expert_queue_items q
  JOIN conversations c ON c.id = q.conversation_id
 WHERE q.specialist_id IS DISTINCT FROM c.specialist_id
 LIMIT 100;

2d. Find AgentRuns with osa_id belonging to a different (org_id, specialist_id) pair than the agent_run carries​

SELECT r.id, r.org_id, r.specialist_id, r.osa_id,
       o.org_id AS osa_org, o.specialist_id AS osa_specialist
  FROM agent_runs r
  JOIN org_specialist_assignments o ON o.id = r.osa_id
 WHERE r.osa_id IS NOT NULL
   AND (r.org_id <> o.org_id OR r.specialist_id <> o.specialist_id)
 LIMIT 100;

Any row here means a producer wrote inconsistent FKs — possible billing impact (per #1121).

2e. Export the suspect data for incident records​

psql "$DATABASE_URL" -c "COPY (
  SELECT * FROM conversations WHERE org_id = '<org_id>' AND created_at >= '<from>' AND created_at < '<to>'
) TO STDOUT WITH CSV HEADER" > incident-<INC-ID>-conversations.csv

Store under s3://humanwork-incidents/<INC-ID>/. Do not paste raw PII into Slack or GitHub.

Step 3 — Find and fix the offending code path​

The matrix violation almost always falls into one of these categories. Walk them in order; first hit ends the search.

3a. Service-layer query missing specialist_id filter​

grep for any TypeORM call against a matrix-row-1 table that filters org_id only:

rg -n "(conversations|messages|expert_queue_items)" api/src --type ts -A 5 \
  | rg "where:.*orgId" \
  | rg -v "specialistId"

For each hit: check whether the endpoint runs in Specialist context (channel webhook, OSA detail, AM Setup post-assignment). If yes, add the specialistId filter. Write a regression spec that constructs two Specialists in the same Org and asserts the query returns only one's data.

3b. RLS interceptor not setting app.current_specialist_id​

Check api/src/common/rls.interceptor.ts resolves the Specialist context for the affected endpoint. The interceptor should call withSpecialistContext(orgId, specialistId, …) (see api/src/common/rls-session-context.service.ts). If it's calling the older withOrgContext instead, that's the bug.

3c. New entity added without docstring matrix-row assignment​

ADR-020 requires every entity with PII-class data to declare its matrix row in the docstring. PR review missed one if a recently-added entity has no row assignment + no specialist_id column.

git log --since="30 days ago" --diff-filter=A --name-only -- "api/src/**/*.entity.ts" "api/src/common/entities.ts"

Audit each new entity.

3d. Backfill bug: legacy row with specialist_id = NULL was served to a Specialist context​

The chk_conversations_specialist_id_not_null constraint is NOT VALID — old rows can be NULL. A Specialist-context endpoint querying with WHERE org_id = ? AND specialist_id = ? will skip those rows (correct). But an endpoint using IN (...) OR NULL patterns may pull them. Grep for IS NULL clauses on specialist_id.

Step 4 — Hotfix and deploy​

For application-layer bugs (3a, 3b, 3c, 3d): patch + spec + standard PR cycle, but deploy via Eusden as a hotfix, not waiting for the next release train.

If the root cause is a missing DB-level constraint (rare; would mean ADR-020 itself is incomplete), open an [Incident] issue under epic #1116 and bring the fix to ADR-020 owners before deploying.

Emergency RLS lockdown (only if app fix can't ship within 4 hours)​

A scorched-earth fallback if the leak is active and ongoing:

-- Halt all writes to the affected table from the application role.
-- This breaks the app but stops further leakage.
REVOKE INSERT, UPDATE ON conversations FROM humanwork_app;

Use only with Platform Lead + Eusden approval. Restoring access requires a Platform-Lead-signed runbook re-grant.

Step 5 — Customer comms​

Drafts owned by the Platform Lead. Send within 4 hours of confirmed leak.

Subject: [Action Required] Incident notification — INC-<ID>

Dear <Org admin>,

On <date> at <UTC time>, we detected that <user_role at your organization>
may have viewed <conversation count> conversation(s) belonging to a
different Specialist in your organization. The conversation content
itself remained within <Org>; no data left your organization.

We immediately:
  1. Revoked the affected user's session
  2. Patched the underlying access control bug (commit <sha>)
  3. Audited all conversations created during the window <from> – <to>

Specific impact:
  - <N> conversation(s) reviewed by the affected user that should not
    have been visible
  - <list of Specialist personas involved>
  - No external data exfiltration detected
  - No billing impact (or: <explain billing impact>)

Steps you may want to take:
  - Review the attached audit log
  - Internal communication to the affected user
  - Reset any credentials shared in the affected conversations

Full postmortem will follow within 5 business days.

Sincerely,
<Platform Lead>

For GDPR-regulated jurisdictions (EU customers): the Data Protection Officer must be notified within 72 hours.

Step 6 — Postmortem​

Open within 1 business day in docs/incidents/INC-<ID>.md. Required sections:

  1. Summary — what leaked, how many rows, how long, who saw it
  2. Timeline — UTC timestamps for: first occurrence, detection, containment, customer comms, fix deploy
  3. Root cause — the exact line of code or migration that violated ADR-020 matrix
  4. Why ADR-020 didn't catch it — was the matrix incomplete? Was the linter/review/spec coverage missing? File an [Epic #1116 follow-up] issue for the gap.
  5. What changed — link to hotfix PR + any new regression specs
  6. Customer impact — orgs affected, conversations affected, comms sent, GDPR-DPO notifications
  7. Action items — owner + due date for each (CHECK constraint promotion, query audit, etc.)

Tag the postmortem incident + isolation + the relevant matrix row (e.g. matrix-row-1).

Prevention checklist​

Items to re-verify after every isolation incident:

  • OrgRlsInterceptor sets both app.current_org_id and (where applicable) app.current_specialist_id
  • New PR's against api/src/conversations/**, api/src/expert-queue/**, api/src/channels/** are reviewed for matrix-row-1 compliance
  • CHECK constraint chk_conversations_specialist_id_not_null is still in place; promote to VALIDATE if backfill complete
  • agent_runs.osa_id and tool_calls.osa_id populated by every write site (#1121 follow-up)
  • LlmCostRollupService writes carry osa_id (#1151) and pass the PR-C audit-fields contract (#1185/#1187 follow-ups) — billing rollups pivot by osa_id and must not aggregate across Specialists in the same Org
  • CLAUDE.md "Multi-Tenancy" section reflects the matrix verbatim

References​

  • ADR-020 Isolation Classification Rules
  • ADR-007 Expert Access Scope
  • Epic #1116 — Domain Model Alignment
  • #1151 — LlmCostRollupService osa_id plumbing
  • #1185 / #1187 — PR-C audit-field follow-ups (Org × Specialist audit pivots)
  • api/src/common/rls-session-context.service.ts — RLS GUC helpers
  • api/src/common/rls.interceptor.ts — request-level RLS setup
Last updated on by Degen