Runbook: Slack production troubleshooting (per-org / per-workspace)

Scope. Extension of slack-oauth-install.md. That runbook stands up the OAuth install; this one is for when a specific org's Slack install misbehaves in prod — "we installed but nothing happens", "connect fails", "replies don't come back". It's the concrete log keywords + DB queries + Sentry filters to trace one org's Slack activity, distilled from the #3630 investigation.

Inputs you need

Two identifiers anchor every search below:

Input	Where to get it	Example
org_id (org UUID)	client URL / ops / the SQL in step 1	f628ef6c-3faf-4fcd-b71a-e0af885e0665
team_id (Slack workspace id, T…)	step 1 (integration_credentials.team_id) or the Slack event body	T0AQDCQ0U14

org_id and team_id are the only identifiers that appear in plaintext across logs / DB / Sentry — use them as the search keys.

---

1. DB — the binding itself

Does this org have an active Slack workspace, and which one?

SELECT *
FROM integration_credentials
WHERE integration_type = 'slack'
  AND org_id = '<ORG_ID>';

• Expect status = 'active'. status = 'token_invalid' = token revoked / 401 (workspace disconnected).
• team_id = the bound workspace; metadata carries team_name / app_id / bot_user_id.
• ⚠️ SELECT * returns encrypted_config (the encrypted bot token). It's ciphertext, but don't paste results into tickets/chat. Narrow the column list if you only need status.

Reverse — which org holds a given workspace? (the "already connected to another organization" case, #3630)

SELECT org_id, status, team_id, metadata->>'team_name' AS team_name, created_at
FROM integration_credentials
WHERE integration_type = 'slack' AND team_id = '<TEAM_ID>';
-- team_id is unique (uq_integration_slack_team) → at most one row.
-- An active + healthy row here means the workspace legitimately belongs to that
-- org — NOT a stale binding. Do not "clean it up" without a business decision.

Which channels are bound, to which Specialist?

SELECT org_id, external_channel_id, specialist_id, bound_by_slack_user,
       respond_to_all, created_at
FROM slack_channel_bindings
WHERE org_id = '<ORG_ID>'
ORDER BY created_at DESC;

Conversations produced via Slack:

SELECT id, specialist_id, channel, channel_thread_id, created_at
FROM conversations
WHERE org_id = '<ORG_ID>' AND channel = 'slack'
ORDER BY created_at DESC
LIMIT 50;

Outbound failures / DLQ (replies not going back):

SELECT id, channel, org_id, status, retry_count, error, payload, created_at
FROM channel_failures
WHERE channel = 'slack' AND org_id = '<ORG_ID>'
ORDER BY created_at DESC;
-- status: queued | acknowledged | drained; error often token_revoked / not_in_channel.

Optional, for "message arrived but bot didn't respond": slack_engaged_threads (mention gate) and channel_context_messages (channel_type = 'slack').

---

2. Logs — keywords by lifecycle phase

Prod app logs (ECS/CloudWatch). Channel IDs are masked (maskIdentifier), but org= and team= are plaintext — search on those.

Install / OAuth callback

Meaning	Keyword
Install succeeded (credential written active)	Slack app installed: org=<ORG_ID> team=<TEAM_ID> (slack-oauth.service.ts:378)
Org already bound to another workspace	Slack install refused: org=
Workspace already owned by another org (#3630)	Slack install refused: team=
Token exchange failed (bad secret / redirect mismatch)	Slack oauth.v2.access not ok: · Slack oauth.v2.access threw:
Callback error / conflict	Slack OAuth callback failed: · ... conflict: · ... taken:

The client-side success signal is the redirect ?slack=connected&team=… (browser only — not in backend logs). Install success in logs = the Slack app installed: line, which corresponds to the status='active' row in step 1.

Inbound routing

Meaning	Keyword
Every inbound event (carries team=)	Slack event: type=<...> team=<TEAM_ID>
Routed to a conversation	Slack message routed: conv=
Deduped (vendor re-delivery)	Slack deduped: event_ts=
Held for Expert review (default HITL)	Slack reply held for expert review: conv=
Routing error	Slack routing error:

Channel binding

Meaning	Keyword
Invited but org has no Specialist (no welcome, no bind)	Slack channel bind skipped: org=<ORG_ID> has no assigned Specialist (slack-channel-binding.service.ts:67)
Bot removed / channel unbound	Slack channel unbound: org=<ORG_ID> channel=

Outbound

Meaning	Keyword
Reply posted / dispatched	Slack reply posted to channel= · Slack reply dispatched: channel=
Workspace disconnected / bot removed (permanent)	token_revoked · token_invalid · channel_not_found · not_in_channel (slack-errors.util.ts)

---

3. Sentry — filter by org / workspace / channel

Org humanity-0r, EU region de.sentry.io, prod project h-work-production.

As of #3721 / PR #3724, channel events are tagged with org_id / team_id / channel / specialist_id, so these filters work directly in Sentry:

org_id:<ORG_ID>
team_id:<TEAM_ID>
channel:slack

Before #3721 is deployed, those tags are empty — Sentry can only tell you a workspace erred, not which org's. In that case fall back to logs (team=<TEAM_ID>) + the DB in steps 1–2. After deploy, verify the tags exist before trusting the filter.

---

Triage order

1. Step 1 query — confirm status='active' and team_id matches. Wrong/absent → install problem (see slack-oauth-install.md Part A/B).
2. Installed but org gets nothing → grep logs Slack event: … team=<TEAM_ID>.
   • No such lines → the workspace has sent no events: bot never invited to a channel / never DM'd. Installed but never used (the #3630 "HI workspace" state).
   • Lines present but no conversation → check slack_channel_bindings (step 1) + Slack channel bind skipped … no assigned Specialist.
3. Replies not going out → channel_failures (step 1) + outbound permanent-error keywords (token_revoked / not_in_channel).
4. Cross-org "already connected" (#3630) → the reverse team_id query in step 1 tells you the owning org; it is not necessarily stale.

Related

• slack-oauth-install.md — the install runbook this extends.
• #3630 — cross-org workspace-binding investigation that motivated this doc.
• #3721 / PR #3724 — adds the org_id / team_id / channel Sentry tags used in step 3.