Skip to main content

Runbook: Email Cutover Rollback (Gmail → Legacy Resend/Cloudflare)

P1. If per-org Gmail email routing (#1438) breaks outbound email for an organization, Specialists in that org cannot send messages to customers. Rolling back to the legacy Resend/Cloudflare path restores service within 30s. See OWNERS.md for on-call contact.

⚠️ PREREQUISITE: This runbook documents the rollback procedure for the use_workspace_email flag implemented in #1438. Do not use this runbook until #1438 is merged and deployed. If that flag is not yet in production, the SQL commands in this runbook will silently succeed without affecting email routing behavior.

This runbook covers rolling back per-organization email routing from Gmail to the legacy Resend/Cloudflare path when the per-org cutover feature (organizations.metadata.use_workspace_email) causes email delivery failures. The legacy path (Resend for transactional, Cloudflare Email Routing for inbound) remains available during cutover for exactly this case. Architecture: ADR-0002 Phase 2 — Specialist Email (channels.md §System 3). Related issue: #688 (parent), #1438 (flag implementation).

Symptoms​

  1. Customer complaints: "Bob sent a message but I never got it." Concentrated within minutes of cutover, in one organization.
  2. No outbound audit log rows in email_audit_log for the affected organization in the last N minutes (where N starts at "minute of cutover").
  3. Sentry errors — Gmail API: send failed or Workspace email: unauthorized / quota exceeded for a specific org_id.
  4. Outbound email dispatcher logs — EmailService: Gmail path failed for org=<id>, fallback unavailable.
  5. Specialist reports — "I clicked Send but the customer says they got nothing."

Step 1 — Confirm the failure is cutover-related, not application-wide​

Before rolling back the flag (org-scoped, requires DB write), rule out:

1a. Is email working for organizations NOT on Gmail cutover?​

SELECT org_id, COUNT(*) AS sent_last_10m
FROM email_audit_log
WHERE direction = 'outbound'
  AND created_at > NOW() - INTERVAL '10 minutes'
  AND org_id != '<suspected_org_id>'
GROUP BY org_id
ORDER BY sent_last_10m DESC
LIMIT 10;

If other orgs are sending normally → failure is cutover-specific. Continue to Step 2.

If ALL orgs are silent → application-wide outage (Resend API down, API pods down). Stop, do not roll back the flag. Go to incident-response.md.

1b. Is the affected org actually using Gmail cutover?​

SELECT id, name, metadata->'use_workspace_email' AS use_workspace_email
FROM organizations
WHERE id = '<org_id>';

Expected: "true" (as JSON boolean or string). If null or "false" → cutover flag is NOT the root cause. Investigate Gmail API credentials or workspace provisioning.

1c. Are Gmail API credentials present for this org?​

SELECT org_id, COUNT(*) AS osa_count,
       SUM(CASE WHEN oauth_refresh_token_encrypted IS NOT NULL THEN 1 ELSE 0 END) AS with_token
FROM org_specialist_assignments
WHERE org_id = '<org_id>'
GROUP BY org_id;

If with_token = 0 → OAuth never completed. Go to oauth-token-refresh.md.

If 1a + 1b + 1c all check out and email still isn't flowing → cutover flag is the problem. Continue to Step 2.

Step 2 — Roll back the cutover flag​

2a. Flip the flag to false​

-- Roll back org to legacy Resend/Cloudflare path
UPDATE organizations
SET metadata = jsonb_set(
  COALESCE(metadata, '{}'::jsonb),
  '{use_workspace_email}',
  'false'::jsonb
),
updated_at = NOW()
WHERE id = '<org_id>';

What happens to in-flight emails during the flag flip?

  • Emails already in the outbound dispatcher queue when you run the UPDATE will complete via the path they started on (Gmail). If Gmail is broken, they will fail and retry.
  • The outbound dispatcher checks the flag at dispatch time (per message), not at queue time.
  • After the UPDATE commits, new messages enqueued will route via the legacy Resend/Cloudflare path.
  • In-flight emails in the Gmail path that fail will be retried. On retry (typically 30s - 2min later), they will read the new flag value and route via Resend.

No lost messages — BullMQ retries failed sends. The worst case is a 2-minute delay for in-flight messages that were mid-send when the flag flipped.

2b. Verify the rollback took effect​

SELECT id, name, metadata->'use_workspace_email' AS use_workspace_email, updated_at
FROM organizations
WHERE id = '<org_id>';

Expected: "false" (or false as JSON boolean). If still "true" → transaction didn't commit; check DB connection and retry.

Step 3 — Verify email flow resumed​

3a. Check outbound dispatcher logs​

# Example: Kubernetes logs for outbound-dispatcher pod
kubectl logs -l app=outbound-dispatcher --tail=50 --timestamps | grep "org=<org_id>"

# Expected log line:
# [timestamp] EmailService: routing org=<org_id> via legacy Resend (use_workspace_email=false)

3b. Smoke test: trigger an outbound send​

If safe (non-production, or production with AM approval):

curl -X POST https://api.<env>.h.work/messages/send \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "org_id": "<org_id>",
    "to": "test+rollback@humanity.org",
    "subject": "Rollback smoke test",
    "text": "Testing legacy path after rollback"
  }'

Expected:

  • 200 response
  • email_audit_log row with direction = 'outbound' created within 5s
  • Resend dashboard shows delivery within 30s

3c. Check metrics dashboard​

  • Grafana (or your observability tool) → Email Routing dashboard:
    • Panel: "Emails by routing path" — should show uptick in resend path, drop in gmail path for the affected org
    • Panel: "Email send errors by org" — should show error rate drop to 0 for org_id=<org_id> within 1 minute
  • Sentry → Issues → filter by org_id=<org_id> and EmailService — error rate should drop to 0 after rollback

3d. Monitoring command: per-org email routing status​

To check email routing status for any organization at any time:

-- Check routing status and recent email activity
SELECT
  o.id AS org_id,
  o.name AS org_name,
  o.metadata->'use_workspace_email' AS use_workspace_email,
  COUNT(eal.id) FILTER (WHERE eal.created_at > NOW() - INTERVAL '10 minutes') AS emails_last_10m,
  MAX(eal.created_at) AS last_email_sent
FROM organizations o
LEFT JOIN email_audit_log eal ON eal.org_id = o.id
  AND eal.direction = 'outbound'
WHERE o.id = '<org_id>'
GROUP BY o.id, o.name, o.metadata;

To list ALL orgs currently on Gmail cutover:

SELECT id, name, metadata->'use_workspace_email' AS use_workspace_email, updated_at
FROM organizations
WHERE metadata->>'use_workspace_email' = 'true'
ORDER BY updated_at DESC;

Step 4 — Communicate​

  1. Slack #ops-incidents: "Rolled back email cutover for org <org_name> (<org_id>) at <UTC time> due to . Outbound resumed via legacy Resend path. Investigating Gmail cutover root cause."
  2. Email impacted AM / org admin: "Some outbound emails between <window> may have been delayed. We've restored the previous email path. All queued messages will be delivered within 2 minutes."
  3. Capture in incident channel:
    • Cutover start time (when use_workspace_email was set to true)
    • Rollback start time (now)
    • Rollback complete time (when Step 3 verification passed)
    • Number of failed sends in the window (from Sentry or email_audit_log error logs)

Step 5 — Troubleshooting: rollback complete but emails still not flowing​

If you completed Step 2 (flag flipped to false) and Step 3 verification (query confirms use_workspace_email=false), but emails STILL aren't sending:

5a. Check legacy Resend API credentials​

# Check environment variable is set
kubectl exec -it deploy/api -- env | grep RESEND_API_KEY

# Expected: RESEND_API_KEY=re_...

If missing → check Secrets Manager / Kubernetes secret:

kubectl get secret humanwork-api-secrets -o jsonpath='{.data.RESEND_API_KEY}' | base64 -d

If still missing or revoked → regenerate in Resend dashboard, update secret, restart API pods.

5b. Check Resend API status​

curl -s https://status.resend.com/api/v2/status.json | jq .

If status.indicator != "none" → Resend is degraded/down. Check their status page; wait for resolution or escalate to Platform Lead for manual SMTP fallback.

5c. Check Cloudflare Email Routing worker (inbound path)​

# Cloudflare dashboard → Email Routing → Overview
# Should show "Active" status
# If "Paused" or "Error" → click Activate

Worker code is in scripts/local/legacy-cloudflare-worker/. If worker was accidentally deleted → redeploy from repo.

5d. Check BullMQ outbound queue​

The outbound dispatcher uses BullMQ. If messages are enqueued but not sending:

# Redis CLI or BullMQ dashboard
# Check queue: "channels-outbound"
# Count waiting jobs:
redis-cli LLEN bull:channels-outbound:wait

# If > 100 → queue is backed up. Check dispatcher pod health.
kubectl get pods -l app=outbound-dispatcher
# If pod CrashLoopBackOff → check logs for error, fix, restart

See bullmq-queue-backlog.md for detailed BullMQ troubleshooting.

5e. DNS MX records (inbound only)​

If OUTBOUND is working but INBOUND is not → this runbook is for outbound cutover rollback. For inbound MX issues, see dns-mx-rollback.md.

Rollback the rollback (re-attempt Gmail cutover)​

After the Gmail cutover root cause is fixed (OAuth tokens, Gmail API quota, workspace provisioning):

  1. Pre-cutover smoke: Verify Gmail Send works for at least one OSA in the org (see oauth-token-refresh.md Step 4).
  2. Re-flip flag: 
    UPDATE organizations
    SET metadata = jsonb_set(
      COALESCE(metadata, '{}'::jsonb),
      '{use_workspace_email}',
      'true'::jsonb
    ),
    updated_at = NOW()
    WHERE id = '<org_id>';
  3. Monitor for 1 hour: Check email_audit_log, Sentry, and Grafana. Watch for send failures.
  4. Declare success only when 100% of outbound for the org lands (no errors for 1 hour).

Step 6 — Escalate​

  • Rollback didn't restore email flow after 5 minutes (Step 5 troubleshooting exhausted) → page Platform Lead + on-call SRE. Possible cascading failure in Resend or DNS.
  • Multiple orgs affected simultaneously → P0, do NOT roll back one-by-one. Page Platform Lead immediately; likely infrastructure-wide issue (Gmail API key revoked, Resend account suspended).
  • Customer SLA breach (e.g., financial services client with 15-min reply SLA) → page CS + Platform Lead together.
Last updated on by Degen