Skip to main content

Runbook: DNS MX Rollback to Cloudflare Worker

2026-06-05 update: MX records for h.work, h852.work, h853.work are already pointing at Google Workspace (confirmed via DoH). Issue #1061 (inbound MX cutover) is effectively done.

P0 during cutover window, P1 afterwards. If the Gsuite DNS MX cutover (#1061) breaks inbound email for an entire domain, every Specialist in that domain stops receiving customer messages. See OWNERS.md for on-call contact.

This runbook covers rolling DNS MX records back from Gsuite to the legacy Cloudflare Email Routing worker during/after the #1061 cutover. The Cloudflare worker (scripts/local/legacy-cloudflare-worker/) is kept parallel during cutover for exactly this case. Architecture: ADR-0002 Phase 2 — Specialist Email (channels.md §System 3).

Symptoms​

  1. Customer complaints: "I emailed Bob but nothing happened." Concentrated within minutes of cutover, in one domain (h.work / h852.work / h853.work).
  2. No channel.email.inbound audit log rows for the affected domain in the last N minutes (where N starts at "minute of cutover").
  3. Cloudflare Email Routing dashboard shows 0 deliveries in the cutover window (means traffic stopped flowing through CF — expected if MX successfully moved to Gsuite, but BAD if Gsuite isn't receiving).
  4. Google Admin → Reports → Email Log Search shows 0 messages or Delivery failed: domain not in tenant errors for the affected domain.
  5. DNS query (dig MX h.work) shows Gsuite MX servers but external senders receiving NDR (Non-Delivery Reports).

Step 1 — Confirm the failure is MX-side, not application-side​

Before rolling back DNS (slow, observable, customer-visible), rule out:

1a. Is the API handling email inbound at all?​

curl -X POST https://api.<env>.h.work/channels/email/inbound \
  -H "Content-Type: application/json" \
  -d '{"from":"test@example.com","to":"bob.acme@h.work","subject":"smoke","text":"hello"}'

Expect 200. If 5xx → application is broken, NOT a DNS/MX issue. Stop, do not roll back DNS. Go to incident-response for API.

1b. Is Gsuite forwarding to our webhook?​

In Google Admin → Apps → Gmail → Routing rules: should have a rule forwarding *@<domain> to our POST /channels/email/inbound. If absent → fix the routing rule first; cutover succeeded at MX level but Gsuite-side routing is misconfigured.

1c. Is the Cloudflare worker still configured?​

Cloudflare → Email Routing → Routes: confirm the routes are still defined (we kept them parallel during cutover; they should not have been deleted yet).

If 1a + 1b + 1c all check out and traffic still isn't flowing → MX cutover is the problem. Continue to Step 2.

Step 2 — Roll back the MX record​

2a. Identify the prior MX values​

The pre-cutover MX records for each domain pointed to Cloudflare's mail relay. Pre-cutover values must have been captured in the cutover prep doc (docs/runbooks/) or in the PR that did the cutover. If not, retrieve from:

# Old DNS provider's API or dashboard — values like:
# h.work.   IN MX 10 mx.cloudflare-mail.net.
# (exact value depends on which CF Email Routing tier we were on)

If you cannot find the prior values within 5 min, escalate (Step 5) — guessing wrong here loses email permanently.

2b. Restore in DNS provider​

# Example: Cloudflare DNS (most common for our domains)
# Cloudflare dashboard → <domain>.work → DNS → MX records → restore the CF mail relay values
# Set TTL = 60s (so a subsequent re-cutover can be done quickly).

Do NOT change the apex A/CNAME — those are unrelated to MX. Only touch the MX rows.

2c. Verify propagation​

dig MX <domain>.work @1.1.1.1   # Cloudflare resolver
dig MX <domain>.work @8.8.8.8   # Google resolver — important, ironic

Both should show the CF mail relay MX values within 1-2 min if TTL was 60s pre-cutover. If still showing Gsuite MX values 5 min in → DNS provider isn't propagating; escalate.

2d. Verify inbound smoke​

echo "rollback test $(date)" | mail -s "MX rollback smoke" bob.acme@<domain>.work
# Watch CF dashboard → Email Routing → Deliveries
# Watch our API logs for /channels/email/inbound

Should see delivery + 200 response within 30s.

Step 3 — Communicate​

  1. Slack #ops-incidents: "Rolled back MX for <domain> at <UTC time> due to . Inbound resumed at
  2. Email impacted AMs / customers: "Some inbound emails between <window> may have bounced. We've restored the previous email path. Please ask your clients to retry any unanswered messages from that window."
  3. Capture in incident channel:
    • Cutover start time (when we changed MX to Gsuite)
    • Rollback start time (now)
    • Rollback complete time
    • Number of NDRs from external senders in the window (best-effort estimate)

Step 4 — Recover lost messages (best-effort)​

External senders should have received NDRs and will resend manually in most cases. For automated senders (SaaS notifications, etc.) that won't resend:

  • No on-platform recovery path: we don't have a queue of "bounced messages from upstream MTAs."
  • Workaround: ask customers if they recall any time-sensitive messages from the window; have them forward to us manually.

Rollback the rollback (re-attempt Gsuite cutover)​

After Gsuite-side root cause is fixed (Step 1b / Google Admin routing rule), schedule another cutover:

  1. Pre-cutover smoke: dummy send to bob.acme@<domain> lands in our DB.
  2. Re-flip MX to Gsuite values.
  3. Wait 24h observation.
  4. Only declare success when 100% of inbound for the domain landed in the API.

Step 5 — Escalate​

  • Lost the prior MX values? Cannot find them within 5 min? → page DNS owner + Platform Lead immediately.
  • DNS provider not propagating after 10 min? → contact DNS provider support.
  • Customer SLA breach (e.g., bank client with 1h reply SLA) → page CS + Platform Lead together.
Last updated on by Degen