Skip to main content

Runbook: DKIM Key Rotation

P2 planned, P1 if drift causes deliverability issues. DKIM (DomainKeys Identified Mail) is the cryptographic signature on our outbound email proving it's really from *@h.work / h852.work / h853.work. Rotating the key periodically is good hygiene; failing to rotate or rotating wrong drops deliverability. See OWNERS.md for on-call contact.

This runbook covers scheduled DKIM key rotation for our Workspace domains. Cadence: annual for prod (h.work), opportunistic for non-prod. Architecture: ADR-0002 Phase 2 — Specialist Email (channels.md §System 3) — DKIM is configured at the Gsuite tenant level, not per-OSA.

Why rotate​

  1. Compromise hygiene — if a private DKIM key ever leaked, all our signed email is forgeable. Rotation invalidates the leaked key.
  2. Cryptographic agility — current Google default is RSA 2048; we may need to bump if Google deprecates it.
  3. Spam-filter reputation — some receivers boost trust on freshly-rotated keys with proper DNS hygiene (rare but real).

Cadence​

DomainCadenceReason
h.work (prod)Annual (each Jan, calendar-blocked)Production hygiene; reputation matters
h853.work (staging)Opportunistic — when we test the prod rotation flowUsed as a rehearsal env
h852.work (dev)Skip unless investigatingDev traffic doesn't matter for deliverability

Env mapping per #1537 canonicalization: h852.work = dev, h853.work = staging, h.work = prod.

Rotate NOT during active onboarding bursts. Pick a low-traffic window (Saturday morning UTC for prod).

Step 1 — Pre-flight (T-7d)​

1a. Schedule + announce​

  • Calendar block ~2h window in #ops.
  • Announce in #engineering 7 days ahead: "DKIM rotation for <domain> on <date>. Expect a brief (10-30 min) period during which both old + new DKIM signatures are valid; deliverability is unaffected."

1b. Capture current state​

# Current DKIM public key in DNS:
dig TXT google._domainkey.<domain>.work

# Should return one TXT record matching what Google Admin shows under
# Apps → Gmail → Authenticate email → DKIM

Save the current v=DKIM1; k=rsa; p=<old-public-key> value somewhere (incident channel pin or runbook prep doc) for rollback reference.

Step 2 — Generate the new key (T-0)​

2a. In Google Admin​

  1. Apps → Gmail → Authenticate email → select <domain> → "Generate new record"
  2. Choose key length: 2048 bit (do not go lower; 1024 is increasingly rejected by receivers)
  3. Select the selector — Google will typically suggest google (or google20XX if you want explicit dating)
  4. Google generates the key and shows: a new v=DKIM1; k=rsa; p=<new-public-key> TXT value to publish at <selector>._domainkey.<domain>

Do NOT click "Start authentication" yet — that activates the new key for signing before DNS has propagated.

2b. Publish to DNS​

In our DNS provider (Cloudflare DNS for *.work):

Type:     TXT
Name:     google._domainkey   (or whatever selector Google gave you)
Content:  v=DKIM1; k=rsa; p=<new-public-key>
TTL:      60s (so a rollback is fast if needed)

2c. Wait + verify propagation​

# Watch from multiple resolvers — at least 1.1.1.1, 8.8.8.8, and one off-network resolver
watch -n 10 'dig TXT google._domainkey.<domain>.work @1.1.1.1 | grep DKIM1'

Wait ~5 min for global propagation. Confirm dig returns the new public key from all 3 resolvers before proceeding.

2d. Activate in Google Admin​

Back in Google Admin → Authenticate email → click "Start authentication". Gmail will now sign outbound mail with the new key.

Step 3 — Verify outbound is signing with the new key​

3a. Send a test message​

# From a Specialist mailbox (or via service-account impersonation), send to an
# external account you control (e.g., your personal gmail):
gmail-send -from bob.acme@<domain>.work -to your-external-email@gmail.com -subject "DKIM rotation smoke"

3b. Inspect received headers​

In the recipient mailbox, view the message source. The DKIM-Signature header should:

  • Reference the new selector (e.g., s=google20XX)
  • Have d=<domain>.work
  • Validate (recipient mailbox should show "signed by: <domain>.work")

3c. Verify pass via tools​

curl "https://dkimvalidator.com/check?domain=<domain>.work&selector=google"
# Or paste the received message into mail-tester.com

Both should report DKIM pass for the new key.

Step 4 — Decommission the old key (T+7d)​

After 7 days of stable signing on the new key (so any in-flight mail signed with the old key has had time to be delivered + verified):

DNS provider → delete the old TXT record at the old selector path
(only delete after 7d; receivers cache DKIM keys for hours-days)

Update the per-domain DKIM record in our internal docs (docs/runbooks/dkim-key-rotation.md calendar block or wherever) with the new active selector + rotation date.

Rollback​

If Step 3 shows deliverability drop or DKIM-fail at receivers:

Rollback within Day 0 (key generated but not activated)​

Simply don't click "Start authentication". Delete the new DNS TXT record. We're back to status quo, no traffic affected.

Rollback after activation (Step 2d done)​

  1. Google Admin → Authenticate email → "Stop authentication" (reverts to no DKIM signing temporarily, OR returns to a prior key if one is still configured).
  2. If we still have the old DNS TXT for the prior selector → mail will resume signing with the old key.
  3. If we already deleted the old DNS TXT → re-publish the old key value (recovered from Step 1b capture). 60s TTL means fast propagation.
  4. Investigate why the new key failed (typo in DNS? selector mismatch? cached old value at Google side?).

Step 5 — Escalate​

  • Deliverability dropping > 5% post-rotation → page Platform Lead. Roll back immediately, investigate offline.
  • DNS provider not propagating the new TXT record after 10 min → contact provider support.
  • Google Admin "Start authentication" button greyed out for 24h+ → Google support ticket; this is a tenant config issue we can't fix client-side.
  • gsuite-tenant-suspended.md — DKIM auth doesn't survive a tenant suspension; restore tenant first, then re-verify DKIM still signing correctly
  • dns-mx-rollback.md — DKIM is unrelated to MX, but both are DNS-side; a clumsy DNS edit could affect both, careful
  • ADR-0002 Phase 2 — DKIM is tenant-level, not per-OSA — same key signs all Specialists on the domain
Last updated on by Degen