Skip to main content

Runbook: OAuth Token Refresh

2026-06-05 update: Specialist mailbox access is now via DWD service-account impersonation (PR #1722); per-OSA OAuth tokens are deprecated. This runbook covers the legacy OAuth path only. For DWD issues see docs/runbooks/gsuite-tenant-suspended.md or file a doc-gap issue.

P1. Each OSA stores an AES-256-GCM-encrypted Gsuite OAuth refresh token on org_specialist_assignments.oauth_refresh_token_encrypted. If refresh fails, the Specialist cannot send/receive mail until reauthorized. See OWNERS.md for on-call contact.

This runbook covers OAuth token failures in the per-OSA Gsuite lifecycle: expired refresh tokens, decryption failures after key rotation, and Google revocations. Implementation: api/src/workspace-lifecycle/oauth-token.service.ts. Architecture: ADR-0002 Phase 2 — Specialist Email (channels.md §System 3).

Symptoms​

  1. Gmail Send failures — outbound dispatcher logs OAuth: invalid_grant or OAuth: refresh_token expired for a specific email_alias.
  2. Inbound email silent drops — Gmail Push Pub/Sub notifications fire but gmail.users.history.list returns 401 with OAuth: invalid_grant.
  3. Sentry OauthTokenService warns — Could not decrypt OAuth refresh token for osa=<id>.
  4. Sentry OauthTokenService errors — Refresh failed: token revoked / expired from Google.
  5. Customer report — "Bob isn't replying to my email" (when the actual root cause is the Specialist's mailbox can't authenticate against Google).

Step 1 — Identify the affected OSA​

SELECT id, org_id, specialist_id, email_alias,
       oauth_granted_at, oauth_revoked_at,
       length(oauth_refresh_token_encrypted) AS token_bytes
FROM org_specialist_assignments
WHERE id = '<osa_id>';

Branches:

  • oauth_refresh_token_encrypted IS NULL → Never granted (OAuth flow never completed). Go to Step 4 (re-OAuth).
  • oauth_revoked_at IS NOT NULL → We explicitly revoked it. Go to Step 4 (re-OAuth).
  • Token bytes look fine but decryption fails → Step 2 (encryption key issue).
  • Token bytes look fine and decryption works but Google rejects → Step 3 (Google revoked).

Step 2 — Decryption failure (most often: MASTER_ENCRYPTION_KEY rotation)​

2a. Diagnose​

The oauth_refresh_token_encrypted column is AES-256-GCM, keyed by MASTER_ENCRYPTION_KEY. If the key was rotated without running the re-encrypt migration, old rows become unreadable.

Check Sentry for Could not decrypt OAuth refresh token. Cross-reference with MASTER_ENCRYPTION_KEY rotation events in the secret-rotation log (or aws secretsmanager describe-secret --secret-id humanwork/<env>/master-key for LastChangedDate).

2b. Fix​

  • If only a few OSAs are affected → individual re-OAuth per Step 4.
  • If many OSAs affected (key rotation regression) → roll back the key to the previous version in Secrets Manager, restart pods, verify decryption resumes. Then plan a controlled re-encrypt migration before re-rotating.

Step 3 — Google-side revocation​

3a. Diagnose​

Google may revoke a refresh token because:

  • The Google Admin revoked the app's authorization manually
  • Token was unused for > 6 months
  • Customer's password was changed (sometimes triggers revocation)
  • Google security event (impossible travel, etc.)

Check the Google Admin → Reports → Audit log → "Token" subtype for the affected user.

3b. Fix​

There's no programmatic way to revive a revoked token. Go to Step 4 (re-OAuth).

Step 4 — Re-OAuth the OSA​

  1. Mark the OSA as needing re-auth: 
    UPDATE org_specialist_assignments
    SET oauth_refresh_token_encrypted = NULL,
        oauth_revoked_at = NOW(),
        oauth_granted_at = NULL
    WHERE id = '<osa_id>';
  2. Notify the AM owning the Org: "Specialist {firstName}'s workspace needs re-authorization. Walk the client through the OAuth flow at /ops/clients/<orgId>/specialists/<osaId> → Reauthorize."
  3. AM clicks Reauthorize → user lands on Google OAuth screen → grants consent → server stores fresh oauth_refresh_token_encrypted + sets oauth_granted_at = NOW().
  4. Verify by triggering a test Gmail Send (any outbound message through the Specialist).

Rollback​

OAuth state changes are reversible in only one direction: you can always re-grant. You cannot un-revoke (Google's contract).

If you accidentally set oauth_revoked_at on a working OSA → re-OAuth flow per Step 4 (same as if it was genuinely revoked).

Step 5 — Escalate​

  • If decryption fails for all OSAs simultaneously: P0, page Platform Lead. Likely root cause: MASTER_ENCRYPTION_KEY is wrong/missing in production. Verify env var; if unrecoverable, restore from Secrets Manager backup.
  • If Google revokes tokens at a rate > 1 per hour for the tenant: contact Google Workspace support — they're flagging the app for unusual activity.
Last updated on by Degen