Skip to main content

Runbook: Gsuite Tenant Suspended

P0 always. Google suspended our entire Workspace tenant. Every Specialist mailbox in the affected tenant is unreachable; every OSA in the org becomes a black hole for inbound + outbound mail. See OWNERS.md for on-call contact.

This runbook covers Google Workspace tenant-wide suspension — distinct from individual user suspension (which is gsuite-license-cap.md Step 3a). Tenant suspension is rare but catastrophic: usually billing default, Terms of Service violation, or detected abuse.

Symptoms​

  1. All workspace-provisioning jobs failing simultaneously across multiple OSAs with Account suspended / Domain in violation.
  2. All inbound email going to NDR (User account is suspended).
  3. All outbound Gmail Send returning 403 account_suspended_by_administrator or domain_suspended.
  4. Google Admin Console banner at top: "Your account is suspended" or "Your subscription is past due".
  5. Email from googleworkspace-noreply@google.com with subject "Important: Your Google Workspace account has been suspended" — read it carefully, it states the reason.

Step 1 — Identify the suspension class (within 30 min)​

Google suspends tenants for 3 distinct reasons, each with a different remediation path:

ClassTriggerRemediation
BillingPayment failed (expired card, hold on account)Fix payment → reinstated within hours
PolicyToS violation (spam, phishing, abuse reports)Appeal via Google support → 24-72h
SecurityGoogle detected suspicious activity (compromise)Verify identity → 24h

The suspension email from Google states which class. If you don't have the email, check Google Admin Console banner text. DO NOT GUESS — the wrong remediation path can extend the suspension.

Step 2 — Mitigate customer impact (parallel with Step 3)​

While the suspension is in effect:

2a. Customer-facing notice​

Send to every Org owner / AM in the affected domain (h.work / h852.work / h853.work based on which tenant is suspended):

Subject: Temporary email delivery issue — <domain>

We're working with Google to restore email service on <domain>. Inbound and outbound messages on your Specialist's @<domain> email may bounce until restored (ETA: ). Please use Slack, WhatsApp, or the web portal as a workaround.

Send from a non-Gsuite address (e.g., personal email or Sendgrid). The whole point is our Gsuite is down.

2b. Mark all OSAs as inbound-paused​

UPDATE org_specialist_assignments
SET metadata = jsonb_set(
    coalesce(metadata, '{}'::jsonb),
    '{tenant_suspended}',
    'true'::jsonb
)
WHERE email_alias LIKE '%@<suspended-domain>';

This is metadata-only; the UI can show a banner per-OSA: "Workspace suspended — waiting for Google."

2c. Outbound dispatcher fallback​

Correction (2026-06-05): Resend does not fall back for Specialist outbound. Resend is reserved for invites / onboarding / transactional mail only; Specialist outbound goes through Gmail-DWD send-as. If Gmail-DWD fails for a Specialist there is no automatic provider fallback — the message stays in queue and surfaces as a dispatch error. Losing the Workspace tenant therefore halts Specialist outbound for that tenant until reinstatement.

Verify dispatcher behaviour during incident:

grep "gmail-dwd send-as failed" <recent-api-logs>

If a Specialist message did not error and somehow went out via Resend → that's a bug (cross-channel leak); file it.

Step 3 — Resolve the suspension​

3a. Billing class​

  1. Google Admin → Billing → Payment methods → identify the failed payment.
  2. Update card on file or pay the past-due balance.
  3. Reinstatement is usually automatic within hours; if not, contact Google Workspace billing support.

3b. Policy class​

  1. Read the suspension email carefully — it cites the policy and (usually) a specific event/message that triggered the flag.
  2. Investigate from our side: did we send phishing-looking emails? Bulk outbound? See email_audit_log for outbound activity around the suspension time.
  3. File an appeal at https://support.google.com/a/contact/reinstatement with:
    • Tenant ID
    • Suspension date/time
    • Our explanation + remediation (e.g., "We've added stricter outbound rate limits and blocked the specific abuse vector")
  4. Wait 24-72h for Google to review.

If you find we were actually doing something policy-violating: stop it immediately, then file the appeal with proof of remediation. Don't hide it; Google sees the audit log too.

3c. Security class​

  1. Google's email will request "identity verification" via a specific link.
  2. Click the link, provide the requested verification (admin's photo ID, etc.).
  3. Wait ~24h for Google to review.
  4. Once restored: rotate all Gsuite-related secrets (GSUITE_SERVICE_ACCOUNT_KEY_JSON, all OAuth refresh tokens) on the assumption that the trigger was a real compromise.

Step 4 — Verify restoration​

When Google indicates reinstated:

  1. Google Admin Console banner clears.
  2. Smoke test: 
    echo "tenant restored $(date)" | gmail-send-via-service-account bob.acme@<domain>
  3. Wait for inbound NDRs to stop (external senders' MTAs retry NDRs for ~72h, so legitimate inbound starts flowing within minutes; old NDRs in queues will gradually drain).
  4. Restart any stuck workspace-provisioning BullMQ jobs per workspace-provisioning-failure.md Step 3.
  5. Clear the per-OSA metadata.tenant_suspended flag: 
    UPDATE org_specialist_assignments
    SET metadata = metadata - 'tenant_suspended'
    WHERE email_alias LIKE '%@<domain>';

Rollback​

Tenant suspension is not user-rollback-able — only Google decides when to reinstate. Until then:

  • Cannot DNS-rollback to Cloudflare worker as an interim — the CF worker forwards to our API, which then tries to read/send via Gsuite, which is down. Rollback would only restore the inbound webhook path, not delivery.
  • Workaround for individual customers: ask them to communicate via Slack / WhatsApp / web portal for the suspension window.
  • Worst case: stand up a parallel Workspace tenant on a backup domain (h-backup.work?) — multi-week setup, not realistic for an active customer SLA event.

Step 5 — Escalate​

  • Page Platform Lead + CEO immediately. Suspension means we're publicly seen as broken by every customer in the affected tenant.
  • Loop in legal if it's policy-class — appeal language matters and we don't want to over-promise.
  • Schedule a post-mortem regardless of class.

Track in incident channel:

  • Class (billing / policy / security)
  • Suspension start time
  • Customer impact estimate (# affected OSAs × messages/day average)
  • Workspace tenant ID
  • Reinstatement ETA per Google email
Last updated on by Degen